Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach
Authors: Aidan M. Swope
Xander H. Rudelis
Kyle T. Story
Venue:  (2020)
Keywords: unsupervised learning
representation learning
deep learning
remote sensing
sensor fusion

Abstract: In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent self-supervised learning approaches are ill-suited to the remote sensing domain. In addition, most remote sensing applications currently use only a small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose a new self-supervised training objective, Contrastive Sensor Fusion, which exploits coterminous data from multiple sources to learn useful representations of every possible combination of those sources. This method uses information common across multiple sensors and bands by training a single model to produce a representation that remains similar when any subset of its input channels is used. Using a dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors. These representations outperform fully supervised ImageNet weights on a remote sensing classification task and improve as more sensors are fused.

Under review as a conference paper at ICLR 2020
REPRESENTATION LEARNING FOR REMOTE SENSING: AN UNSUPERVISED SENSOR FUSION APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent self- supervised learning approaches are ill-suited to the remote sensing domain. addition, most remote sensing applications currently use only small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose Contrastive Sensor Fusion, which exploits coterminous data from multiple sources learn useful representations of every possible combination of those sources. This method uses information common across multiple sensors and bands by training a single model produce representation that remains similar when any subset of its input channels is used. Using dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors. These representations outperform fully supervised ImageNet weights on remote sensing classification task and improve more sensors are fused. Our code is
self-supervised training objective,
available bttpe://storage. cloud. gogge. con/ ppwp Lo-gab wtw.saatttssssssss/////sss ott.tsst_ccoooooooooo wcdd..dde.....
bridge
Bare Rock
stadium
Breakwater
substation
Bridge
farnland
Dam
quarry
Farm (building)
Farmland
Forest
Golf Course
Quarry
Stadium
farm
forest
Substation
Water
Water
golf
Figure 1:
Learned representations of sample image scenes, visualized with PCA followed by t-SNE and colored by OpenStreetMap category. Without any labels, Contrastive Sensor Fusion has learned a representation that groups remote sensing images into semantically meaningful categories.
1
INTRODUCTION
Remote sensing data has become broadly available at the petabyte scale, offering unprecedented visibility into natural and human activity across the Earth. Many techniques have been recently

---

Under review as a conference paper at ICLR 2020
developed for applying this data with machine learning to solve geospatial tasks like semantic segmentation (Audebert et al. 2016)) (Kampffmeyer et al. (2016), broad search (Keisler et al. 2019), and classification (Maggioni al. (016), (Sherrah 2016).
Due to the complexity and visuospatial nature of solving problems with acrial imagery, it natural to use deep convolutional neural networks, but CNNs typically require large amounts of labeled data to achieve good performance. In remote sensing. these labels are usually scarce and hard to obtain; semantic segmentation requires boundaries be labeled single-pixel precision. A modern approach to the problem of data scarcity is semi-supervised learning, which uses unlabeled data to ease the task of learning from small amounts of labeled data. This approach particularly well-suited to remote sensing because of the amount of unlabeled data available.
Figure 2: Coterminous remote sensing imagery from three different sensors: Airbus SPOT, NAIP (visualized here near-infrared, red, and green), and Airbus Pléiades (see Appendix [A.3] for details). As seen here, images contain many small components (roads, buildings, structures, trees) and adjacent locations can look completely different (c.g., the transition from buildings to grass). We leverage these multiple views to generate representations with any subset of available sensors or channels.
While most self-supervised and unsupervised image analysis techniques focus on natural imagery, remote sensing differs in several critical ways, requiring a different approach. Where pictures taken by a human photographer often have one few subjects, remote sensing images like those in Figure contain numerous objects such buildings, trees, or factories. Additionally, the important content can change unpredictably within just few pixels or between images at the same location from different times. Multiple satellite and aerial imaging platforms capture images of the same locations on earth with a wide variety of resolutions, spectral bands (channels), and revisit rates, such that any specific problem can require a different combination of sensor inputs (Reiche al. 2018 Rustowicz ct al. 2019).
Recent research in semi semi-supervised learning has led wealth of methods that achieve success
problems like classifying natural images (van den Oord et al. 2018 | Tian et al., 2019 Hénaff et al. 2019) and understanding language (Mikolov | 2013) Devlin et al. 2018). These approaches almost universally rely on the "distributional hypothesis" " the property that parts of the data that are close in time or space are similar. Previous work (Jean al. 2019) has learned representations for overhead imagery using the distributional hypothesis. However, we argue that it is less applicable in remote sensing, due the aforementioned differences between overhead imagery and these domains. The distributional hypothesis still applies, but the scale of correlated patches is smaller, making it difficult harvest scenes for related patches. We therefore modify these techniques to build on the intuition that similar layouts of objects on the ground should have similar representations, regardless of the combination of sensors.
In this paper, we use this idea to develop a method of learning representations for overhead imagery. which we call Contrastive Sensor Fusion (CSF). We train a single model, unsupervised, to produce a representation of scene given any subset of the sensors used during training over that scene. During training, we form two "views" of each scene from a small random subset of the available sensor channels (channel/sensor dropout). Then both views encoded with the same network and the resulting representations are compared, using a contrastive loss to encourage the two views to have similar representations.
2

---

Under review as a conference paper at ICLR 2020
This model is trained -20 TB dataset consisting of 47 million scenes, each consisting of four bands for each of three different sensors over that scene. We perform several experiments, showing that the unsupervised encoder learns to produce representations of many different combinations of sensors that separate images into semantiaaly-mmannigg in the supeevseee setting. The representations are transferred as-is, with no fine-tuning step or stacking additional models.
classes and perform well on real problems
2
BACKGROUND
Contrastive methods train models by using the representation of one observation to predict the representation of different but related observation. Examples include pairing frames from the same video (Wang and Gupta 2015) or words from the same local context (Mikolov et al.) (013)) The recent method Contrastive Predictive Coding (CPC) uses representation of the current time step in sequence to predict the same encoder's representation of a future time step (van den Oord et al. 2018 Hénaff al. 2019). CPC has been shown to learn expressive representations of natural imagery by using the representation of one image sub-patch to predict the representation of adjacent sub-patches patches, contrasting against patches drawn from other images. However, this adjacent- patch technique fails for remote sensing imagery, which can change abruptly between adjacent patches.
Contrastive Multiview Coding (CMC, Tian et al. 2 is related approach that compares multiple views of the same context. Unlike CPC, CMC does not assume that representations of image patches should be stable in space and benefits from having many views of scene, making it applicable to remote sensing. On the other hand, CMC relies on fixed partitioning of data sources, forcing practitioners to decide ahead of time which sensor combinations to fuse. It also trains a separate model for each view, limiting how much information can be shared between views.
We use the InfoNCE loss introduced by van den Oord et al. (018). Intuitively, its role is to train the encoder to make representations of the same scene similar across views. Computing the loss the level of representations instead of pixels sidesteps issues with reconstruction based loss functions like autoencoders use, which dominated by low-level properties of pixel space like overall brightness instead of high-level information; however, directly predicting one representation from another leads to trivial solutions, such as representing every scene with the zero vector. Instead, contrastive losses like InfoNCE ask the network to classify which representation among a set including many "noise examples" belongs the same scene.
We borrow a number of ideas directly from Deep InfoMax (Hjelm et al. 2019) and its successor, Augmented Multiscale Deep InfoMax (AMDIM) (Bachman et al.) [2019]. These methods learn by contrasting representations at various layers and spatial locations of the same network. AMDIM increases the difficulty of this task by independently augmenting each view. Computing the loss at multiple levels and sharing the network weights between views also increase supervisory signal to the model. As with CPC, however, AMDIM assumes that distinct patches from the same image are closely related, which is problematic in remote sensing.
Finally, we tested the non-contrastive method of Split-Brain Autoencoders (Zhang et al., 2017) 7), but found them to be less effective than contrastive methods. Other pixel-space methods (e.g. Singh et al. 2018) exist for remote sensing representation learning, but these face the issues with reconstruction- based losses mentioned above.
3 CONTRASTIVE SENSOR FUSION
We introduce Contrastive Sensor Fusion (CSF, Figure 3 a technique for learning unsupervised representations of every combination of its input sensors. CSF learns by creating two views of each location by randomly sampling two sensor combinations, encoding both views with the same CNN (sharing weights), and comparing representations across locations with a contrastive loss that aligns representations of the same location regardless of the input sensor combination.
Our method can be seen as an extension of Contrastive Multiview Coding, with three critical differences. First, instead of dividing the input channels into fixed groups during training. CSF drops out channels at random, forcing the network to learn to encode every combination of channels. Second, CSF uses "Siamese network" (Bromley et al. 1993) training scheme, sharing weights
3

---

Under review as a conference paper at ICLR 2020
across the encoders of every sensor combination rather than training a separate encoder for each view. Third, CSF computes contrastive loss at multiple layers representation, which helps to learn localized representations. In sense, CSF can be seen computationally tractable way of training an ensemble of all syponentiallymmany" splits of views CMC at once, with weight sharing between each encoder and loss at multiple levels. This is analogous to the extension of NADE models [Larochelle and Murray 2011) to ensembles of arbitrary ordering (Uria et al. 2014).
3.1 LEARNING FROM MULTIPLE VIEWS
Like CMC, CSF learns by contrasting representations of multiple views of the same input. Given set of sensor looks X over scene, aim create views (X), V (X) that contain the same high-level information (e.g. object identities and location, scene type, topography) but differ much as possible the pixel level. order match representations of the same scene, the model must encode both views in such way that the scene remains distinctive. If the views differ enough, this forces the encoder capture high-level information about the scene, since that the most salient remaining information common to both views, while discarding nuisance factors like lighting, resolution, and differences between sensors.
View (())
View ((4)
channel dropout random flips random rotation
channel dropout random flips random rotation random cropping brightness jitter contrast jitter
random cropping brightness jitter
contrast jitter
(B, 128, 128, 12)
(B, 128, 128, 12)
Co- Co-located sensor inputs
ResNet Stack (B, 32, 32, 256)
ResNet Stack (B, 32, 32, 256)
ResNet Stack (B. 16. 16, 512)
ResNet Stack (B. 16, 16, 512)
+ ResNet Stack (B, 8, 8, 1024)
1 ResNet Stack (B, 1024)
Features Za, (B, 65536)
Contrastive Loss daL (Za,1, Lz,2)
Features Zs.2 (B, 65536)
ResNet Stack (B, 4, 2048)
Features Z4,1 (B. 32768)
Contrastive Loss duh (Za.1, Zo.2)
Features Za,2 (B. 32768)
ResNet Stack 4 (B, 4, 2048)
Encoder copy 1
Encoder copy 2
Figure 3: Contrastive Sensor Fusion architecture during training. Weights shared across encoder copies. The contrastive loss trains the encoder to represent the same (different) location the same (different) way regardless of sensor/channel combination. The process create views is explained in more detail in Figure A.6 and the computation of the loss is detailed in Appendix A.2
To create view V(X), we randomly set large fraction of the channels in X to zero, which effectively fuses different combination of sensor channels for each example. With channel dropout rate of we scale the remaining channels by T-p as in Srivastava et al. (2014). Similarly, at inference time the encoder with any subset of the channels it tranedd with, replacing the others with zeros and scaling up remaining channels by the same factor as was used during training.
We augment images by randomly cropping a small number of pixels from X, applying random flips and rotations, and jittering the brightness and contrast of each remaining channel; these increase the difficulty of the contrastive task and improve the robustness of the representation nuisances.
Given n sensors, there are 2" possible sensor combinations (counting all subsets and discounting the empty set), and grows, training a separate model for each infeasible.
4

---

Under review as a conference paper at ICLR 2020
Because CSF uses single network for multiple sensors with potentially different resolutions, numbers of channels, and even modalities, care must be taken to design an encoder which can fuse them effectively. In experiments,
used three sources of optical imagery with slightly different pixel resolutions (see A.3). We bilinearly upsampled the two lower-resolution sources to match the highest resolution, then concatenated the sensors' channels depth- depth-wise such that each pixel fed the first convolution covers the same physical area.
3.2 INFONCE LOSS
CSF compares representations at multiple levels of the encoder using the InfoNCE loss. As van den Oord et al. (2018) show, InfoNCE maximizes the mutual information (MI) between representations. We design our vves-view augmentation to destroy MI between views except for the high-level understanding, so maximizing the MI between representations directly trains the encoder to learn robust and expressive representations of its input.
At each of several layers the encoder, contrastive loss computed as follows. Let V be a stochastic function which maps set of sensor looks X to view, and let Zz be a function which maps a view to the encoder's layer- representation of that view. We apply V to X twice to produce two views and encode both into layer-Z, representations, which we denote by 2[, (v(L) and z = Zz (X))) Let ø be a scoring function on this representation space. á plays the role of contrastive learning's predictive model; given two views, it produces the logit of the two views belonging the same scene. Given collection of other scenes Xnoise {X X * X} and their layer-L representations noise (V(X))) .... (x(")))) used noise examples to classify among, the loss at layer L for example X is the InfoNCE loss presented in CPC,
Clorward (X) = log
exp (( (
)) + Σ (x)( Z))
This is the loss function for predicting view from view which we refer to as the "forward" loss. Because the problem is symmetric (we are comparing representations of the same encoder run over two identically-creater views), we can use the same procedure and set of negatives to predict view from view 2, the "backward" loss. The total contrastive loss for layer L, Lt, is the sum of the two, and the total loss for example X is
Liol MeLL
where AL weights the contrastive loss at layer L. Following the intuition that two views of scene are most similar in abstract high-level ways, we use nonzero weight on only last layers of the last two residual stacks, and weigh the last layer twice as heavily.
In all experiments, we use of d( z(²)) = z(()
and draw negative examples from other scenes in the batch. This allows very efficient computation of the contrastive loss. See A.2 for more details.
Computing contrastive loss at multiple layers is efficient, since it requires applying the encoder only once per view, and is beneficial for number of reasons. Since modern CNNs use multiple layers of pooling, higher layers in the encoder are naturally more translation invariant and so multi-layer loss helps CSF learn well-localized features useful for segmentation. Additionally, the extra supervisory signal helps earlier layers learn more quickly.
3.3 TRAINING
During training, both copies of the encoder share weights. As argued above, this trains single model which can function with any of its input channels missing. In addition, weight-sharing makes the encoder more parameter-efficient. This can lead to information sharing across sensors; for example, the network may learn to combine the blue channel of one sensor with the green channel of another early in the network, since the two bands often provide similar information.
2 For sensors of very different resolutions, training a separate head for each sensor hefore combining the representations may be appropriate.
5

---

Under review as a conference paper at ICLR 2020
In contrast to CPC and AMDIM, computing the loss does not require masking the encoder using an encoder with a restricted receptive field size. CSF learns representations of whole images and contrasts representations from distinct scenes rather than distinct parts of the same image, standard architecture can be used for the encoder. Following van den Oord et al. (2018), we use a ResNet encoder (He al. 2013). We choose ResNet-50 instead of 100 for experiments because its smaller memory footprint enables us to use larger batch size. Larger batch sizes result in more accurate loss since the negative examples for NCE the other scenes in the batch. We train on single TPU v2 with batch size of 2048.
During training we schedule the channel dropout rate increase linearly from 0.66 over the first 8000 batches. This is form of curriculum learning (Bengio et al., (2099 where the contrastive task starts in easier setting where more band are retained, and gets gradually harder throughout training: we observed faster convergence with this channel dropout schedule. We also linearly warm up the learning rate over the first 3000 batches (Goyal et al. (017). Other hyperparameters are kept constant; when making each view, randomly crop away 32 pixels and randomly jitter the brightness and contrast of each retained band by at most 25%.
4 EXPERIMENTS
We test this method by training CSF network on 47 million image triples from three imaging platforms: Airbus SPOT, USDA NAIP, and Airbus Pléiades, with pixel resolutions of 150 em, 100 cm, and 50 cm, respectively. Each has four channels: red, green, blue, and near-infrared (NIR). See Appendix A..3 for details. To visualize the feature space and evaluate the learned representations, built dataset of 8400 samples based OpenStreetMap (OSM) features from 12 classes of distinctive objects; example images are shown in Figure and Figure Each object is seen in all three sensors used to train the encoder. See Appendix A.4 for details.
0.5
0.5
Fraction of Neighhors Same Class
0.4
0.3
CSF features, hi-res first
0.2
0.2
CSF features, lo-res first
ImageNet hi-res
0.1
0.1
CSF features 12ch
Random 12ch
ImageNet, lo-res
CSF features
Random RGB
Random, lo-res first
ImageNet RGB
0
4
12
10
25
50
100
Input Channels Used
A Nearest Neighbors
We compare the clustering of features based on OSM class using a nearest neighbor metric. The plots show the fraction of same-class neighbors for each point (k: 10) as input channels are added (Ieft), and the fraction of same-class neighbors as a function of A (right). One, two, and three-channel experiments always use a single sensor, taking the red band only, the red and green bands, and the RGB bands respectively. Our features outperform ImageNet's 's in this unsupervised clustering metric and improve when multiple sensors are fused.
Figure 4:
We test the performance of the trained CSF network with several experiments. For each scene in the OSM-based dataset, we produce representations from ResNet50V2 encoder initialized with weights either learned through CSF, from sully-supervised ImageNet classification, set randomly. We use various combinations of input channels and sensors (although ImageNet weights are limited input channels). We reduce the dimensions of the generated representations principal component analysis (PCA). In Figure T we visualize the CSF representations by embedding the 200 principal components of the features obtained from 12 input channels into two dimensions with t-SNE, Next,

---

Under review as a conference paper at ICLR 2020
measure the quality of clustering by identifying the k-nearest neighbors to each point in representation space comp-cenmn PCA) and count the fraction that belong to the OSM class of that point. We
also test classification performance by training k-nearest neighbors (KNN) classifier with & = 10 and leave-one-out cross validation on every point and measure the classification accuracy (Table T..
We report the average of each metric over the OSM dataset.
Experiment results from several encoder-channel combinations are presented in Figure as function of non-zero channels (panel 1) choice of k (panel 2). Because ImageNet weights only allow three
or fewer input bands, we test ImageNet weights applied to either the highest resolution (Pléiades) or
lowest resolution (SPOT) RGB imagery. K-NN classifier accuracy is presented Table [
4.1 REPRESENTATION QUALITY
First, evaluate the quality of learned representations by considering how well they cluster scenes by OpenStreetMap object category. Plotting the resulting points colored by OSM label (Figure [ we observe that most classes form distinctive clusters. Neither the encoder nor the dimensionality- reduction algorithms have access to these labels; therefore, we can observe directly that CSF clusters images into semantically meaningful categories.
Table 1: We compare the representation quality of CSF and ImageNet weights for various sen- sor/channel combinations by training a KNN classifier on our OSM dataset. This table shows the fraction of 10-nearest neighbors belonging to the same class, and the accuracy of the classifier.
Weights
Channels
10-NN same-class (%)
Accuracy (%)
ImageNet
Airbus SPOT RGB
34.69
47.96
Airbus Pléiades RGB
43.34
57.41
CSF
Airbus SPOT RGB
39.10
53.16
Airbus SPOT RGB + near-IR
42.31
55.71
Airbus Pléiades RGB
43.40
57.47
Airbus Pléiades RGB near-IR
45.68
60.23
All sensors, RGB + near-IR
50.22
64.06
The clustering quality is quantified in Figure 4 and Table [ In every case for both nearest-neighbor class and classification accuracy, representations learned through CSF outperform those learned with all ImageNet labels (Figure 4). We emphasize that training the ImageNet encoder required 14 million labels whereas the CSF encoder was trained purely unsupervised. Though overhead imagery differs significantly from ImageNet's labeled natural images, out-performing ImageNet is not trivial; previous work (Singh et al., 2018) demonstrates that supervised ImageNet pretraining is a strong baseline for representations of remote sensing imagery
Furthermore, as we add channels the unsupervised CSF encoder continues to improve. We also investigated all other orders of adding sensors, and chose display the order which resulted in lowest scores first; that is, adding the lower resolution data before the higher resolution data. This demonstrates limitation of transfer learning from datasets like ImageNet: the resulting weights can only be used with channels present in the dataset. Most labeled data is RGB, so supervised transfer learning prevents us from taking advantage of multiple co-registered data sources.
4.1.1 MOTIVATION
This OSM classification task demands that representation understand the kinds of objects and relationships useful for broad range of remote sensing applications. It demonstrates that the features learned, whether high-level or low-level, are features which allow discriminating between semantically distinct sets of input images. Intuitively, representations which do not cluster well among semantic classes require learning more complex transformations to separate the classes.
We compare results a multiclass classification problem, expect supervised training with ImageNet lahels to be particularly effective here relative e.g. segmentation task.
7

---

Under review as a conference paper at ICLR 2020
Fine-tuning experiments have the potential demonstrate this fact, and note this future avenue for research. However, we believe that special care will need to be made to show that features continue generalize to tasks where labeled data is not available for fine-tuning.
4.2 SENSOR FUSION
Next, we investigate our main claim: that CSF learns to fuse different sensors with minimal degra- dation in feature quality when only a subset of sensors are available. We observe that clustering quality increases monotonically as channels are added (first panel of Figure Though the biggest gains come from using high-resolution imagery, fusing multiple low-resolution sensors noticeably improves representations above having single higher-resolution sensor. We were surprised to discover that fusing SPOT and NAIP imagery with CSF results in higher performance than CSF representations of nhigher-resolutin at input channels to purr-ppuple points at I 4 input channels. In the second panel of Figure [ the fully fused 12-channel CSF network outperforms 3-band ImageNet at all values of k.
Pléiades imagery, seen by comparing the light-purple point
In every case, performance increases as more channels added, regardless of order. This demonstrates that CSF is effectively combining the information present across multiple sensors, and that CSF has learned a good representation for each of its possible views. We expect that as non-optical data sources are added, CSF will outperform transfer learning from ImageNet to even greater degree. This result suggests that CSF has much to gain from adding more sensor views than we try in this work.
SPOT
Component
NAIP
PHR
SPOT
Component 2
NAIP
PHR
SPOT
Component 3
NAIP
PHR
Figure 5: For each of the first three principal components of the 12-channel CSF representation space, we show 10 images from each single sensor (with inputs for the other two sensor zeroed) that maximally activate these directions. These principal components of representation space represent contain concepts (fields, bridges, and bare ground concrete) stable across sensor combinations,
Finally, in F we demonstrate that CSF learns high-level representations that are stable across sensors. As before, we identify the principal components of the encoder's representaitons of the 12-channel OSM dataset (which, as Szegedy et al. (2014) note, are not distinguished from individual channels). Then, encode each sensor's s views individually and visualize the images which maximally activate those linear combinations Figure 5) We find that each direction visualized corresponds the same high-level concept, regardless of which sensor produced it. This suggests
8

---

Under review as a conference paper at ICLR 2020
that CSF learns to fuse multiple sensors in early layers into disentangled and censor-invarian features which will be easy for later layers and transfer tasks to use due to their directional consistency.
5 CONCLUSIONS AND FURTHER WORK
In this work we present Contrastive Sensor Fusion (CSF), a new self-supervised training objective to learn fused representations of multiple overhead image sources. CSF uses contrastive loss to train an encoder that can produce a shared representation from any subset of available channels across multiple sensors. Using dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors, out-perforinng fully-supervised ImageNet weights which required 14 million labels to train. We show through experiments that the network is successfully fusing multiple sensor information into representations that improve with additional views. We also show that, through sensor fusion, multiple low resolution sensors can outperform a single higher resolution sensor. While this work considered only optical sensors with similar resolutions, remote sensing practitioners frequently use varity of sensors including non-optical and hyperspectral imagery with many channels. We expect multi-sensor representations of these to outperform supervised learning transferred from natural imagery an greater degree than demonstrated here.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vladimir Agafonkin. A new algorithm for finding a visual center of a polygon.
https:
//blog.mapbox com aaaaaa-nwwwwall
2-pollyonn-76777442222 2016.
Nicolas Audebert, Bertrand Le Saux, and Sébastien Lefevre. Semantic segmentation of earth observation data using multimodal and multi-scale deep networks. In Asian conference on computer vision, pages 180-196. Springer, 2016.
Philip Bachman, Devon R. Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: .090000100 2019.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09, pages 41 41-48, New York, NY, USA, 2009. ACM. ISBN 97881--6058--111111 doi: 10.1 10.1145/1553374 1553380. http: acm. http:///.......o....oo....61//// 1553380
Jane Bromley, James Bentz, Leon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Sackinger, and Rookpak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7:25, 08 1993. doi: 00144///02100149390000
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.044 2018.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. ArXiv, abs/1 706.02677, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1 2015. URL
Olivier J. Hénaff, Ali Razavi, Carl Doersch, S. M. Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: 1905.09272, 2019.
Devon R. Hjelm, Alex Fedorov, Samuel aavoie-Marchildoen Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019. URL https?
//openreview.o net feret/forree forum? wwcpummmidppe r3 cKX
Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David B. Lobell, and Stefano Ermon. Tile2vec: Unsupervised representation learning for spatially distributed data. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolula, Hawaii, USA, January 27 February 1, 2019., pages 3967-3974, 2019.
Michael Kampffmeyer, Arnt-Borre Salberg, and Robert Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1 1-9, 2016.
Ryan Keisler, Samuel W. Skillman, Sunny Gonnabathula, Justin Pochnelt, Xander Rudelis, and Michael S. Warren. Visual search over billions of aerial and satellite images. Computer Vision and Image Understanding, 187:102790, 2019.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Geoffrey
Gordon, David Dunson, and Miroslav Dudik, editors, Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 29-37, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL http: 7/proceed ngs. ml prwssrsssaa. //rrch/ 1ella.lll.. 11m.mmml
10

---

Under review as a conference paper at ICLR 2020
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Convolutional neural networks for large-scale remote-sensing, image classification. IEEE Transactions on Geoscience and Remote Sensing, (5(2):645-653) 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013.
Johannes Reiche, Eliakim Hamunyela, Jan Verbesselt, Dirk Hoekman, and Martin Herold. Improving near-real time deforestation monitoring in tropical dry forests by combining dense sentinel-1 time series with landsat and ALOS-2 PALSAR-2. Remote Semsing of Environment, 147-11,,,,, January 2018. doi: 10. 10.10166j.se.2011.110..m
/tLhttps::/ http://oo..orgggo httpatta////cb.oog/11100000/////
2017.10.034
Rose M. Rustowicz, Robin Cheong, Lijing Wang, Stefano Ermon, Marshall Burke, and David Lobell. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Jamie Sherrah. Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery. arXiv preprint 10..vv:606.02220 2016.
Suriya Singh, Anil Batra, Guan Pang. Lorenzo Torresani, Saikat Basu, Manohar Paluri, and C. V. Jawahar. Self-supervised feature learning for semantic segmentation of overhead imagery. In BMVC, page 102, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15(1): 1929 1958, January 2014. ISSN 1532-4435. URL http: potttp//dl.... //t.///lmmcmmmmm acm. org/ citation. can..cfm cfm?
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. URLhttt:://arat wtwpaawwww.....oaaaaaaoa wwt./o/g////////oooo/ooooooo abrgg//bs//11..... 6199
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Pro- ceedings of the 31 International Conference on International Conference on Machine Learning
Volume 32, ICML' 14, pages 1-467-1-475. JMLR.org. 2014. URL http: //11 acm. org
citation, cww.iionncmmiic.mm 30:d03304440 3044805. 3044859
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:/807.033: /807.03748, 2018.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794-2802, 2015.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058-1067, 2017
11

---

Under review as a conference paper at ICLR 2020
APPENDIX
A.I VIEW CREATION AND LOSS DETAIL.
For each location
Resample to 0.5m resolution
Multiple coterminous scenes
View Xs,1
Asymmetric channel dropout, random cropping. flips rotations, brightness and contrast jitter
View X
B,
These two views constitute one batch element
Features z
encoder copy
encoder copy 2
(1)) 1,
((2)
2 (2)
Features (12))
S =
(1) B::
(1)
B.
B logsoftmax(S) + logsoftmax (ST) n.a.
Figure A.6: This figure shows the flow of data during training, from the creation of views to the loss
calculation. The loss is shown for one feature extraction block i. ( (2) is batchwise matrix of feature similarities; the diagonal corresponds to the similarity of representation matched batch elements.
A.2 EFFICIENTLY COMPUTING THE CONTRASTIVE LOSS
The primary advantage of unsupervised training is that it grants access to much more data. To take full advantage of this, each training step must be efficient.
Using ((=(1), 2(2) ²(1)T" _(2) and drawing negative NCE samples from other images in the batch enables computing the contrastive loss very efficiently. Given batch of two feature maps, (1), (2)
12

---

Under review as a conference paper at ICLR 2020
both of shape N2) where B is our batch size and N is the flattened size of each spatially down-same down-sampled feature map, we define their similarity matrix as
(1)
B.
S z() 2(2)²
:
(A.1)
1 (2)
2 B2 B.
Each element of this matrix measures, per batch, how similar (under () elements' features generated from view are to all other elements' features generated from view 2. Thus, the diagonal elements represent similarity features from samples of the same origin but different dropout, cropping, and color and contrast jitter. We measure the neural net's s ability to associate features of identical origin with one another, rather than negatives in the same batch, treating this as a classification problem. We take the log-softmax along each axis to find predicted associations from view I to view 2 and vice-versa.
The contrastive loss this layer is computed the mean of diagonal elements of the log-softmax of both the similarity matrix and its transpose
Ccontristive (1 (2)
Σ B
[diag (log-softmax(S)) diag (log-softmax (s²)))B
(A.2)
The diagonal elements of the log-softmax of the similarity matrix give the loss for predicting view from view I, and performing the same process with the similarity matrix's 's transpose (equivalently, the softmax along the other axis) gives the loss for predicting view from view
A.3 TRAINING DATASET
Our training dataset uses imagery collected by three different sources, each with four channels (bands): red, green, blue, and near-infrared (NIR).
SPOT
Satellite Pour l'Observation de la Terre (Satellite for observation of Earth) Consists of world-wide imagery at a pansharpened resolution of 150 em. Our imagery comes from the OneAtlas basemap published by Airbus, derived from images acquired by SPOT-6/7 satellites.
NAIP National Agriculture Imagery Program
Consists of roughly yearly aerial photography at a 100 cm resolution over each of the states in the contiguous United States. From the USDA's Farm Service Agency.
PHR Pléiades High Resolution
Data acquired from the OneAtlas basemap consisting of mcasurements from two satellites, Pléiades-Hi 1A and HH IB, at pansharpened resolution of 50 cm and limited mostly to urban areas.
Although NAIP, for instance, a project utilizing multiple different aircraft- cameras, we treat them all as one data source for the purposes of this paper. We similarly treat the imagery from the SPOT and PHR satellite constellations as single data sources.
The intersection of the coverage of these satellites gives us area of about 532,000 km2, which we tile into images at a resolution 50 em and size 128-by-128 pixels (64-by-64 meters). This gives 130 million possible distinct locations over the contiguous United States. We save just under 72 million of these as triplets of measurements, one image per sensor type, and randomized the collection date that our dataset would capture persistent rather than transient features the Earth. The upstream loss during training saturated at 47 million images, thus we stopped training and saved the model at this point.
13

---

Under review as a conference paper at ICLR 2020
A.4 OPEN STREET MAP CLASSIFICATION DATASET
The OSM classification experiments use 12 labels:
man_made=bri.ge
2. man_nade-breakwwtee
3. building=fazm
4. power=substatiion
5. leisure=stadium
6. Leisure-golf_course
7. waterway=dam
8. landuse=quarry
9. landuse=farmland
10. landuse=forest
11. natural=water
12. natural-bare_rocc
All of these features are polygons. To find location on each feature which is still representative of the label when cropped, we find the approximate pole of inaccessibility, which is the point farthest from the exterior of the shape (Agafonkin) 2016). Then, images are upsampled to 50 cm on-the-gnt the-ground resolution and further cropped to extent of 128-by- 128 pixels, for a total spatial extent of square of side lengths of 64 m.
14


Reviews:
[{"review_id": "SylM88mHoB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to create unsupervised representations of remote sensing images. The essential idea is to enforce similarity between representations of multiple views obtained by subsetting channels from multiple co-terminus sensor outputs.  This is implemented by training with the InfoNCE loss on high-level features (last two layers of a ResNet 50) of two views of the same image and multiple view of other images which are passed through the same weight-shared network. The evaluation is based on a custom task of classifying OSM labels. The results show that (a) the learned representation (up to 3 chosen channels) outperform a pre-trained ImageNet fine-tuned for the classification task, (b) multiple low resolution sensors can outperform a higher resolution sensor, and (c) visualisation of images indicates that stable high-level representations across sensors. \n\nThe paper makes a worthwhile contribution in introducing contrastive methods to satellite imagery - a domain well suited for analysing how contrastive methods work and also rich in applications of sensor fusion/augmentation for remote sensing applications. The experiments indicate that this is a promising direction, and the authors have helpfully open-sourced the dataset they used. The paper, however, is short on several accounts. Primarily, the experimental evaluation is sparse in detail and rigour. In addition, the paper makes unsubstantiated claims to “argue out” certain baselines from being compared. Finally, the results indicate modest improvements on a custom task, and thus remain inconclusive. \n\nThe paper bases all experimental evaluation and conclusions thereof on the custom task of classifying 12 classes of distinctive objects on 8400 images obtained from OpenStreetMap (OSM). These include diverse categories spanning generic bodies (water, forest), specific structures (substation, bridge), similar items (farm, farmland). The paper does not discuss why this is a good task (is it challenging, is it representative of remote sensing applications, are these most frequent OSM classes). Indeed, the experimental results provide no intuition on the classification task (eg. confusion matrix is essential especially given the middling accuracy of about 0.5). \n\nThe paper makes a strong claim that the “distributional hypothesis” (of expecting that spatial or temporal similarity reflect in representation similarity) does not apply to satellite imagery because “remote sensing imagery … can change abruptly between adjacent patches”. No evidence is provided of what I believe is an unintuitive claim. Indeed, existing work [1] (as cited in the paper) uses this hypothesis to create representations for satellite images. Glaringly, having made the claim, the paper does not compare proposed approach with the representations computed in [1].  The paper also makes the claim that computing representation loss should be done with high-level features rather than individual pixels. Again, this is not substantiated as the paper does not compare the proposed methods with simple auto-encoder baselines. In the absence of these baselines, the results of this paper on a custom data-set remain inconclusive. \n\nThe evaluation leaves out several other expected experiments. A few suggested conditions for ablation tests are listed:\n(a) Different augmentation across channels\n(b) Different values of \\lambda_L (only an extreme case of last two layers has been presented)\n(c) At least one other CNN-backbone, perhaps a deeper ResNet\n(d) Different orderings of introducing the channels (not only by low or high resolution ordering)\n(e) Different sizes of the tiles for learning representations (curiously the paper does not mention the size of the tiles as used currently)\n\nFinally, and more broadly, the proposed approach aims to compensate the lack of supervised labels by exploiting redundancy across multiple sensors/channels. In the chosen setup, this redundancy is very high as 4 different RGBI co-terminus sensors are chosen, thereby requiring augmenting and drop-out to simulate some variation. A more realistic or challenging setup would be required to evaluate the underlying ideas.\n\n[1] Jean, Neal, et al. \"Tile2Vec: Unsupervised representation learning for spatially distributed data.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\n--\n\nUpdate in response to rebuttal: \nThe authors agreed to most of the points raised, but provided no clear suggestions in addressing them. I re-emphasise that the empirical results on a single custom dataset based on OSM is limited. The author response to this remains vague. Further none of the reasonable baselines have been compared against, a point which the authors ignored in the rebuttal. In light of this, rating remains the same. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e3XLhVtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an unsupervised method to embed (remotely sensed) image patches such that they are clustered by high-level \"image content\" rather than by low-level pixel statistics. The argument is that in remote sensing one often has access only to a subset of all possible images/channels, so it would be useful to get the same embedding for each subset depicting the same scene. The basic idea is to generate, during training, two versions of the same patch by randomly dropping out channels, and to penalise the difference between the two corresponding embeddings, using the InfoNCE loss. The latter ensures that the patch remains \"recognisable\" among a set of random non-matching patches, so as to avoid learning a trivial embedding that is the same for any input.\n\nThe high-level goal, to train an encoder that always gives a similar encoding (and consequently similar downstream results) for a given patch, independent of the sensor channels used, is certainly appealing. It is also true that one would like to do this with unsupervised learning or transfer learning, since labelled data is scarce in remote sensing. Unfortunately, in my understanding the paper does not make much progress towards that goal.\n\nNovelty is limited, the work is a variation of the CMC method. Yes, it splits channels randomly, not by views; and uses a Siamese architecture, not separate encoders; which it can afford to do since there are no viewpoint differences. And it computes the loss at more than one layer - which is a fairly trivial and popular thing to do. Moreover it is not well justified in this case, since all that matters in a feed-forward architecture is that the final embeddings are similar. Constraining lower layers might empirically help - in particular speed up learning - but conceptually there is no reason to reduce the lower layers' capacity to somehow turn different inputs into the same output. If they are really very different (think RADAR images) this could in principle even hurt. \n\nBoth in the discussion and in the experiments, I miss crucial and obvious baselines. First, it does in my view not make sense to use pre-trained ImageNet features as the only baseline. Yes, they are known to work surprisingly well also for remote sensing data - but, unsurprisingly, not nearly as well as pre-training on similar remote sensing images. A more natural baseline would be to pre-train on the same type of imagery, with a proxy task for which it is easy to get labels automatically  - actually the paper implicitly suggests that labels from OpenStreetMap might be a good candidate. Beyond reducing the domain gap, this also has the advantage that one is not limited to 3 channels, and that one can use an architecture suitable for the task (e.g., standard architecture with a lot of pooling are known to not perform well for semantic segmentation of remote sensing data). Labels for supervised learning might in many remote sensing tasks be scarce, but to get any old labels and pre-train a reasonable proxy task is still easy, as map data is abundant.\n\nPerhaps even more important is another natural baseline. In spirit, the proposed method is closer to a generative embedding, since the supervision signal is not a discriminative task, but rather encourages embeddings to stay unique for each patch, and hence in principle fully decodable. Arguably, the most obvious way to achieve the same thing is a Siamese pair of auto-encoders for the two training patches, with an additional loss on the difference between the latent representations. Without that baseline it is impossible to tell whether the InfoNCE loss does a good job - in some sense it feels like making the task harder if one demands not only that a patch of water can be decoded from the embedding, but that it should be distinguishable from any other patch of water that resembles it even in raw pixel space.\n\nThe data used in the experiments is somewhat unconvincing, as it is really the opposite of diverse, multi-modal remote sensing channels. The same four R-G-B-NIR channels from sensors with not so different resolutions. The claim that the method homes in on high-level similarity and overcomes low-level radiometric differences would be much more credible if one uses short-wave infrared, thermal, perhaps even Radar images. Moreover, the gains are disappointingly small: even in the best case of 12 channels, the improvement over ImageNet pre-training is roughly from 4.5/10 correct neighbours to 5/10 correct neighbours. The even lower gains with fewer channels underline the need to have a pre-trained baseline without a massive domain gap. The classification accuracy of kNN does improve with the proposed embedding (at least for some combinations - for Pleiades RGB it is the same), but that experiment is unnatural. If one has the labels to do kNN, one might as well use them to train a proper classifier, i.e., add a couple of fully connected layers and fine-tune in a supervised fashion.\n\nA number of technical details in the experiments remain unclear. Section 4.2 suggests the channels in Fig. 4 were added in a fixed order - why? There are many potentially viable combinations of, say, 4 channels, not just one. Furthermore, Fig. 5 suggests to me that the learned similarities are in fact not \"high-level\", but (up to rotation invariance) fairly low-level texture properties like \"homogeneous green patch\", \"bright straight line on dark background\", etc. I am not sure what suggests to the authors that there is much sensor invariance. Again, channels with really different wavelength would be more convincing here.\n\nIn terms of presentation, the paper repeatedly makes strong, but unsupported claims about the special properties of remote sensing images.  It makes no sense to state that standard pictures often have \"one subject\" - ImageNet does, but a look at standard datasets like Cityscapes or DepthInTheWild shows that it is not true in general. In the same vein, I don't see why the content changes more unpredictably at the same location across time - arguably the rate of change is much lower, because only large-scale changes impact the image, and because there aren't many dynamic occluders. And the \"distributional hypothesis\" certainly also holds for remote sensing images, just at different scale - otherwise people would not use all sorts of smoothness priors, super-pixel segmentations, etc. when working with them. And I do not understand why the adjacent-patch approach of CPC needs nearby patches to be similar - in my view it only requires that nearby patches are not conditionally independent, so basically any type of patterns.\n\nA small detail: some references, while not wrong, seem rather arbitrary. E.g., (Wu 2018) is a little-known random example. Why not use earlier, more standard references - for semantic segmentation of remote sensing data with deep learning one could for example think of (Maggiori, Audebert, Marmanis, Kampffmeyer, Sherrah, ...).\n\nOverall, while the paper brings up a valid question, it does not give a convincing answer. The justification of the method and its novelty is to some degree contrived, the dataset is ill-suited to really prove the point, and the natural baselines are missing.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xNtSz1qr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for representation learning on remote sensing data/satellite imagery inspired by recent unsupervised contrastive multiview representation learning methods (CPC, DIM, CMC). The method relies on the InfoNCE objective to contrast two different views of the data obtained by randomly cropping image patches, color jittering, channel dropping etc. The proposed method is compared to an ImageNet pretrained network in a classification task based on OpenStreetMap (OSM) features and show to outperform the ImageNet pretrained classifier. \n\nThe paper is well-written and the method is clearly explained. To my knowledge there is little prior work on unsupervised/self-supervised learning on remote sensing/satellite data and this data suits the contrastive/multiview framework very well. In that regard I appreciate the direction explored by the paper.\n\nHere are some questions and concerns:\n\n- I think the paper is lacking some important details, for example how are the unsupervised and ImageNet-pretrained representations transferred? Fine-tuning, or learning a classifier on top of the frozen representation? In the case of fine-tuning, the ImageNet baseline could be extended to more channels.\n\n- As an additional baseline, how does a network trained from scratch on the available labeled training data perform?\n\n- The authors evaluate on a single data set, that seems to not have been used previously. To make the evaluation more solid it would be good to compare on other data sets, for example on EuroSAT [1], and with other, possibly supervised classification methods, see, e.g., [1, 2].\n\n- Did you do ablations on augmentations used? For example, is zeroing out channels more effective than copying other channels instead?\n\n- Both forward and backward prediction losses are used, but it seems that the loss is symmetric. Does adding the backward prediction loss really help?\n\nOverall, I like the direction the paper is exploring, but I think it would greatly benefit from adding detail on the outlined aspects and extending the evaluation.\n\n\n[1] Helber P, Bischke B, Dengel A, Borth D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2019 Jun 14;12(7):2217-26.\n[2] Kussul N, Lavreniuk M, Skakun S, Shelestov A. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters. 2017 Mar 31;14(5):778-82.\n\n\n\n---\nUpdate after the rebuttal:\nThanks to the authors for their detailed response. While the authors agree that some points should be improved, no attempts were made to actually strengthen the method in a revision. In particular, I do think evaluating the proposed method on prior benchmarks that were mostly used in the context of supervised methods (such as EuroSAT) is very important. Also, I still think comparison to from scratch training is crucial. Both comparisons are common practice in prior work such as CPC, DIM, and CMC (specifically, ImageNet). I therefore do not increase my rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method for learning representations of remote sensing images from multiple views. The main ideas is to use the InfoNCE loss to learn from multiple views of the data. 

The reviewers had a few concerns about this work which were not adequately addressed by the authors. I have summarised these below and would strongly recommend that the authors address these in subsequent submissions:

1) Experiments on a single dataset and a very specific task: Authors should present a more convincing argument about why the chosen dataset and task are challenging and important to demonstrate the main ideas presented in their work. Further, they should also report results on additional datasets suggested by the reviewers. 
2) Comparisons with existing works: The reviewers suggested several existing works for comparison. The authors agreed that these were relevant and important but haven't done this comparison yet. Without such a comparison it is hard to evaluate the main contributions of this work. 

Based on the above objections raised by the reviewers, I recommend that the paper should not be accepted.
Decision: Reject