Title: REPRESENTATION LEARNING FOR REMOTE SENSING: AN UNSUPERVISED SENSOR FUSION APPROACH

Abstract: In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent selfsupervised learning approaches are ill-suited to the remote sensing domain. In addition, most remote sensing applications currently use only a small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose a new self-supervised training objective, Contrastive Sensor Fusion, which exploits coterminous data from multiple sources to learn useful representations of every possible combination of those sources. This method uses information common across multiple sensors and bands by training a single model to produce a representation that remains similar when any subset of its input channels is used. Using a dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors. These representations outperform fully supervised ImageNet weights on a remote sensing classification task and improve as more sensors are fused. Our code is available at https://storage.cloud.google.

1. INTRODUCTION:

Remote sensing data has become broadly available at the petabyte scale, offering unprecedented visibility into natural and human activity across the Earth. Many techniques have been recently developed for applying this data with machine learning to solve geospatial tasks like semantic segmentation (Audebert et al., 2016) , (Kampffmeyer et al., 2016) , broad-area search (Keisler et al., 2019) , and classification (Maggiori et al., 2016) , (Sherrah, 2016) .

Due to the complexity and visuospatial nature of solving problems with aerial imagery, it is natural to use deep convolutional neural networks, but CNNs typically require large amounts of labeled data to achieve good performance. In remote sensing, these labels are usually scarce and hard to obtain; semantic segmentation requires boundaries to be labeled at single-pixel precision. A modern approach to the problem of data scarcity is semi-supervised learning, which uses unlabeled data to ease the task of learning from small amounts of labeled data. This approach is particularly well-suited to remote sensing because of the amount of unlabeled data available.

Figure 2 : Coterminous remote sensing imagery from three different sensors: Airbus SPOT, NAIP (visualized here in near-infrared, red, and green), and Airbus Pléiades (see Appendix A.3 for details). As seen here, images contain many small components (roads, buildings, structures, trees) and adjacent locations can look completely different (e.g., the transition from buildings to grass). We leverage these multiple views to generate representations with any subset of available sensors or channels.

While most self-supervised and unsupervised image analysis techniques focus on natural imagery, remote sensing differs in several critical ways, requiring a different approach. Where pictures taken by a human photographer often have one or few subjects, remote sensing images like those in Figure 2 contain numerous objects such as buildings, trees, or factories. Additionally, the important content can change unpredictably within just a few pixels or between images at the same location from different times. Multiple satellite and aerial imaging platforms capture images of the same locations on earth with a wide variety of resolutions, spectral bands (channels), and revisit rates, such that any specific problem can require a different combination of sensor inputs (Reiche et al., 2018; Rustowicz et al., 2019) .

Recent research in semi-supervised learning has led to a wealth of methods that achieve success on problems like classifying natural images (van den Oord et al., 2018; Tian et al., 2019; Hénaff et al., 2019) and understanding language (Mikolov et al., 2013; Devlin et al., 2018) . These approaches almost universally rely on the "distributional hypothesis": the property that parts of the data that are close in time or space are similar. Previous work (Jean et al., 2019) has learned representations for overhead imagery using the distributional hypothesis. However, we argue that it is less applicable in remote sensing, due to the aforementioned differences between overhead imagery and these domains. The distributional hypothesis still applies, but the scale of correlated patches is smaller, making it difficult to harvest scenes for related patches. We therefore modify these techniques to build on the intuition that similar layouts of objects on the ground should have similar representations, regardless of the combination of sensors.

In this paper, we use this idea to develop a method of learning representations for overhead imagery, which we call Contrastive Sensor Fusion (CSF). We train a single model, unsupervised, to produce a representation of a scene given any subset of the sensors used during training over that scene. During training, we form two "views" of each scene from a small random subset of the available sensor channels (channel/sensor dropout). Then both views are encoded with the same network and the resulting representations are compared, using a contrastive loss to encourage the two views to have similar representations. This model is trained on a ~20 TB dataset consisting of 47 million scenes, each consisting of four bands for each of three different sensors over that scene. We perform several experiments, showing that the unsupervised encoder learns to produce representations of many different combinations of sensors that separate images into semantically-meaningful classes and perform well on real problems in the semi-supervised setting. The representations are transferred as-is, with no fine-tuning step or stacking additional models.

2. BACKGROUND:

Contrastive methods train models by using the representation of one observation to predict the representation of a different but related observation. Examples include pairing frames from the same video (Wang and Gupta, 2015) or words from the same local context (Mikolov et al., 2013) . The recent method Contrastive Predictive Coding (CPC) uses a representation of the current time step in a sequence to predict the same encoder's representation of a future time step (van den Oord et al., 2018; Hénaff et al., 2019) . CPC has been shown to learn expressive representations of natural imagery by using the representation of one image sub-patch to predict the representation of adjacent sub-patches, contrasting against patches drawn from other images. However, this adjacent-patch technique fails for remote sensing imagery, which can change abruptly between adjacent patches.

Contrastive Multiview Coding (CMC, Tian et al. (2019) ) is a related approach that compares multiple views of the same context. Unlike CPC, CMC does not assume that representations of image patches should be stable in space and benefits from having many views of a scene, making it applicable to remote sensing. On the other hand, CMC relies on a fixed partitioning of data sources, forcing practitioners to decide ahead of time which sensor combinations to fuse. It also trains a separate model for each view, limiting how much information can be shared between views.

We use the InfoNCE loss introduced by van den Oord et al. (2018) . Intuitively, its role is to train the encoder to make representations of the same scene similar across views. Computing the loss at the level of representations instead of pixels sidesteps issues with reconstruction-based loss functions like autoencoders use, which are dominated by low-level properties of pixel space like overall brightness instead of high-level information; however, directly predicting one representation from another leads to trivial solutions, such as representing every scene with the zero vector. Instead, contrastive losses like InfoNCE ask the network to classify which representation among a set including many "noise examples" belongs to the same scene.

We borrow a number of ideas directly from Deep InfoMax (Hjelm et al., 2019) and its successor, Augmented Multiscale Deep InfoMax (AMDIM) (Bachman et al., 2019) . These methods learn by contrasting representations at various layers and spatial locations of the same network. AMDIM increases the difficulty of this task by independently augmenting each view. Computing the loss at multiple levels and sharing the network weights between views also increase supervisory signal to the model. As with CPC, however, AMDIM assumes that distinct patches from the same image are closely related, which is problematic in remote sensing.

Finally, we tested the non-contrastive method of Split-Brain Autoencoders (Zhang et al., 2017) , but found them to be less effective than contrastive methods. Other pixel-space methods (e.g. Singh et al. 2018 ) exist for remote sensing representation learning, but these face the issues with reconstructionbased losses mentioned above.

3. CONTRASTIVE SENSOR FUSION:

We introduce Contrastive Sensor Fusion (CSF, Figure 3 ), a technique for learning unsupervised representations of every combination of its input sensors. CSF learns by creating two views of each location by randomly sampling two sensor combinations, encoding both views with the same CNN (sharing weights), and comparing representations across locations with a contrastive loss that aligns representations of the same location regardless of the input sensor combination.

Our method can be seen as an extension of Contrastive Multiview Coding, with three critical differences. First, instead of dividing the input channels into fixed groups during training, CSF drops out channels at random, forcing the network to learn to encode every combination of channels. Second, CSF uses a "Siamese network" (Bromley et al., 1993) training scheme, sharing weights across the encoders of every sensor combination rather than training a separate encoder for each view. Third, CSF computes a contrastive loss at multiple layers of representation, which helps to learn localized representations. In a sense, CSF can be seen as a computationally tractable way of training an ensemble of all exponentially-many1 splits of views in CMC at once, with weight sharing between each encoder and loss at multiple levels. This is analogous to the extension of NADE models (Larochelle and Murray, 2011) to ensembles of arbitrary ordering (Uria et al., 2014) .

3.1. LEARNING FROM MULTIPLE VIEWS:

Like CMC, CSF learns by contrasting representations of multiple views of the same input. Given a set of sensor looks X over a scene, we aim to create two views V (1) (X), V (2) (X) that contain the same high-level information (e.g. object identities and location, scene type, topography) but differ as much as possible at the pixel level. In order to match representations of the same scene, the model must encode both views in such a way that the scene remains distinctive. If the views differ enough, this forces the encoder to capture high-level information about the scene, since that is the most salient remaining information common to both views, while discarding nuisance factors like lighting, resolution, and differences between sensors. To create a view V (X), we randomly set a large fraction of the channels in X to zero, which effectively fuses a different combination of sensor channels for each example. With a channel dropout rate of p, we scale the remaining channels by 1 1-p as in Srivastava et al. (2014) . Similarly, at inference time we run the encoder with any subset of the channels it was trained with, replacing the others with zeros and scaling up remaining channels by the same factor as was used during training.

We augment images by randomly cropping a small number of pixels from X, applying random flips and rotations, and jittering the brightness and contrast of each remaining channel; these increase the difficulty of the contrastive task and improve the robustness of the representation to nuisances.

Because CSF uses a single network for multiple sensors with potentially different resolutions, numbers of channels, and even modalities, care must be taken to design an encoder which can fuse them effectively. In our experiments, we used three sources of optical imagery with slightly different pixel resolutions (see A.3). We bilinearly upsampled2 the two lower-resolution sources to match the highest resolution, then concatenated the sensors' channels depth-wise such that each pixel fed to the first convolution covers the same physical area.

3.2. INFONCE LOSS:

CSF compares representations at multiple levels of the encoder using the InfoNCE loss. As van den Oord et al. (2018) show, InfoNCE maximizes the mutual information (MI) between representations. We design our cross-view augmentation to destroy MI between views except for the high-level understanding, so maximizing the MI between representations directly trains the encoder to learn robust and expressive representations of its input.

At each of several layers L in the encoder, a contrastive loss is computed as follows. Let V be a stochastic function which maps a set of sensor looks X to a view, and let Z L be a function which maps a view to the encoder's layer-L representation of that view. We apply V to X twice to produce two views and encode both into layer-L representations, which we denote by Z

(1)

L ≡ Z L (V (1) (X)) and Z (2) L ≡ Z L (V (2) (X)).

Let φ be a scoring function on this representation space. φ plays the role of contrastive learning's predictive model; given two views, it produces the logit of the two views belonging to the same scene. Given a collection of n other scenes X noise = { X1 , . . . Xn = X} and their layer-L representations Z L,noise = {Z L (V ( X1 )), . . . Z L (V ( Xn ))} used as noise examples to classify among, the loss at layer L for example X is the InfoNCE loss presented in CPC,

EQUATION

L , Z

EQUATION

This is the loss function for predicting view 2 from view 1, which we refer to as the "forward" loss. Because the problem is symmetric (we are comparing representations of the same encoder run over two identically-created views), we can use the same procedure and set of negatives to predict view 1 from view 2, the "backward" loss. The total contrastive loss for layer L, L L , is the sum of the two, and the total loss for example X is

L tot = L λ L L L

where λ L weights the contrastive loss at layer L. Following the intuition that two views of a scene are most similar in abstract high-level ways, we use nonzero weight on only last layers of the last two residual stacks, and weigh the last layer twice as heavily.

In all experiments, we use φ(Z

EQUATION

EQUATION

L and draw negative examples from other scenes in the batch. This allows very efficient computation of the contrastive loss. See A.2 for more details.

Computing contrastive loss at multiple layers is efficient, since it requires applying the encoder only once per view, and is beneficial for a number of reasons. Since modern CNNs use multiple layers of pooling, higher layers in the encoder are naturally more translation invariant and so multi-layer loss helps CSF learn well-localized features useful for segmentation. Additionally, the extra supervisory signal helps earlier layers learn more quickly.

3.3. TRAINING:

During training, both copies of the encoder share weights. As argued above, this trains a single model which can function with any of its input channels missing. In addition, weight-sharing makes the encoder more parameter-efficient. This can lead to information sharing across sensors; for example, the network may learn to combine the blue channel of one sensor with the green channel of another early in the network, since the two bands often provide similar information.

In contrast to CPC and AMDIM, computing the loss does not require masking the encoder or using an encoder with a restricted receptive field size. CSF learns representations of whole images and contrasts representations from distinct scenes rather than distinct parts of the same image, so a standard architecture can be used for the encoder. Following van den Oord et al. ( 2018), we use a ResNet encoder (He et al., 2015) . We choose ResNet-50 instead of ResNet-101 for our experiments because its smaller memory footprint enables us to use a larger batch size. Larger batch sizes result in more accurate loss since the negative examples for NCE are the other scenes in the batch. We train on a single TPU v2 with a batch size of 2048.

During training, we schedule the channel dropout rate to increase linearly from 0 to 0.66 over the first 8000 batches. This is a form of curriculum learning (Bengio et al., 2009) where the contrastive task starts in an easier setting where more band are retained, and gets gradually harder throughout training; we observed faster convergence with this channel dropout schedule. We also linearly warm up the learning rate over the first 3000 batches (Goyal et al. 2017) . Other hyperparameters are kept constant; when making each view, we randomly crop away 32 pixels and randomly jitter the brightness and contrast of each retained band by at most 25%.

4. EXPERIMENTS:

We test this method by training a CSF network on 47 million image triples from three imaging platforms: Airbus SPOT, USDA NAIP, and Airbus Pléiades, with pixel resolutions of 150 cm, 100 cm, and 50 cm, respectively. Each has four channels: red, green, blue, and near-infrared (NIR). See Appendix A.3 for details. To visualize the feature space and evaluate the learned representations, we built a dataset of 8400 samples based on OpenStreetMap (OSM) features from 12 classes of distinctive objects; example images are shown in Figure 1 and Figure 5 . Each object is seen in all three sensors used to train the encoder. See Appendix A.4 for details. We test the performance of the trained CSF network with several experiments. For each scene in the OSM-based dataset, we produce representations from a ResNet50V2 encoder initialized with weights either learned through CSF, from fully-supervised ImageNet classification, or set randomly. We use various combinations of input channels and sensors (although ImageNet weights are limited to 3 input channels). We reduce the dimensions of the generated representations by principal component analysis (PCA). In Figure 1 , we visualize the CSF representations by embedding the 200 principal components of the features obtained from 12 input channels into two dimensions with t-SNE. Next, we measure the quality of clustering by identifying the k-nearest neighbors to each point in representation space (2048-component PCA) and count the fraction that belong to the OSM class of that point. We also test classification performance by training a k-nearest neighbors (KNN) classifier with k = 10 and leave-one-out cross validation on every point and measure the classification accuracy (Table 1 ). We report the average of each metric over the OSM dataset.

Experiment results from several encoder-channel combinations are presented in Figure 4 as a function of non-zero channels (panel 1) or choice of k (panel 2). Because ImageNet weights only allow three or fewer input bands, we test ImageNet weights applied to either the highest resolution (Pléiades) or lowest resolution (SPOT) RGB imagery. k-NN classifier accuracy is presented in Table 1 .

4.1. REPRESENTATION QUALITY:

First, we evaluate the quality of learned representations by considering how well they cluster scenes by OpenStreetMap object category. Plotting the resulting points colored by OSM label (Figure 1 ), we observe that most classes form distinctive clusters. Neither the encoder nor the dimensionalityreduction algorithms have access to these labels; therefore, we can observe directly that CSF clusters images into semantically meaningful categories. The clustering quality is quantified in Figure 4 and Table 1 . In every case for both nearest-neighbor class and classification accuracy, representations learned through CSF outperform those learned with all ImageNet labels (Figure 4 ). We emphasize that training the ImageNet encoder required 14 million labels whereas the CSF encoder was trained purely unsupervised. Though overhead imagery differs significantly from ImageNet's labeled natural images, out-performing ImageNet is not trivial; previous work (Singh et al., 2018) demonstrates that supervised ImageNet pretraining is a strong baseline for representations of remote sensing imagery3 .

Furthermore, as we add channels the unsupervised CSF encoder continues to improve. We also investigated all other orders of adding sensors, and chose to display the order which resulted in lowest scores first; that is, adding the lower resolution data before the higher resolution data. This demonstrates a limitation of transfer learning from datasets like ImageNet: the resulting weights can only be used with channels present in the dataset. Most labeled data is RGB, so supervised transfer learning prevents us from taking advantage of multiple co-registered data sources.

4.1.1. MOTIVATION:

This OSM classification task demands that a representation understand the kinds of objects and relationships useful for a broad range of remote sensing applications. It demonstrates that the features learned, whether high-level or low-level, are features which allow discriminating between semantically distinct sets of input images. Intuitively, representations which do not cluster well among semantic classes require learning more complex transformations to separate the classes.

Fine-tuning experiments have the potential to demonstrate this fact, and we note this as a future avenue for research. However, we believe that special care will need to be made to show that features continue to generalize to tasks where labeled data is not available for fine-tuning.

4.2. SENSOR FUSION:

Next, we investigate our main claim: that CSF learns to fuse different sensors with minimal degradation in feature quality when only a subset of sensors are available. We observe that clustering quality increases monotonically as channels are added (first panel of Figure 4 ). Though the biggest gains come from using high-resolution imagery, fusing multiple low-resolution sensors noticeably improves representations above having a single higher-resolution sensor. We were surprised to discover that fusing SPOT and NAIP imagery with CSF results in higher performance than CSF representations of (higher-resolution) Pléiades imagery, as seen by comparing the light-purple point at 8 input channels to dark-purple points at ≤ 4 input channels. In the second panel of Figure 4 , the fully fused 12-channel CSF network outperforms 3-band ImageNet at all values of k.

In every case, performance increases as more channels added, regardless of order. This demonstrates that CSF is effectively combining the information present across multiple sensors, and that CSF has learned a good representation for each of its possible views. We expect that as non-optical data sources are added, CSF will outperform transfer learning from ImageNet to an even greater degree. This result suggests that CSF has much to gain from adding more sensor views than we try in this work. Finally, in Figure 5 we demonstrate that CSF learns high-level representations that are stable across sensors. As before, we identify the principal components of the encoder's representaitons of the 12-channel OSM dataset (which, as Szegedy et al. (2014) note, are not distinguished from individual channels). Then, we encode each sensor's views individually and visualize the images which maximally activate those linear combinations in Figure 5 . We find that each direction visualized corresponds to the same high-level concept, regardless of which sensor produced it. This suggests that CSF learns to fuse multiple sensors in early layers into disentangled and sensor-invariant features which will be easy for later layers and transfer tasks to use due to their directional consistency.

5. CONCLUSIONS AND FURTHER WORK:

In this work we present Contrastive Sensor Fusion (CSF), a new self-supervised training objective to learn fused representations of multiple overhead image sources. CSF uses a contrastive loss to train an encoder that can produce a shared representation from any subset of available channels across multiple sensors. Using a dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors, out-performing fully-supervised ImageNet weights which required 14 million labels to train. We show through experiments that the network is successfully fusing multiple sensor information into representations that improve with additional views. We also show that, through sensor fusion, multiple low resolution sensors can outperform a single higher resolution sensor. While this work considered only optical sensors with similar resolutions, remote sensing practitioners frequently use a varity of sensors including non-optical and hyperspectral imagery with many channels. We expect multi-sensor representations of these to outperform supervised learning transferred from natural imagery to an even greater degree than demonstrated here.

both of shape (B, N 2 ) where B is our batch size and N 2 is the flattened size of each spatially down-sampled feature map, we define their similarity matrix S as

S = Z (1) × Z (2) T =     Z (1) 1,: • Z (2) 1,: • • • Z (1)

B,: • Z

(2) 1,:

. . . . . .

Z:

(1) 1,: • Z

(2) B,:

Z (1) B,: • Z (2) B,:     (A.1)

Each element of this matrix measures, per batch, how similar (under φ) elements' features generated from view 1 are to all other elements' features generated from view 2. Thus, the diagonal elements represent similarity of features from samples of the same origin but different dropout, cropping, and color and contrast jitter. We measure the neural net's ability to associate features of identical origin with one another, rather than negatives in the same batch, treating this as a classification problem.

We take the log-softmax along each axis to find predicted associations from view 1 to view 2 and vice-versa.

The contrastive loss at this layer is computed as the mean of diagonal elements of the log-softmax of both the similarity matrix and its transpose The diagonal elements of the log-softmax of the similarity matrix give the loss for predicting view 2 from view 1, and performing the same process with the similarity matrix's transpose (equivalently, the softmax along the other axis) gives the loss for predicting view 1 from view 2.

A.3 TRAINING DATASET:

Our training dataset uses imagery collected by three different sources, each with four channels (bands): red, green, blue, and near-infrared (NIR).

SPOT Satellite Pour l'Observation de la Terre (Satellite for observation of Earth) Consists of world-wide imagery at a pansharpened resolution of 150 cm. Our imagery comes from the OneAtlas basemap published by Airbus, derived from images acquired by SPOT-6/7 satellites.

NAIP National Agriculture Imagery Program:

Consists of roughly yearly aerial photography at a 100 cm resolution over each of the states in the contiguous United States. From the USDA's Farm Service Agency.

PHR Pléiades High Resolution:

Data acquired from the OneAtlas basemap consisting of measurements from two satellites, Pléiades-HR 1A and Pléiades-HR 1B, at a pansharpened resolution of 50 cm and limited mostly to urban areas.

Although NAIP, for instance, is a project utilizing multiple different aircraft-based cameras, we treat them all as one data source for the purposes of this paper. We similarly treat the imagery from the SPOT and PHR satellite constellations as single data sources.

The intersection of the coverage of these satellites gives us an area of about 532,000 km 2 , which we tile into images at a resolution of 50 cm and of size 128-by-128 pixels (64-by-64 meters). This gives 130 million possible distinct locations over the contiguous United States. We save just under 72 million of these as triplets of measurements, one image per sensor type, and randomized the collection date so that our dataset would capture persistent rather than transient features on the Earth. The upstream loss during training saturated at 47 million images, thus we stopped training and saved the model at this point.

:

Given n sensors, there are

n -1 possible sensor combinations (counting all subsets and discounting the empty set), and so as n grows, training a separate model for each is infeasible.

For sensors of very different resolutions, training a separate head for each sensor before combining the representations may be appropriate.

We compare results on a multiclass classification problem, so we expect supervised training with ImageNet labels to be particularly effective here relative to e.g. a segmentation task.

Figures and Tables:

Figure 1: Learned representations of out-of-sample image scenes, visualized with PCA followed by t-SNE and colored by OpenStreetMap category. Without any labels, Contrastive Sensor Fusion has learned a representation that groups remote sensing images into semantically meaningful categories.

Figure4: We compare the clustering of features based on OSM class using a nearest neighbor metric. The plots show the fraction of same-class neighbors for each point (k = 10) as input channels are added (left), and the fraction of same-class neighbors as a function of k (right). One, two, and three-channel experiments always use a single sensor, taking the red band only, the red and green bands, and the RGB bands respectively. Our features outperform ImageNet's in this unsupervised clustering metric and improve when multiple sensors are fused.

Figure5: For each of the first three principal components of the 12-channel CSF representation space, we show 10 images from each single sensor (with inputs for the other two sensor zeroed) that maximally activate these directions. These principal components of representation space represent contain concepts (fields, bridges, and bare ground / concrete) stable across sensor combinations.

contrastive (Z (1) , Z (2) ) = 1 B B b diag (log-softmax(S)) + diag log-softmax S T b (A.2)

Figure 3: Contrastive Sensor Fusion architecture during training. Weights are shared across encoder copies. The contrastive loss trains the encoder to represent the same (different) location the same (different) way regardless of sensor/channel combination. The process to create views is explained in more detail in Figure A.6, and the computation of the loss is detailed in Appendix A.2.

Table content: <table><tr><td>View V (1)</td><td/><td/><td/><td>View V (2)</td></tr><tr><td/><td>channel dropout</td><td/><td>channel dropout</td><td/></tr><tr><td/><td>random flips</td><td/><td>random flips</td><td/></tr><tr><td/><td>random rotation</td><td/><td>random rotation</td><td/></tr><tr><td/><td>random cropping</td><td/><td>random cropping</td><td/></tr><tr><td/><td>brightness jitter</td><td/><td>brightness jitter</td><td/></tr><tr><td/><td>contrast jitter</td><td/><td>contrast jitter</td><td/></tr><tr><td>(B, 128, 128, 12)</td><td/><td/><td/><td/></tr><tr><td/><td/><td>Co-located sensor inputs</td><td/><td/></tr><tr><td/><td/><td/><td>Features Z3,2</td><td>ResNet Stack 3</td></tr><tr><td/><td/><td/><td>(B, 65536)</td><td>(B, 8, 8, 1024)</td></tr><tr><td/><td/><td>+</td><td/><td/></tr><tr><td>ResNet Stack 4</td><td>Features Z4,1</td><td>Contrastive Loss</td><td>Features Z4,2</td><td>ResNet Stack 4</td></tr><tr><td>(B, 4, 4, 2048)</td><td>(B, 32768)</td><td>λ4L (Z4,1, Z4,2)</td><td>(B, 32768)</td><td>(B, 4, 4, 2048)</td></tr><tr><td>Encoder copy 1</td><td/><td/><td/><td>Encoder copy 2</td></tr></table>

We compare the representation quality of CSF and ImageNet weights for various sensor/channel combinations by training a KNN classifier on our OSM dataset. This table shows the fraction of 10-nearest neighbors belonging to the same class, and the accuracy of the classifier.

Table content: <table><tr><td>Weights Channels</td><td colspan="2">10-NN same-class (%) Accuracy (%)</td></tr><tr><td>ImageNet Airbus SPOT RGB</td><td>34.69</td><td>47.96</td></tr><tr><td>Airbus Pléiades RGB</td><td>43.34</td><td>57.41</td></tr><tr><td>CSF Airbus SPOT RGB</td><td>39.10</td><td>53.16</td></tr><tr><td>Airbus SPOT RGB + near-IR</td><td>42.31</td><td>55.71</td></tr><tr><td>Airbus Pléiades RGB</td><td>43.40</td><td>57.47</td></tr><tr><td>Airbus Pléiades RGB + near-IR</td><td>45.68</td><td>60.23</td></tr><tr><td>All sensors, RGB + near-IR</td><td>50.22</td><td>64.06</td></tr></table>

Peer review (iclr2020):
The paper proposes an unsupervised method to embed (remotely sensed) image patches such that they are clustered by high-level *image content* rather than by low-level pixel statistics. The argument is that in remote sensing one often has access only to a subset of all possible images/channels, so it would be useful to get the same embedding for each subset depicting the same scene. The basic idea is to generate, during training, two versions of the same patch by randomly dropping out channels, and to penalise the difference between the two corresponding embeddings, using the InfoNCE loss. The latter ensures that the patch remains *recognisable* among a set of random non-matching patches, so as to avoid learning a trivial embedding that is the same for any input. The high-level goal, to train an encoder that always gives a similar encoding (and consequently similar downstream results) for a given patch, independent of the sensor channels used, is certainly appealing. It is also true that one would like to do this with unsupervised learning or transfer learning, since labelled data is scarce in remote sensing. Unfortunately, in my understanding the paper does not make much progress towards that goal. Novelty is limited, the work is a variation of the CMC method. Yes, it splits channels randomly, not by views; and uses a Siamese architecture, not separate encoders; which it can afford to do since there are no viewpoint differences. And it computes the loss at more than one layer - which is a fairly trivial and popular thing to do. Moreover it is not well justified in this case, since all that matters in a feed-forward architecture is that the final embeddings are similar. Constraining lower layers might empirically help - in particular speed up learning - but conceptually there is no reason to reduce the lower layers* capacity to somehow turn different inputs into the same output. If they are really very different (think RADAR images) this could in principle even hurt. Both in the discussion and in the experiments, I miss crucial and obvious baselines. First, it does in my view not make sense to use pre-trained ImageNet features as the only baseline. Yes, they are known to work surprisingly well also for remote sensing data - but, unsurprisingly, not nearly as well as pre-training on similar remote sensing images. A more natural baseline would be to pre-train on the same type of imagery, with a proxy task for which it is easy to get labels automatically - actually the paper implicitly suggests that labels from OpenStreetMap might be a good candidate. Beyond reducing the domain gap, this also has the advantage that one is not limited to 3 channels, and that one can use an architecture suitable for the task (e.g., standard architecture with a lot of pooling are known to not perform well for semantic segmentation of remote sensing data). Labels for supervised learning might in many remote sensing tasks be scarce, but to get any old labels and pre-train a reasonable proxy task is still easy, as map data is abundant. Perhaps even more important is another natural baseline. In spirit, the proposed method is closer to a generative embedding, since the supervision signal is not a discriminative task, but rather encourages embeddings to stay unique for each patch, and hence in principle fully decodable. Arguably, the most obvious way to achieve the same thing is a Siamese pair of auto-encoders for the two training patches, with an additional loss on the difference between the latent representations. Without that baseline it is impossible to tell whether the InfoNCE loss does a good job - in some sense it feels like making the task harder if one demands not only that a patch of water can be decoded from the embedding, but that it should be distinguishable from any other patch of water that resembles it even in raw pixel space. The data used in the experiments is somewhat unconvincing, as it is really the opposite of diverse, multi-modal remote sensing channels. The same four R-G-B-NIR channels from sensors with not so different resolutions. The claim that the method homes in on high-level similarity and overcomes low-level radiometric differences would be much more credible if one uses short-wave infrared, thermal, perhaps even Radar images. Moreover, the gains are disappointingly small: even in the best case of 12 channels, the improvement over ImageNet pre-training is roughly from 4.5/10 correct neighbours to 5/10 correct neighbours. The even lower gains with fewer channels underline the need to have a pre-trained baseline without a massive domain gap. The classification accuracy of kNN does improve with the proposed embedding (at least for some combinations - for Pleiades RGB it is the same), but that experiment is unnatural. If one has the labels to do kNN, one might as well use them to train a proper classifier, i.e., add a couple of fully connected layers and fine-tune in a supervised fashion. A number of technical details in the experiments remain unclear. Section 4.2 suggests the channels in Fig. 4 were added in a fixed order - why? There are many potentially viable combinations of, say, 4 channels, not just one. Furthermore, Fig. 5 suggests to me that the learned similarities are in fact not *high-level*, but (up to rotation invariance) fairly low-level texture properties like *homogeneous green patch*, *bright straight line on dark background*, etc. I am not sure what suggests to the authors that there is much sensor invariance. Again, channels with really different wavelength would be more convincing here. In terms of presentation, the paper repeatedly makes strong, but unsupported claims about the special properties of remote sensing images. It makes no sense to state that standard pictures often have *one subject* - ImageNet does, but a look at standard datasets like Cityscapes or DepthInTheWild shows that it is not true in general. In the same vein, I don*t see why the content changes more unpredictably at the same location across time - arguably the rate of change is much lower, because only large-scale changes impact the image, and because there aren*t many dynamic occluders. And the *distributional hypothesis* certainly also holds for remote sensing images, just at different scale - otherwise people would not use all sorts of smoothness priors, super-pixel segmentations, etc. when working with them. And I do not understand why the adjacent-patch approach of CPC needs nearby patches to be similar - in my view it only requires that nearby patches are not conditionally independent, so basically any type of patterns. A small detail: some references, while not wrong, seem rather arbitrary. E.g., (Wu 2018) is a little-known random example. Why not use earlier, more standard references - for semantic segmentation of remote sensing data with deep learning one could for example think of (Maggiori, Audebert, Marmanis, Kampffmeyer, Sherrah, ...). Overall, while the paper brings up a valid question, it does not give a convincing answer. The justification of the method and its novelty is to some degree contrived, the dataset is ill-suited to really prove the point, and the natural baselines are missing.