In-Domain Representation Learning For Remote Sensing
Authors: Maxim Neumann
Andre Susano Pinto
Xiaohua Zhai
Neil Houlsby
Venue:  (2020)
Keywords: Representation learning
remote sensing

Abstract: Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community. To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form. We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning". The established baselines achieve state-of-the-art performance on these datasets. 


Under review as a conference paper at ICLR 2020
IN-DOMAIN REPRESENTATION LEARNING FOR REMOTE SENSING
Anonymous authors Paper under double-bling review
ABSTRACT
Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community. To address it and to establish base- lines and a common evaluation protocol in this domain, we provide simplified access diverse remote sensing datasets in standardized form. Specifically, we investigate in-domain representation learning to develop generic remote sens- ing representations and explore which characteristics are important for a dataset to be good source for remote sensing representation learning. The established baselines achieve state-of-the-art performance on these datasets using in-domain representations.
INTRODUCTION
Remote sensing via computer vision and transfer learning is an important domain address climate change as outlined by RRlnicc et al. (2019). Among others, research in remote sensing promises to help in solving challenges food security (precision farming), water sustainability, disaster preven- tion wwwdd//landliddss/etttquuuu and monitoring of carbon stocks and fluxes or the air quality.
forecasting), deforestation or wild fire detection, urban planning,
The number of Earth observing satellites is constantly increasing, with currently over 700 satellites monitoring many aspects of the Earth's surface and atmosphere from space, generating terabytes of imagery data every day. However the ground truth data acquisition is costly, usually requiring extensive campaign preparation, people and equipment transportation, and in-field gathering of the characteristics under question.
While there are remote sensing communities working on applying general deep learning methods to remote sensing problems, this domain has received relatively little attention from the representation learning community. Given its importance, it is still in an early development stage in comparison the progress made representation learning on natural and medical images (eg. by Raghu et al. 2019)). )
Some reasons for this are the diversity of data sources (satellite types, data acquisition modes, reso- lutions), the need of domain knowledge and special data processing, and the wide and scattered field of applications. The scarcity of standard recognized benchmark datasets and evaluation frameworks is another.
For long time there were only small labeled remote sensing datasets available. Only recenctly new large-scale datasets (eg. by Zhu et al. (2018); Sumbul et al. (2019)) have been generated for remote sensing problems. However, consistent evaluation framework still missing and the performance is usually reported non-standard splits and with varying metrics, making reproduction and quick research iteration difficult.
To address this, we provide five representative and diverse remote sensing datasets in standardized form for easy reuse. In particular, we explore the importance of in-domaín representation learning for remote sensing at various data sizes and establish new state-of- ofate-of-the-art bascline results. The main goal of this work is to develop general remote sensing representations that can be applied by researchers to other unseen remote sensing tasks.

---

Under review as a conference paper at ICLR 2020
By providing these standardized datasets, common problem definition and baselines, hope this work will simplify and enable faster iteration of research on remote sensing and inspire general representation learning experts to test their newest methods in this critical domain.
In summary, the main contributions of this work are as follows:
Exploring in-domain representation learning to train generic remote sensing representa- tions.
2. Generating 5 existing remote sensing datasets in standardized format and establishing common evaluation protocol. Publishing the best trained representations for easy reuse in transfer learning application
3. Establishing state-of the-the-art baselines for the BigEarthNet, EuroSAT, RESISC- 45, So2Sat, and UC Merced datasets.
RELATED WORK
REPRESENTATION AND TRANSFER LEARNING
Shortly after AlexNet (Krizhevsky et al. 2012) was trained on ImageNet (Deng et al., 2009), rep- resentations obtained from were used as off thel- feature extractor networks (Razavian al.] 2014). Fine-tuning approaches were also explored and led to better results than aandom-iittiaiizatinn even when the tasks or domains differ (Yosinski et al.] 2014; Kornblith al. 2019). These ap- proaches extend beyond natural images and have been used the in medical domain, leading Raghu et al. (2019) to question on how they work. Simultaneously there have been many improvements on training representations. Mahajan et al. (2018) explored training bigger models on larger datasets with weaker labels and (Ngiam et al., 2018; Cui et al.) 2018) showed gains by closer matching the domains. Very promising the more sample-efficient approaches using semi- self-supervision (Zhai et al. 2019a).
DEEP LEARNING IN REMOTE SENSING
Common remote sensing problems include land-use and land cover (LULC) classification, physical and bio-gco-chemical parameter estimation, target detection, time-series analysis, pan-sharpening and change detection. Many of these tasks be solved or helped by deep Icarning approaches related to classification, object detection, semantic segmentation, or super-resolution. Reviews of
some of these approaches be found for instance in (Ball et al. 2017 Zhu al. 2017 2019).
Remote sensing data is acquired different modes: optical, multi- and hyper-spectral, synthetic aperture radar (SAR), lidar, spectrometer (Elachi & Van Zyl (006). Each of these modes has its own acquisition geometry and specific characteristies providing unique and complimentary information about the Earth's surface. In dependence of the instrument, the remote sensing imagery is available from very high resolutions at centimeter scale (acrial optical or radar, eg. for urban monitoring) to very low resolution kilometer scale (eg. for atmosphere and ocean surface monitoring). Another important satellite data characteristic is the revisit time (how fast does a satellite revisit the same location, and constructs time series), which can range from daily to multiple months.
The majority of deep learning approaches in remote sensing are currently based on optical imagery high (0.3-1 m) to medium (10-30 m) resolution, obtained from aerial imagery (such as seen on Google Earth) publicly available satellite data (eg. NASA's Landsat or ESA's Sentinel-2 satel- lites), respectively. Though, multi-spectral, hyper-spectral, and radar imagery increasingly being used well.
REPRESENTATIONTTRANEER LEARNING REMOTE SENSING
Since labeling remote sensing data expensive, for long time there was no equivalent to ImageNet and most benchmark datasets were small, The probably most used remote sensing benchmarking dataset, UC Merced (Yang & Newsam 2010), has only 2100 images with 100 images per class.
To be released upon publication.
2

---

some datatasets are perfectly balanced, while others are highly unbal- Under review as a conference paper at ICLR 2020
Because it not possible to train large state-of-ther network small dataset from scratch, remote sensing researchers started to build smaller networks, as for instance in Luus et al. (2015).
Another direction was to use models pre-trained on natural images. For instance, Marmanis et al. (2016) used the extracted CNN features from the pre-traing ree-trained Overfeat model (Sermanet et al. 2013)
to feed to another CNN model. Castelluceio et al. (2015) used GoogLeNet (Szegedy et al., 2015) pre-trained on ImageNet to fine-tune UC Merced. Similarly, Nogueira et (2017) used pre-trained models on natural imagery to fine-tune five models based of cross-validation splits. Afterwards, they trained an additional SVM on the fine-tuned features to achieve best performance.
Although there are works in using pre-trained representations, there is hardly any work on training representations specific for solving tasks on this domain. An example is the multi-layer genera- tive adversarial network (GAN) (Goodfellow al., 2014) approach for unsupervised representation learning presented by Lin et al. (2117). Xie et al. (2015) developed transfer learning pipeline for mapping poverty distribution based on transfering the nighttime lights prediction task. Not specifically addressing representation learning, but still related application using conditional GANs (Mirza & Osindero] (2014) was demonstrated for cloud removal from RGB images by fusing RGB and near infrared bands in Enomoto et al. (2017), or by fusing multi-spectral and synthetic aperture radar data (Grohnfeldt et al. 2018).
3 DATASETS
For this work, five diverse datasets were selected. We prioritized newer and larger datasets that are quite diverse from each other, address scene classification tasks, and include at least optical imagery.
Image data comes either from aerial high nigh-resolution imagery or from satellites. Three datasets in- clude imagery from the European Space Agency's 's (ESA) Sentinel-2 satellite constellation, that pro- vides medium resolution imagery of the Earth's 's surface every three days. The multi-spectral imager on Sentinel-2 delivers next the 3 RGB channels additional channels at various frequencies (see Appendix A. One dataset includes co-registered imagery from a duallpollrimetric synthetic aper- ture radar (SAR) instrument of ESA's Sentinel-1.
Besides the differences in data sources, number of training samples, number of classes, image sizes and pixel resolutions (summarized in Table D the datasets are also quite diverse across:
Intra- and inter-class visual diversity: some datasets have high in-class and low between- classes diversity and vice versa.
Label imbalance:
anced.
Label domain: land-use land cover (LULC), urban structures, fine ecological labels.
Label quality: from fine human selection to weak labels from auxiliary dataset
While having only 5 datasets might not allow us to completely disentangle the confounding factors of remote sensing representation learning. it should still help us in understanding the important factors for good remote sensing representation learning datasets.
3.1 INDIVIDUAL DATASETS
In addition the descriptions of the individual datasets in this section, Appendix A shows example images for all datasets and the distribution of classes.
BigEarthNet (Sumbul et al., 2019) is a challenging large-scale multi-spectral dataset consisting of 590,326 image patches from the Sentinel-2 satellite. 12 frequency channels (including RGB) are provided, each covering pixel. This is a multi-label dataset where each image is annotated by multiple land-cover classes. The label distribution is highly unbalanced ranging from 217k images of "mixed forest" label to only 328 images with label "burnt areas". About 12% of the patches are fully covered by seasonal
area of 1.2 x 1.2 km with resolutions of 10 m, 20 m, and 60 m per
Some datasets and images affected by not filtered-out cloud and snow coverage that makes the correct. classification of the samples difficult.
3

---

Under review as a conference paper at ICLR 2020
Table Overview of considered remote sensing datasets.
Name
year
Source
Size
Classes
Image size
Resolution
Problem
Nee
2019
Sentinel-2
590k
120x 120*
10-60
multi-label
EuroSAT
2019
Sentinel-2
27k
10
64x64
10 m
multi-class
RESISC-45
2017
aerial
31.5k
256x256
0.2-60+
multi-class
So2Sat
2019
Sentinel-1/2
376k
17
32x32
10
multi-class
UC Merced
2010
aerial
2.1k
256x256
0.3 m
multi-class
*Image size varies in dependence of resolution from 120x to 60x60 to 20x20.
snow, clouds or cloud shadows. The only available public baseline metrics include precision and recall values of 69.93% and 77.1%, respectively, for using shallow CNN on reduced dataset, after removing the snow and cloud affected samples.
EuroSAT (Helber et al. 2019) is another recently published dataset containing 27,000 images from Sentinel-2 satellites. All 13 frequency bands of the satellite are included, Each image covers an area of 640 640 meters and is assigned to one of 10 LULC classes, with 2000 to 3000 images per class. Because the classes are quite distinctive, very high accuracies can be achieved when using the entire dataset for training.
NWPU RESISC-45 Cheng et al. 2017) dataset an aerial dataset consisting of 31,500 RGB images divided into 45 scene classes. Each class includes 700 images with size of 256 x 256 pixels. This is the only dataset with varying spatial resolution ranging from 20 cm to more than 30 meters. The data covers wide range of countries and biomes. During the construction, the authors paid special attention to have classes with high same-class diversity and between-clase similarity to make it more challenging.
So2Sat LCZ-42 (Zhu et al. 2018) is dataset consisting of co-registered SAR and multi-spectral 320 x 320 m image patches acquired by the Sentinel-1 and Sentinel-2 remote sensing satellites, and the corresponding local climate zones (LCZZ) (Stewart & Oke 2012) labels. The dataset is distributed over 42 cities across different continents and cultural regions of the world. This is another challenging dataset and it is intended for learning features to distinguish various urban zones. The challenge of this dataset is the relatively small image size (32x32) and the relatively high inter-class visual similarity.
UC Merced Land-Use Dataset (Yang & Newsam 2010 is a high-resolution (30 em per pixel) dataset that was extracted from acrial imagery from the United States Geological Survey (USGS) National Map over multiple regions in the United States. The 256 x 256 RGB images cover land-use classes, with 100 images per class. This is relatively small datasets that has been widely benchmarked for remote sensing scene classification task since 2010 and for which nearly perfect accuracy can be achieved with modern convolutional neural networks (Castelluccio al. 2015 Marmanis et al., 2016; Nogueira el al. 2017
3.2 RELEASE OF STANDARDIZED DATASETS
To simplify access the data and its usage, imported and published these datasets
For reproducability and a common evaluation framework, standard train, validation, and test splits using the 60%, 20%, and 20% ratios, respectively, were generated for all datasets except So2Sat
For the So2Sat dataset, the source already provides train and validation splits. To generate the test split, the original upstream validation is separated into validation and test splits with the 25% and 75% ratios, respectively.
To be updated/released upon publication.
4

---

Under review as a conference paper at ICLR 2020
4 REMOTE SENSING DATA PROCESSING
The remote sensing domain is quite distinctive from natural image domain and requires special attention during pre-processing and model construction. Some characteristics are:
Remote sensing imput data usually comes at higher precision (16 or 32 bits).
The number of channels variable, depending on the satellite instrument. RGB channels are only subset of multi- or hyper-spectral imagery dataset. Other data sources might have no optical channels (eg. radar or lidar) and the channels distribution can be determined by polarimetric, interferometric or frequency diversity.
The range of values varies largely from dataset dataset and between channels. The values distribution can be highly skewed.
Many quantitative remote sensing tasks rely on the absolute values of the pixels.
The images acquired from space are usually rotation invariant.
Source data can be delivered different product levels (for instance w/ or w/o atmospheric correction, co-registration, orthorectification, radiometric calibration, etc.).
Especially lower resolution data aggregates lot of information about the illuminated sur- face in single pixel since it covers large area.
Image axes might be non-standard, eg. representing range and azimuth dimensions.
This some requirements on pre-processing and encourages to adjust data augmentation of the input pipeline for remote sensing data.
Specifically for the problems discussed in this paper, it is recommended to rescale and clip the range of values per channel (accounting for outliers). Data augmentation that affects the intensity of the values should be discarded. On the other hand, one can reuse the rotation invariance and extend the augmentation to perform all rotations and flipping (providing 7 additional images per sample). Given multi- suetti-ppectral data, such Sentinel-2 based BigEarthNet, EuroSAT and So2Sat, one can use other subsets of channels instead of RGB including all available ones.
EXPERIMENTAL RESULTS
The main goal is develop representations that can be used across a wide field of remote sensing applications on unseen tasks. The training and evaluation protocol follows two main stages: (1) upstream training of the representations model based on some out- in-domain data, and (2) down- stream evaluation of the representations by transferring the trained representation features to the new downstream tasks. For the upstream training on in-domain proxy datasets the entire data used, that cannot include any data from the downstream tasks. The quality of the trained representations and their generalizability is often evaluated on reduced downstream training sizes to assess efficiency of the representations.
5.1 EXPERIMENTAL SETUP
In order to simplify the training procedure and draw more general conclusions, all experiments use the same ResNet50 V2 architecture (He et al. (2016) and configuration. However, due to the varied number of classes and training samples in the various datasets, we perform sweeps over small set of syper--praamter and augmentations as described in detail in Appendix B
5.2 EVALUATION METRICS
We report performance results using accuracy metrics commonly used in computer vision tasks: for multi-class problems we use the Top- global accuracy metric, which denotes the percentage of correctly labeled samples. For multi-label problems we use the average precision (mAP) metric, which denotes the mean over the average precision (AV) values (AV the integral over the precision-recall curve) of the individual labels.
5

---

Under review as a conference paper at ICLR 2020
Table Performance of trained In-Domain and ImageNet representations (rows) when using 1000 training examples for downstream tasks (columns). Emphasized (bold font) are the best accuracies per downstream task (column).
Source Target
BigEarthNet
EuroSAT
RESISC-45
So2Sat
UC Merced
ImageNet
25.10
96.84
84.89
53.69
99.02
BigEarthNet
96.45
78.43
50.91
99.61
EuroSAT
27.10
79.59
52.99
98,05
RESISC-45
27.59
97.14
54.43
99.61
So2Sat
26.30
96.30
77.70
97.27
UC Merced
26.86
96.73
85.73
53.52
To measure and aggregate relative performance improvement over datasets performing quite dif- ferent accuracy levels, the logit transformation of the accuracy is preferred (Kornblith et al. 2019):
P logit(p) log I I-p) p)
sigmoid(p)
where p is the accuracy rate (Top-1 1 or mAP) and A is the corresponding logit accuracy. It captures the importance of accuracy change different accuracy levels provide a more fair evaluation of the achievement.
5.3 COMPARING IN-DOMAIN REPRESENTATIONS
To obtain in-domain representations, first train models either from scratch or by fine-tuning Im- ageNet on each full dataset. The best of these models are then used as in-domain representations to train models on other remote sensing tasks (excluding the one used to train the in-domain represen- tation).
For an initial evaluation of the different in-domain representation source data, Table Table shows cross-table of evaluating each trained in-domain and ImageNet representation each of the down- stream tasks. The representations were trained using full datasets upstream, while the down-stream tasks used only 1000 training examples to better emphasize the differences.
can be observed that with 1000 training examples, the best results all come from fine-tuning the in-domain representations.
Additionally, despite having 2 distinctive groups of high-resolution acrial (RESISC-45, UC Merced) and medium-resolution satellite datasets (BigEarthNNt,, EuroSAT and So2Sat), the representations trained on RESISC-45 were able to outperform the others in all tasks (BigFarthNet representa- tions tied for the UC Merced dataset) and it the only representation consistently outperform ImageNet-based representations.
That RESISC-45 would perform good both aerial and satellite data unexpected. The reason is most likely related to the fact that RESISC-45 is the only dataset that has images with various resolutions. Combined with the large number of classes that have high within class diversity and high between-class similarity it seems to be able to train good representations for wide range of remote sensing tasks, despite not being very big dataset. This learning can be reused adjust augmentation schemes for remote sensing representation learning by stronger varying the scale of images with randomized zooms and potentially other affine transformations.
The best representations for the RESISC-45 itself come from the other aerial dataset, UC Merced. The relatively small training size of was counter-balanced by an aggressive augmentation (crop & rot as described in Appendix B.3).
Counter to the expectation that bigger datasets should train better representations, the two biggest datasets, BigEarthNet and So2Sat, didn't provide the best representations (except of Big representations for UC Merced). We hypothesize that this might be due to the weak labeling and the low training accuracy obtained in these datasets. It is possible that the full potential of these large- scale datasets was not yet fully utilized and other self- or superviie representation learning approaches could improve the performance.

---

Under review as a conference paper at ICLR 2020
Logit accuracy gain
Logit accuracy gain
2
1.0
ImageNet
1
InDomain
BigEarthNet
EuroSAT
RESISC-45
So25at
uc Merced
0.0
100
1000
Full
(a) Over datasets
Over sizes
Figure I: Aggregated mean relative improvement in logit accuracy of fine-tuning from ImageNet and in-domain representations comparison to training from scratch.
5.4 LARGE-SCALE COMPARISON
Having trained in in-domain representations, we can now evaluate and compare the transfer quality of fine-tuning the best in-domain representations with fine-tuning ImageNet and training from scratch at various training data sizes. The results using the default experimental setup are shown in Table
In cases, fine-tuning from ImageNet is better than training from scratch. And all but one case, fine-tuning from an in-domain representation for transfer is even better.
The only exception the BigEarthNet dataset its full size. Is expected that having a large dataset should reduce the need for pre-training, but the gap between in-domain and ImageNet pre-training quite big. We don't have an explanation for this yet and this needs be further investigated.
Overall, these results establish new nate-of-the-ar baselines for these datasets, as summarized in Table Note that some results are not comparable: RESISC-45 has been previously evaluated only on 20% of data, So2Sat has public benchmarking result to our knowledge, and the only published result of BigEarthNet is based on a cleaner version of the dataset (after removing the noisy images containing clouds and snow) and only precision and recall metrics
reported.
Fig. summarizes the results of Table across all datasets and training sizes. Logit accuracy metric used, which fits better for aggregation of accuracies across wide ranges. Fig. [a emphasizes that the smaller datasets (FuroSAT, RESISC-45, UC Merced) profit more from in-domain knowledge than the larger datasets (BigFarthNet, So2Sat). As is re-iterated again in Fig. [bb using in-domain representations leads to higher accuracy gain when small number of training examples are available, which is of most interest for remote sensing applications reduce the ground truth data acquisition.
5.5 LIMITED NUMBERS OF TRAINING EXAMPLES
To look closer into in-domain representation learning for small number of training examples, trained models with small training sizes ranging from to 2500 (samples were randomly drawn disregarding class distributions). We used simplified set of hyper-parameter that might not deliver the most optimal performance, but still allows to observe the general trends. As shown Fig. 2 the improvement of using in-domain representations clearly visible for the EuroSAT, RESISC-45 and UC Merced datasets. These are the 3 smaller datasets with higher quality labels. The results are less conclusive for the BigEarthNet and So2Sat datasets that have more noisy labels.
Table 3: Accuracy over different training methods and number of used training samples.
Bat 100 1k
EuroSAT
RESISC-45
So2Sat
UC Merced
Full
100
1k Full
100
1k Full
100
1k Full
100
Ik Full
Scratch ImageNet
14.5 21.4 72.4
63.9
91.7 98.5
21.4
56.1 95.6
33.9
47.0
62.1
50.8
91.2 95.7
17.8
25.1
75.4
87.3
96.8
99.1
44.9
84.9
96.6
44.9
53.7
63.1
79.9
99.0 99.2
InDomain
18.8 27.6
69.7
91.3
97.1
99.2
49.0
85.7
96.8
46.4
54.4
63.2
91.0
99.6
99.6

---

Under review as a conference paper at ICLR 2020
Table 4: Results on selected remote sensing datasets. All results use Top-1 accuracy, except for BigEarthNet which uses precision/recall and mean average precision. Our best results were obtained by fine-tuning in-domain representations except for BigEarthNet which was obtained by fine-tuning ImageNet.
Dataset
Reference
Result
Sumbul et al. 20119
69.93% 77.1%% (P/R) 75.36% (mAP)
BigEarthNet
Ours
Helber et al. Ours
2019
98.57%
EuroSAT
99.20%
Cheng al. (0177
90.36%
RESISC-45
Ours
96.83%
So2Sat
Ours
63.25%
Marmanis
(016))
92.4%
Castelluccio et al. 2015)
97,
UC Merced
Nogueira et 201
99.41% 11
Ours
99.61%
1.0
0.3
Accuracy
Accuracy
0.2
0.50
0.6
0.25
0.4
25
250
2500
25
250
2500
250
2500
Training samples
Training samples
Training samples
(a) Beg
(b) EuroSAT
(c) RESISC-45
1.00
0.5
Accuracy
Accuracy
0.75
0.4
InDomain ImageNet
0.50
FromScratch
0.25
25
250
2500
25
250
2500
Training samples
Training samples
So2Sat
(c) UC Merced
Figure 2: T accuracy rate or mean average precision (mAP) on validation set after training with a given method over a limited number of training examples on each dataset.
CONCLUSION
We present common evaluation benchmark for remote sensing representation learning based on five diverse datasets. The results demonstrate the enhanced performance of in-domain represen- tations, especially for tasks with limited number of training samples, and achieve state-of-the-art performance. The five analyzed datasets and the best trained in-domain representations are pub- lished for easy reuse by the public.
We investigate dataset characteristics to be a good source for remote sensing representation learning. As the experimental results indicate, having multi-resolution dataset helps to train more general- izable representations. Other factors seem to be label quality, number of classes, visual similarity across the classes and visual diversity within the classes. Surprisingly, observed that representa- tions trained on the large weakly-supg weakkl-supervised datasets were not as successful as that of a smaller and more diverse human-curated dataset.
8

---

Under review as a conference paper at ICLR 2020
However, some results were inconclusive and require more investigation. Understanding the main factors a good remote sensing dataset for representation learning is a major challenge, solving which could improve performance across wide range of remote sensing tasks and applications.
REFERENCES
John E. Ball, Derek T. Anderson, and Chee Seng Chan. Comprehensive survey of deep learning in theories, tools, and challenges for the community. Journal of Applied Remote Sensing, 11(04):1, September 2017. ISSN 1931-3195, doi: 10.1117/1.JR..11.0000..
remote sensing:
Marco Castelluccio, Giovanni Poggi, Carlo Sansone, and Luisa Verdoliva. Land Use Classification in Remote Sensing Images by Convolutional Neural Networks. August 2015. URL http:
//axxiv. abs/ 1....0000000000 00092
G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): 1800)-8868-8333 October 2017. ISSN 0018-9219. doi: 10. (Q.1109//ROC..22122227999
Yin Cui, Yang Song. Chen Sun, Andrew Howard, and Serge J. Belongie. Large scale fine-graine gee-granndd categorization and domain-specifit transfer learning. CoRR, abs/1806.066 2018. Uttht
//atxii..rrg/abs/180000 06193
Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Fei-fei. Imagenet: A large-scale hierarchical image database. In In CVPR, 2009.
Charles Elachi and Jakob J Van Zyl. Introduction to the physics and techniques of remote sensing, volume 28. John Wiley & Sons, 2006.
Kenji Enomoto, Ken Sakurada, Weimin Wang, Hiroshi Fukui, Masashi Matsuoka, Ryosuke Naka- mura, and Nobuo Kawaguchi. Filmy cloud removal satellite imagery with multispectral con- ditional generative adversarial nets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 48-56, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680 2014.
C. Grohnfeldt, M. Schmitt, and X. Zhu. A conditional generative adversarial network to fuse and multispectral optical data for cloud removal from sentinel-2 images. In IGARSS 2018 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 1726-1729, July 2018. doi: 10.1109/IGARSS. 2018 8.811921.1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision. Springer, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 12(7):22172266, July 2019. ISSN doi: 10. 109/sstars.20019 2019..9901119112.. 2918222.
2151-1535.
URL httpp://xx... www.::ww.c.oooooo.o..ooooo org 1109/
JSTARS 2019.2918242
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. URL
http://arxivv
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc.. 2012. URL http::/ wwterrs.niipe nips cc. cctppppp////pccccccpccc
4824 1824-imaqedd ieiiimmageeette www.eenttt-laasssaaatiiioo. withhhnn-------- htt.::/ooo.-tte-eeeeoooooooooooo
ort.aaaaaattttttoooooooooooooooa orks.
pdf

---

Under review as a conference paper at ICLR 2020
Daoyu Lin, Kun Fu, Yang Wang, Guangluan Xu, and Xian Sun. MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification. IEEE Geoscience and Remote Sensing Letters, 14(11):2092296, November 2017. ISSN 1558-0571. doi: 10.1109/1gss2017.
2752750. URL. http:///: doi. wtt.:doo......... 111.0111111111111 OT/.GRRS/R...... 2017 2011..25277.7.. 2752750
Luus, B.P. Salmon, F. Bergh, and B. Maharaj. Multiview deep learning for land-use classifica- tion. IEEE Geoscience and Remote Sensing Letters, 12:1-5, 10 2015. doi: 10.1109/LGRS.200 2483680.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the limits of weakly supervised pretraining. CoRR, abs/ abss/1805.oo 000500932, 2018. URL BRLhhtp::/ http::://axxiv.rroe
wwwpoag.////oooooooe/t...oe..aee www.aa/s.....e.... 00932
Dimitrios Marmanis, Mihai Datcu, Thomas Esch, and Uwe Stilla. Deep Learning Earth Observa- tion Classification Using ImageNet Pretrained Networks. IEEE Geoscience and Remote Sensing Letters, 13(1):105-109 jan 2016. ISSN 1545-598X. doi: 10. L.L10//LGRS.20152299922.. URL http://ieeexpin
leee. org/ /ttp://mmmmmmm/322////////
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1755 2014.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc Le, and Ruoming Pang- Domain adaptive transfer learning with specialist models. CoRR, abs/1811.07056, 2018. URL
Keiller Nogueira, Otvio A.B. Penatti, and Jefersson A. dos Santos. Towards better exploiting convolutional neural networks for remote sensing scene classification. Pattern Recogn., 61 (C):539-558) January 2017. ISSN 0031-3203. doi:
patcoo.....o..00......00 URL
0atpe//...1001111111 pat 2016.000000000000000000000..2200 07.001
Maithra Raghu, Chiyuan Zhang, Jon M. Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning with applications medical imaging. CoRR, abs/1902.07208, 2019. URL
hittp://arxiv. wttp:///aaa////..0222222/
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. CoRR, abs/1403.6382,, 2014. URL http:
wlarxiv.org/ee www./r///aaaaaaaaeee.eeeeeeeee/e 403. 6382
David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brows Alexan- dra Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Kording, Carla Gomes, Andrew Y. Ng, Demis Hassabis, John Platt, Felix Creutzig, Jennifer Chayes, and Yoshua Bengio. Tackling climate change with machine learning, 2019.
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun. Over- feat: Integrated recognition, localization and detection using convolutional networks, 2013.
LDD. Stewart and T. Oke. Local climate zones for urban temperature studies. Bulletin of the American Meteorological Society, 1990879-1900, 12 2012. doi: 1751/75/B SEW-BEAMM--------O- D-11-00019.1
Gencer Sumbul, Marcela Charfuelan, Begm Demir, and Volker Markl. Bigearthnet: large-scale benchmark archive for remote sensing image understanding, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, pp- 1-9, 06 2015. doi: 10.1 2009/CVP..001..29990
Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping. September 2015. URL http: 7/arxiv //aastw..rrg/ab//551.oooo
10

---

Under review as a conference paper at ICLR 2020
Yi Yang and Shawn Newsam. Bag-of- oi--o---vull-wodds and spatial extensions for land-use classifica-
tion. In Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geo- graphic Information Systems GIS '10, pp. 270, New York, New York, USA, 2010. ACM Press. ISBN 9781450304283.
doi: 10. 14.1145/1866 169999 117779901119999 1668299. URL httphtt:p//potaa acm, orrg/
ci tation on wwwtiion.cfffaae..oooo 1tt.mmdddoddd111101111 1869829
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3320-3328 Curran Associates, Inc., 2014. URL 5347 hww-hoowwweee www.steranneeeeeee are features in -eee--nnurll--ttwook.. pdf
inpeerssnippi nips. ctppsccr/pprrrr neural
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S:L: Self-supervised semi- supervised learning, CoRR, (bs/1995,,,660, 2019a.
03670
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark, 2019b.
Ming Zhu, Yongning He, and Qingyu He. A Review of Researches Deep Learning in Remote Sensing Application. International Journal of Geosciences, 10011)1-11, 2019. ISSN 2156- 8359. doi: 10.4236/ij 101000 URL http://wwwww.ip.oo/////oooo/// aspx?DoI=10 4234443361211 2019. 101001
Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep Learning in Remote Sensing: A Comprehensive Review and List of Re- sources. IEEE Geoscience and Remote Sensing Magazine, 5(4): dec 2017. ISSN 2168-6831. doi: 10.1 00.109//RSS.200..7223300 URL htLLttttt///eeexpplecc ieee. org/ document/ 81 8113128/
Xiaoxiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Hossein Bagheri, Jian Kang. Hao Li, Lichao Mou, Guicheng Zhang, Matthias Hberle, Shiyao Han, Yuansheng Hua, Rong Huang, Lloyd Hughes, Yao Sun, Michael Schmitt, and Yuanyuan Wang. So2Sat LCZ42, 2018. URL https: //oo:///meiaaum... ub. w.b.tum tum. de/ (e94454690)

---

Under review as a conference paper at ICLR 2020
A
ADDITIONAL DATASET INFORMATION
This section provides additional information on the used datasets and data sources.
A.1 SENTINEL-2 SATELLITE
Sentinel-2 multi-spectral satellite constellation from the European Space Agency (ESA). Since 2017 two satellites are in operation delivering 5-days revisit at equator and 2-3 days at high lati- tudes. The characteristics the 13 bands are presented in Table
Table 5: Sentinel-2 channel characteristics (Abbreviations: Wavelength Infra-Red).
NIR: Near Infra-Red, SWIR: Short-
Band and Highest Sensitivity Target
Spatial Resolution [m]
Central Wavelength [nm]
B01 Aerosols
60
443
B02 Blue
490
B03 Green
10
560
B04 Red
10
665
B05 Red edge
20
705
B06 Red edge 2
20
740
B07 Red edge 3
20
783
B08 NIR
10
842
BOSA Red edge
20
865
BO9 Water vapor
60
945
B10 Cirrus
1375
B11 SWIR 1
20
1610
B12 SWIR 2
20
2190
12

---

Under review as a conference paper at ICLR 2020
A.2 BIGEARTHNET
The following figures show some example images and label distribution for the BigHarthNet dataset.
Sample
Sample
Sample
Sample
Sample
Sample
Sample
Sample
Sample 9
Sample 10
Sample 11
Sample 12
Sample 13
Sample 14
Sample 15
Sample 16
Sample 17
Sample 18
Sample 19
Sample 20
Sample 21
Sample
Sample
Sample 24
Sample
Figure 3: BigFarthNet image examples. Some images might be affected by seasonal snow, clouds or cloud shadows, which is not reflected in the land cover labels of this dataset.
Labels for the examples in Fig.
Vineyards, Broad-leaved forest (2 labels)
2. Coniferous forest, Mixed forest, Transitional woodland/shrub (3 labels)
3. Sea and ocean label)
4. Land principally occupied by agriculture, with significant areas of natural vegetation, Coniferous forest, Mixed forest, Water bodies (4 labels)
5. Non-irrigated arable land, Mixed forest (2 labels)
6. Land principally occupied by agriculture, with significant areas of natural vegetation, Coniferous forest, Mixed forest, Transitional woodland/shrub (4 labels)
7. Sea and ocean labels)
8. Coniferous forest, Mixed forest (2 labels)
13

---

Under review as a conference paper at ICLR 2020
9. Discontinuous urban fabric, Industrial commercial units, Coniferous forest, Mixed for- est, Transitional woodland/shrub (5 labels)
Non-irrigated arable land, Pastures (2 labels)
11. Coniferous forest, Mixed forest, Water bodies (3 labels)
12. Sea and ocean (1 labels)
13. Sparsely vegetated areas, Peatbogs (2 labels)
14. Coniferous forest, Mixed forest, Transitional woodland/shrub (3 labels)
15. Continuous urban fabric (1 label)
Complex cultivation patterns, Land principally occupied by agriculture, with significant areas of natural vegetation, Broad-leaved forest (3 labels)
17. Land principally occupied by agriculture, with significant areas of natural vegetation, Broad-leaved forest, Selerophyllous vegetation, Transitional woodland/shrub (4 labels)
18. Agro-forestry areas, Broad-leaved forest, Transitional woodland/shrub (3 labels)
19. Non-irrigated arable land, Land principally occupied by agriculture, with significant areas of natural vegetation, Coniferous forest, Mixed forest (4 labels)
20. Non- Non-irrigated arable land label)
21. Coniferous forest, Mixed forest, Transitional woodland/shrub, Water bodies (4 labels)
22. Coniferous forest, Mixed forest (2 labels)
23. Non-irrigated arable land, Land principally occupied by agriculture, with significant areas of natural vegetation, Agro-forestry areas, Sclerophyllous vegetation, Transitional wood- land/shrub, Water bodies (6 labels)
24. Discontinuous urban fabric, Non-irrigated arable land, Inland marshes (3 labels)
25. Land principally occupied by agriculture, with significant areas of natural vegetation, Broad- forest, Coniferous forest (3 labels)
14

---

Under review as a conference paper at ICLR 2020
250000
200000
150000
Annual crops associated with permanenty crow
Agro forestryyy wreas
Beaches, dunes, sands
Airports
Broad levved forest Bumt areas
Bare rock
Complex onltvaation pattsrn mes
Intertitt applatiion Land principally occupied by spricuiture with slgnificnnt areas natural vegelation
Condaal leguome
Conifferou forest
Conarrction onnttuuctooono sites
Decentiane ultan labric
Fruit trees and berry plantations
urban labrec
industrial commerca commerciaal unts
Green urban preas
Exuaries
inland murshic
Mineral extractioo gites
Hours and heathlons Natural gransland
Mizou foreas
Man-irigeteed Olive groves
Road and rad networks and associnn wotoocttddddooo lans
land
Pasturez
peatbogs
Port areas
Rice fleils
scierophyltoup vey vegelania attion
Salt marshec
Salines
Sparsely verelatos areas Sport and lesure faciitins
Sea and ocean
Transitiona aoodian mondaaannnnnn andiannub
Vineerrd Water bedias
Water courses
Dump
Permanently iriigttd petteel
Figure 4: BigEarthNet labels distribution counts.
15

---

Under review as a conference paper at ICLR 2020
A.3 EUROSAT
The following figures show some example images and label distribution for the EuroSAT dataset.
Residential
Highway
Herracceousseegetaiionoeeem Prrmantttto
SeaLake
River
PermanentCrop
Pasture
Forest
Residential
Forest
Forest
SeaLake
PermanentCrop
Forest
Pasture
Industrial
Highway
Forest
Highway
PermanentCrop
SeaLake
Pasture
SeaLake
HerbaaceossVegetation
Figure 5: EuroSAT image examples.
3000
2500
2000
1500
1000
ArnualCrop
Forest
Herbaceoud Prruaceouvvegeee vegetatioon
Highway
Instustial
Pasture
PermaneniCrop
Residential
River
Sealake
Figure 6: EuroSAT class distribution counts.
16

---

Under review as a conference paper at ICLR 2020
A.4 RESISC-45
The following figures show some example images and label distribution for the RESISC-45 dataset.
tennis court
commercial area
river
circular farmland
palace
church
church
golf course
church
sea ice
freeway
bridge
bridge
bridge
rectangular farmland
baseball diamond
meadow
beach
intersection
beach
stadium
meadow
lake
chaparral
railway station
Figure 7: RESISC-45 image examples.
700
600
500
400
300
200
100
o A fesies
& cequous
airplane
aurport
Biamond
court beach
bridge
holid dhaperral
church
Cermant
aren seideedd
paring
a (e)
tse sattiono
a
so ponoppunai
uae Ao
Aoton sooe aurzun toog att unt pooooss eseeds ddd
diant
basebali basketball
Henee commercted
yorr coglee The
drcular
Figure 8: RESISC-45 class distribution counts.
17

---

Under review as a conference paper at ICLR 2020
A.5 SO2SAT
The following figures show some example images and label distribution for the So2Sat dataset.
Low plants
Bush scrub
Compact mid-rise
Water
Compact mid-rise
Water
Low plants
Large low-rise
Open low-rise
Dense
Dense trees
Compact high-rise
Water
Low plants
Sparsely built
Open low-rise
Open high-rise
Open low-rise
Open low-rise
Open low-rise
Scattered trees
Scattered trees
Low plants
Water
Dense trees
Figure 9: So2Sat image examples.
50000
40000
20000
10000
Bara rock or paved
Bare soil sand
Bush or scrub
Compact high-rti ris
Compact low-rise
Compact mid-rss
Dense trees
Heavy Industry
Large fow-rise
Lightweight low-rise Low plants
Open high-rise
Open low-rise
Open mid-rise
Scattered trees
Sparsely built
Water
Figure 10: So2Sat class distribution counts.
18

---

Under review as a conference paper at ICLR 2020
A.6 UC MERCED
The following figures show some example images and label distribution for the UC Merced dataset.
harbor
intersection
agricultural
mobilehomepark
denseresidential
sparseresidential
golfcourse
chaparral
overpass
river
parkinglot
mobilehomepark
buildings
storagetanks
runway
freeway
chaparral
parkinglot
agricultural
sparseresidential
overpass
chaparral
buildings
golfcourse
overpass
Figure 11: UC Merced image examples.
100
40
20
agricultural
airplane
basebalidiamond
beach
buildings
chaparral
denseresidential
forest
freeway
golf-ourse
herbor
intersection
mediumresioe mennresidenteel
mobilehonneeark
overpass
parkinglot
river
runway
gparseresidentaa
storagetents
tenniscourt
Figure 12: UC Merced class distribution counts.
19

---

Under review as a conference paper at ICLR 2020
B TRAINING SETUP
B.1 ARCHITECTURE AND HYPER- HYPER-PARAMETER PARAMETERS
All the models trained in this work the ResNet50 v2 architecture (He al., 2016) and were trained using SGD with momentum set to 0.9 on large batch sizes 512 or 1024. In total we configure and sweep only a fixed of eyper-parameter per model, namely:
learning rate: (0.1,
weight decay: {0.01, 0.0001}.
training schedules: {short, medium, long),
preprocessing: (resize, crop & rot) (described in Appendix B.3
All the training schedules use linear warm-up for learning rate over the first w steps/epochs and decrease the learning rate by 10 per each learning phase p in the schedule. The learning rate schedules are given by:
short: p {750, 1500, 2250, 2500} steps, w 200 steps.
medium: p {3000, 6000, 9000, 10000} steps, 500 steps.
long: [30, 60, 80, 90} epochs, = 5 epochs.
These hyperparameter settings follow approximately the setup in ZZai et al. 2019b) with minor modifications for the schedule and preprocessing. In in inital phase, more extensive hyperparam- eter sets were tried out, but with not much effect on the best performance and therefore for the experiments presented in this paper limit the configurations the ones described above.
B.2 DETAILED SWEEPS PER EXPERIMENT
The models for experiments in Table 2 and Table were trained sweeping over all hyper param- eters. The best performing models oblained from fine-tuning ImageNet were used as in-domain representations for all experiments (excluding same up- and down-stream dataset configurations).
For Fig. 2 the models were trained by sweeping over the learning rate and using only the short and medium training schedules. The weight decay was set to 0.0001, and the preprocessing was set to resize.
B.3 PRE-PROCESSIING AND DATA AUGMENTATION
Data pre-prose pe-peoeeesing and augmentation can have a significant impact on performance. Therefore, in the reported results we used 2 rre-processing settings described below.
resize resize the original RGB input to 224x224 both at training and evaluation time.
crop & rot during training resize the original RGB input to 256x256, perform a random crop of 224x224 and apply one of 8 random rotations (90 degrees and horizontal flip).
During evaluation resize to 256x256 and perform a central crop of 224x224.
Table 6 shows the difference of each strategy when fine-tune from ImageNet.
Table 6: Accuracy of each pre-processing strategy when fine-tuning an ImageNet representation.
BigEarthNet 100
Au 1k
RESISC 45
So2Sat
UC Merced
Full
100
Full
100
1k
Full
100
Full
100
Full
resize
17.3 24.5
75.4
85.2 96.0
98.9
37.5 78.6 95.8
44.9 51.8
63.1
69.1 98.2 99.2
crop & rot
17.8
25.1
73.4
87.3 96.8
99.1
44.9
84.9
96.6
43.8 53.7
59.6
79.9 99.0 99.2
20


Reviews:
[{"review_id": "ryxO0GjqYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explored the representation learning problem on remote sensing domain. In particular, it provided several standardized remote sensing data sets with standard train/validation/test splits and various metrics for fairly comprehensive evaluation. It showed that compared to fine-tuning on ImageNet or learning from scratch, in-domain representation could produce better baseline results for remote sensing at various training data sizes. \nOverall, I prefer to reject this paper based on the following reasons:\n(1) It claimed to address the question “what characteristics should a dataset have to be a good source\nfor remote sensing representation learning”. But this question has not been fully explored. Empirically, multi-resolution data sets (e.g., RESISC-45) could be used as the base data set for better in-domain representation learning. However, the essential reason on this explaining this phenomenon is not still clear.\n(2) The authors did not provide the convincing evidence regarding how to generate the standard train/validation/test splits for all the data sets. The data split might be the reason to explain the superior performance of in-domain representation learning.\n(3) On evaluating the in-domain representations (shown in Table 2), all the representation models were trained with 1000 training examples for performance comparison. It is not convincing whether the best in-domain representation models are affected by the training size. Later, it used the best in-domain representation models to estimate the state-of-the-art baselines on various training size. It is not clear how to select the best in-domain representation models.\nMinor comments:\n(1) In Section 5.5, the authors trained the model with randomly sampled training examples. It might be more convincing if they can show the mean and variance of model performance using multiple randomly sampled examples.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lyKDBp9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors present a study on 5 remote sensing datasets where in-domain representation learning is studied and various methods to train and transfer knowledge between them are assessed. In addition, the authors aim to release these 5 datasets with appropriate training and test conditions to the public.\nAn interesting point of the paper is that ImageNet representations are used to either pretrain, finetune or train networks on each dataset and to transfer that performance to each other dataset. The performance of ImageNet is highly competitive int his context, demonstrating the quality of its features.\nThe paper is not proposing technical novelties of any kind throughout.\n\nComments:\n-pixel real estate and resolution makes it unclear which dataset is bigger, though the authors attempt to discuss this. The 'winning' dataset (in terms of its usefulness for transfer) seems to have the highest resolution and image size, which may easily be the reason that models can better be trained here.\n- Some of the tasks the authors attempt might best be attempted with different models. For example, not every remote sensing task is about classification of a patch. In some cases, segmentation into types of areas would be more appropriate. As such, the basic premise of the paper that the 5 tasks are similar and can be treated as one benchmarkable object seems quite brittle. A better approach seems to surely be to treat each problem with a more tailored approach?\nAdding the right inductive biases into each problem may also help mitigate the lack of specific domain data when aiming for high performance in each domain.\n- The authors miss a chance to consider this an exercise to perform a version of multitask learning for many of their experiments of similar structure, i.e. aerial images. In many cases satellite images will require similar RGB features. Why should we not use knowledge from all  datasets to get the best features? I was looking forward to seeing a discussion on this topic but unfortunately did not find it.\n\nDecision:\nOverall, this paper handles an interesting collection of data, but does not add much interesting novelty to the field of representation learning and fails to leverage some opportunities in those datasets to do sth. interesting (i.e. better transfer and/or better models). \nI also find the approach to tackle all 5 datasets and all problems in each dataset as a similar problem too simplistic. Remote sensing data, as many real-world datasets, is a wonderful playground for rich modeling ideas and inductive biases to make things work with less data or particularly structured data. \nIt is commendable to prepare the datasets for publication and the basic results the authors show make for good first baselines here, but ultimately do not meet the bar for novelty or scientific insights.\nThe paper might be a stronger fit for a targeted workshop on its topic.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hyge3Jjv5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper discusses transfer learning in remote sensing image-classification tasks.\nThe key finding is: while transfer learning from ImageNet dataset for remote sensing, performance can be slightly improved if the model weights are fine-tuned on an auxiliary remote sensing dataset (hence in-domain transfer).\n​\n​\n### Strengths\n- The paper is well motivated for the need of good representation learning for remote sensing, and is written well making it easy to follow.\n- It compiles a nice list of five datasets available for remote sensing, along with their characteristics. Also, the list of common properties of such datasets (e.g. rotation invariant) is good for data pre-processing for engineering applications in remote sensing.\n- The paper provides baseline accuracies on five important datasets in remote sensing.\n​\n​\n### Weaknesses\nApart from the contributions of compilation of different datasets which are already present, and classification results on them, there is not much novelty in the paper. There are several major weaknesses:\n​\n1. **About datasets**: The paper emphasizes a lot on the lack of standard datasets and metrics in remote sensing. However, the contribution is limited to compiling existing datasets and defining a 60-20-20 train-validation-test split on them. There is no new data collection, no insight in how these existing diverse datasets could be combined together, or why this particular definition of split is used. Hence, the reviewer cannot count this as a contribution.\n2. **About representation learning**: Since the source domain (ImageNet), in-domain source domain and the target domain all solve classification tasks, a better term to use would be *transfer learning* instead of representation learning. The reviewer recommends to demonstrate how these learned \"representations\" are used for other downstream tasks or otherwise show some other representation learning methods (semi-supervised or unsupervised), which is not same as the downstream task. Hence, it would be inappopriate to sell it as representation learning, in its current form.\n3. **About in-domain learning**: Since the in-domain transfer is also done by fine-tuning an existing model which was trained on ImageNet, it is expected to do somewhat better than Imagenet only. The paper just clarifies this fact in practice, and the performance gains are also statistically insignificant. Again, this is not a significant contribution.\n4. **About hyperparameter sweeping**: The minor improvements observed due to in-domain transfer is also unreliable, since very hyperparameters are tried across datasets. E.g. only 2 values of learning rates (0.1 and 0.01) are tested. This could also explain why BigEarthNet and So2Sat had poor training accuracies. The reviewer also disagrees with using only 1k samples for selecting best dataset to transfer from - explained in minor weaknesses.\n5. **About experiments**: In-domain transfer is tested by fine-tuning on one dataset and then again fine-tuning on another dataset. It was observed that the dataset RESISC-45 does the best in this scenario. The reviewer disagrees with paper's hypothesis about noisy labels resulting in this artifact. The reviewer suspects that this is because RESISC-45 contains labels which are most appropriate for classification transfer in other datasets (and also it is very balanced). Can the authors comment on this? If another method of representation learning than classification is employed, then it is very possible that other larger datasets result in better representations. The comparison across datasets currently is strange, since all it tells is which dataset has most similar classification labels to other datasets on average. To remove this problem in comparison, the representation learning task and testing task should be different.\n6. **About conclusions and contributions**: The experimental setup in this paper is insufficient to claim that it investigates *what characteristics should a dataset have to be a good source for remote sensing representation learning*. Transfer from ImageNet works because it contains a large number of classes, along with data diversity and its sheer size. This cannot be said for the datasets analyzed in this paper, since these factors are not analyzed individually, and the representation learning method is not standardized across all datasets (classification is not standardized since different datasets have different classes), making the comparison unfair.\n​\n​\n​\n#### Minor weaknesses:\n- Table 4: Since existing BigEarthNet result contains precision/recall, that is what the authors should provide for their method.\n- The best dataset to transfer from is evaluated only on 1k samples, which is too small. The accuracy changes ~25% to ~70% accuracy for BigEarthNet when going from 1k samples to full dataset. Hence the trend is unpredictable, and some other dataset could have potentially performed better on full dataset.\n- Remove blank page at the end\n​\n### Suggestions to Improve\nFor this to be a comprehensive survey on in-domain representation learning in remote sensing, the reviewer recommends to add:\n​\n1. Standardized representation learning methods to compare across datasets.\n2. Results on more downstream tasks (other than classification; examples mentioned in the paper introduction) to evaluate representation learning.\n​\nThe reviewer understands these could be out of scope of this paper. However in its current state, the contributions are not significant enough.\n​\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the reviewers point out, this paper requires a major revision and fleshing out of the claimed contribution before it is suitable for conference presentation.
Decision: Reject