Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:
The authors present a study on 5 remote sensing datasets where in-domain representation learning is studied and various methods to train and transfer knowledge between them are assessed. In addition, the authors aim to release these 5 datasets with appropriate training and test conditions to the public.
An interesting point of the paper is that ImageNet representations are used to either pretrain, finetune or train networks on each dataset and to transfer that performance to each other dataset. The performance of ImageNet is highly competitive int his context, demonstrating the quality of its features.
The paper is not proposing technical novelties of any kind throughout.

Comments:
-pixel real estate and resolution makes it unclear which dataset is bigger, though the authors attempt to discuss this. The 'winning' dataset (in terms of its usefulness for transfer) seems to have the highest resolution and image size, which may easily be the reason that models can better be trained here.
- Some of the tasks the authors attempt might best be attempted with different models. For example, not every remote sensing task is about classification of a patch. In some cases, segmentation into types of areas would be more appropriate. As such, the basic premise of the paper that the 5 tasks are similar and can be treated as one benchmarkable object seems quite brittle. A better approach seems to surely be to treat each problem with a more tailored approach?
Adding the right inductive biases into each problem may also help mitigate the lack of specific domain data when aiming for high performance in each domain.
- The authors miss a chance to consider this an exercise to perform a version of multitask learning for many of their experiments of similar structure, i.e. aerial images. In many cases satellite images will require similar RGB features. Why should we not use knowledge from all  datasets to get the best features? I was looking forward to seeing a discussion on this topic but unfortunately did not find it.

Decision:
Overall, this paper handles an interesting collection of data, but does not add much interesting novelty to the field of representation learning and fails to leverage some opportunities in those datasets to do sth. interesting (i.e. better transfer and/or better models). 
I also find the approach to tackle all 5 datasets and all problems in each dataset as a similar problem too simplistic. Remote sensing data, as many real-world datasets, is a wonderful playground for rich modeling ideas and inductive biases to make things work with less data or particularly structured data. 
It is commendable to prepare the datasets for publication and the basic results the authors show make for good first baselines here, but ultimately do not meet the bar for novelty or scientific insights.
The paper might be a stronger fit for a targeted workshop on its topic.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ### Summary
This paper discusses transfer learning in remote sensing image-classification tasks.
The key finding is: while transfer learning from ImageNet dataset for remote sensing, performance can be slightly improved if the model weights are fine-tuned on an auxiliary remote sensing dataset (hence in-domain transfer).
​
​
### Strengths
- The paper is well motivated for the need of good representation learning for remote sensing, and is written well making it easy to follow.
- It compiles a nice list of five datasets available for remote sensing, along with their characteristics. Also, the list of common properties of such datasets (e.g. rotation invariant) is good for data pre-processing for engineering applications in remote sensing.
- The paper provides baseline accuracies on five important datasets in remote sensing.
​
​
### Weaknesses
Apart from the contributions of compilation of different datasets which are already present, and classification results on them, there is not much novelty in the paper. There are several major weaknesses:
​
1. **About datasets**: The paper emphasizes a lot on the lack of standard datasets and metrics in remote sensing. However, the contribution is limited to compiling existing datasets and defining a 60-20-20 train-validation-test split on them. There is no new data collection, no insight in how these existing diverse datasets could be combined together, or why this particular definition of split is used. Hence, the reviewer cannot count this as a contribution.
2. **About representation learning**: Since the source domain (ImageNet), in-domain source domain and the target domain all solve classification tasks, a better term to use would be *transfer learning* instead of representation learning. The reviewer recommends to demonstrate how these learned "representations" are used for other downstream tasks or otherwise show some other representation learning methods (semi-supervised or unsupervised), which is not same as the downstream task. Hence, it would be inappopriate to sell it as representation learning, in its current form.
3. **About in-domain learning**: Since the in-domain transfer is also done by fine-tuning an existing model which was trained on ImageNet, it is expected to do somewhat better than Imagenet only. The paper just clarifies this fact in practice, and the performance gains are also statistically insignificant. Again, this is not a significant contribution.
4. **About hyperparameter sweeping**: The minor improvements observed due to in-domain transfer is also unreliable, since very hyperparameters are tried across datasets. E.g. only 2 values of learning rates (0.1 and 0.01) are tested. This could also explain why BigEarthNet and So2Sat had poor training accuracies. The reviewer also disagrees with using only 1k samples for selecting best dataset to transfer from - explained in minor weaknesses.
5. **About experiments**: In-domain transfer is tested by fine-tuning on one dataset and then again fine-tuning on another dataset. It was observed that the dataset RESISC-45 does the best in this scenario. The reviewer disagrees with paper's hypothesis about noisy labels resulting in this artifact. The reviewer suspects that this is because RESISC-45 contains labels which are most appropriate for classification transfer in other datasets (and also it is very balanced). Can the authors comment on this? If another method of representation learning than classification is employed, then it is very possible that other larger datasets result in better representations. The comparison across datasets currently is strange, since all it tells is which dataset has most similar classification labels to other datasets on average. To remove this problem in comparison, the representation learning task and testing task should be different.
6. **About conclusions and contributions**: The experimental setup in this paper is insufficient to claim that it investigates *what characteristics should a dataset have to be a good source for remote sensing representation learning*. Transfer from ImageNet works because it contains a large number of classes, along with data diversity and its sheer size. This cannot be said for the datasets analyzed in this paper, since these factors are not analyzed individually, and the representation learning method is not standardized across all datasets (classification is not standardized since different datasets have different classes), making the comparison unfair.
​
​
​
#### Minor weaknesses:
- Table 4: Since existing BigEarthNet result contains precision/recall, that is what the authors should provide for their method.
- The best dataset to transfer from is evaluated only on 1k samples, which is too small. The accuracy changes ~25% to ~70% accuracy for BigEarthNet when going from 1k samples to full dataset. Hence the trend is unpredictable, and some other dataset could have potentially performed better on full dataset.
- Remove blank page at the end
​
### Suggestions to Improve
For this to be a comprehensive survey on in-domain representation learning in remote sensing, the reviewer recommends to add:
​
1. Standardized representation learning methods to compare across datasets.
2. Results on more downstream tasks (other than classification; examples mentioned in the paper introduction) to evaluate representation learning.
​
The reviewer understands these could be out of scope of this paper. However in its current state, the contributions are not significant enough.
​

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper explored the representation learning problem on remote sensing domain. In particular, it provided several standardized remote sensing data sets with standard train/validation/test splits and various metrics for fairly comprehensive evaluation. It showed that compared to fine-tuning on ImageNet or learning from scratch, in-domain representation could produce better baseline results for remote sensing at various training data sizes. 
Overall, I prefer to reject this paper based on the following reasons:
(1) It claimed to address the question “what characteristics should a dataset have to be a good source
for remote sensing representation learning”. But this question has not been fully explored. Empirically, multi-resolution data sets (e.g., RESISC-45) could be used as the base data set for better in-domain representation learning. However, the essential reason on this explaining this phenomenon is not still clear.
(2) The authors did not provide the convincing evidence regarding how to generate the standard train/validation/test splits for all the data sets. The data split might be the reason to explain the superior performance of in-domain representation learning.
(3) On evaluating the in-domain representations (shown in Table 2), all the representation models were trained with 1000 training examples for performance comparison. It is not convincing whether the best in-domain representation models are affected by the training size. Later, it used the best in-domain representation models to estimate the state-of-the-art baselines on various training size. It is not clear how to select the best in-domain representation models.
Minor comments:
(1) In Section 5.5, the authors trained the model with randomly sampled training examples. It might be more convincing if they can show the mean and variance of model performance using multiple randomly sampled examples.

review_title: Official Blind Review #1

Meta-review: As the reviewers point out, this paper requires a major revision and fleshing out of the claimed contribution before it is suitable for conference presentation.
Decision: Reject