SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For each model, we replaced the last layer with a randomly initialised layer of the appropriate shape for the task at hand. We use different learning rates for the last layer (which starts from random weights) and for the backbone (which starts from pre-trained weights). The best learning rates were selected using the highest accuracy or Intersection over Union (IoU) on the validation set over 16 trials  $^{8}$ . After choosing the hyperparameters, we repeated the training for 10 seeds. To minimize overfitting, we selected the best time step using accuracy (or IoU) on the validation set and we reported the test metrics at the chosen time step. We use AdamW [42] to train convolution architectures and SGD to train transformer architectures.
6.2 Classification
6.2.1 Baselines Naming Schema
Each baseline name starts with the corresponding architecture: ResNet18 and ResNet50: standard ResNet architectures [26]; ConvNeXt-B: the base architecture of ConvNeXt [41]; ViT-T and ViT-S: ViT architectures [19] of size tiny and small respectively; SwinV2-T: a SwinV2-tiny architecture [40];
Then, keywords provide details about the training procedure: SeCo: a ResNet50 model trained on Sentinel 2 data with temporal contrastive loss across seasons [46]; MoCo-S2 and DINO-S2: model trained with self-supervision on Sentinel data [70] (RGB and Multispectral pre-trained weights); Rnd: weights are randomly initialised; timm: pre-trained weights are obtained from the timm library, usually from training on ImageNet; +R-Multi: we manually augment an RGB architecture by randomly initialising the weights of the missing channels in the 1st layer; multi: the pre-trained model has multispectral channels.
6.2.2 Comparing Baselines on RGB only
In Figure 4, we report bootstrapped IQM of the normalized accuracy (Sec 4.1) for the six datasets of the classification benchmark, as well as aggregated results<sup>9</sup>. In this first experiment, all models can only see the RGB channels.
These results offer valuable information across 10 common baselines in the literature. We denote the outstanding performance of ConvNext and SwinV2 compared to other models. It is by a large margin the best models in aggregated results and almost systematically outperforms all models on all datasets. We can also observe the large difference between Scratch ResNet18 and ResNet18 on all datasets. This highlights the importance of using a pre-trained model. Also, perhaps disappointingly, the existing model pre-trained on remote sensing data does not exhibit any improvement compared to their timm pre-trained weights, i.e., ResNet18-MoCo-S2, ResNet50-MoCo-S2, and ResNet50-SeCo-S2 are all comparable to ResNet18 on the aggregated performance. On the other hand, in Section 6.2.4, we see that ResNet50-MoCo-S2-multi can leverage multispectral data to slightly surpass ResNet50-timm.
Another insight that can be gained from these results is how useful a dataset is at discriminating baselines, i.e., a dataset where most baselines perform equally would have limited utility in our benchmark. To this end, we had to discard GeoLifeClef 2022 [12] as all models were performing equally badly<sup>10</sup>. m-eurosat also offers limited discriminativity as most models obtain very high accuracy (see Figure 9). To make this dataset harder, we subsample down to 2000 training samples. We can now see that smaller models tend to perform better on this dataset, but the discriminativity remains fairly low.
6.2.3 Accuracy vs training set size
As part of the benchmark, we also provide official subsets of the training sets with train ratios of (0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1) [1].
The range of selected learning rates is different for each model and is selected based on early experiments, see appendix for details.
<sup>9</sup>We note that the variance of the results represents the uncertainty of the mean (IQM) which is significantly smaller than the variance of the raw seeds presented in Figure 9 in Appendix.
<sup>10</sup>We suspect this dataset to have high aleatoric uncertainties.
<sup>11</sup>Reporting results on all 7 subsets increases the number of experiments by  $7\mathrm{x}$ . However, in Figure 13 (see Appendix), we show that the convergence time is proportional to the training set size. This means that training on all seven subsets takes on average about 1.88 times longer than just training on the full training set.
Figure 5 depicts a different perspective on the models. First, we can observe the noise due to the hyperparameter selection process that is not accounted for by repeating 10 seeds with fixed hyperparameters. Also, we see that ConvNeXt often becomes better than SwinV2 as the training set decreases. This coincides with the common observations that transformer architectures tend to be more data-hungry, but also tend to outperform convolution architectures in the high data regime [18]. We note also, that ConvNeXt-B-timm only requires  $2\%$  of the training set to obtain aggregated performances comparable to that of ResNet18-Rnd. This impressive factor of  $50\mathrm{x}$  on data efficiency highlights the importance of developing new architectures and new pre-training methods. Finally, we can observe an increase in the discriminativity of the datasets as the training set decreases, specifically for m-eurosat, when the task becomes more difficult, the strong baselines stand out even more. The discriminativity of datasets is further studied in Section A.5.
6.2.4 Leveraging Multispectral Information
We now study the effect of leveraging multispectral information during the pre-training phase and during the fine-tuning phase. We do so by fixing the backbone to either ResNet50 (Fig. 6) or ViT-S (Fig. 7) and exploring various weight initialisation schema. Since we could only find pre-trained models for Sentinel-2, we limit this experiment to the four datasets satisfying this criterion.
We found that using a model pre-trained on RGB-only (timm pre-trained) and augmenting the architecture by randomly initialising the weights of the missing channels in the first layer (+RMulti) does not lead to systematic improvement. Moreover, the fine-tuning time is largely extended since we have to wait until the newly initialised weights on the first layer fully converge. On the other hand, the ResNet50 pre-trained on Sentinel-2 using DINO or MoCo [70] leads to a modest performance increase on average. When looking at ViT-S (Fig. 7), incorporating multi-spectral only leads to a systematic performance decrease.
6.3 Segmentation
We defer experiments on the Segmentation benchmark to Appendix A.3, where we provide experiments on six baselines (ResNet18, ResNet50, ResNet101)  $\times$  (U-Net, DeepLabV3) with pre-trained weights provided by the timm library. While ResNet101-DeepLabV3 performs best in aggregate, it still underperforms on some datasets.
6.4 Resource Usage
See Appendix A.6 for detailed resource usage of each algorithm evaluated in this section. We report the number of parameters, memory usage, the time required for a forward pass, and the convergence time for fine-tuning on downstream tasks. While memory footprint can increase by a factor of  $4\mathrm{x}$  for a model like SwinV2 and ConvNeXt-B compared to ResNet50, their forward pass is only twice as slow.
We developed a new benchmark for evaluating pre-trained models on remote sensing downstream tasks. This involves adapting a variety of remote sensing datasets to a more conventional machine learning pipeline and providing code for fine-tuning and evaluating individual tasks. We expect that this benchmark will stimulate the development of new foundation models that could lead to better generalization on a variety of earth monitoring downstream tasks and could open up opportunities for new applications.
**Limitations** Our benchmark does not extensively evaluate all desired features of a pre-trained model for earth monitoring. For example, it does not evaluate its ability to fine-tune temporal data nor perform fusion with other types of data such as text or weather. The spatial coverage of the benchmark covers most continents and improves coverage over individual datasets. However, the spatial coverage could still be largely improved to include a much wider range of countries and biomes. Finally, as pre-trained models become stronger, they will get closer to the theoretical limit of generalization performance, i.e. approaching the aleatoric uncertainty of the dataset. Under such a regime, we expect a bigger overlap between error bars when comparing 2 different models.
[1] Diab Abuaiadah and Alexander Switzer. Remote sensing dataset for detecting cows from high resolution aerial images. 2022.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix C
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.6.
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] They are creative common datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Let's discuss it here. There is no text data and remote sensing data is unlikely to contain PII. The only high-resolution dataset is NeonTree and it is collected over protected forests in the United States.