GEO-Bench: Toward Foundation Models for Earth Monitoring
Authors: Alexandre Lacoste
Nils Lehmann
Pau Rodriguez
Evan David Sherwin
Hannah Kerner
Björn Lütjens
Jeremy Andrew Irvin
David Dao
Hamed Alemohammad
Alexandre Drouin
Mehmet Gunturkun
Gabriel Huang
David Vazquez
Dava Newman
Yoshua Bengio
Stefano Ermon
Xiao Xiang Zhu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: computer vision
earth monitoring
earth observation
dataset
benchmark
TL;DR: Benchmark containing 6 classification and 6 segmentation datasets tailored to evaluate large pretrained model on remote sensing data

Abstract: Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks. 
Such models, recently coined foundation models, have been transformational to the field of natural language processing.
Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited.
To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results to enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models.
We believe that this benchmark will be a driver of progress across a variety of Earth monitoring tasks.

GEO-Bench: Toward Foundation Models for Earth Monitoring
Alexandre Lacoste*
Nils Lehmann" 22
Pau Rodriguez!
Evan David Sherwin
Hannah Kemer*
Bjöm Lütjens*
Jeremy Irvin
David Dao
Hamed Alemohammad
Alexandre Drouin 1.8
Mehmet Gunturkun
Gabriel Huang 1.9
David Vazquez'
Dava Newman
Yoshua Bengio
Stefano Ermon
Xiao Xiang Zhu?
ServiceNow Research
2 Technical University of Munich 3 Stanford University
Arizona State University Mila-Quebec
MIT ETH Zurich 7 Clark University
9 Uniiersityy of Montreal
Abstract
Recent progress in self-supervision has shown that pre- training large neural networks vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks. Such models, recently coined foundation models, have been transformationll the field of natural language processing. Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited. To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models. We believe that this benchmark will be driver of progress across a variety of Earth monitoring tasks.
1 Introduction
Earth monitoring with machine learning- bused methods plays an increasing role in climate change mitigation and adaptation well as climate science [57]. Related applications include methane source detection [61, 16), forest carbon quantification [44], extreme weather prediction [49], and crop monitoring [34, 14). Across many of these applications, pre-trained models (e.g., a ResNet trained on ImageNet) have been used to increase generalisation performance. Improvement of the pre-trained models has been shown reduce the need for large labelled datasets in some contexts [11] and can improve model generalisation outside of the training distribution [28]. Recent studies exploring the scaling of such pre-trained models found that increasing the size of an unsupervised (or weakly supervised) dataset as well as properly scaling the model led to an even greater increase in performance under various metrics [33, 55].
While the training of such large-scale models is usually reserved for industrial research groups with very large computer clusters, the publication of pre-trained models creates vast opportunities for the entire research and technology community (including communities of domain experts outside of machine learning). These large pre-trained models were recently coined as foundation models [6] as they might serve as foundations for sub-fiet fieldiillsds of machine leaming. Specifically, the publication of large pre-trained models like BERT [15] and GPT-3 [7] led to a paradigm shift in the field of natural language processing (NLP). This inspired similar shift in the field of computer vision with the release of models like CLIP [55] and DINO [9]. While CLIP performs well on various types of vision tasks, it still under-performs
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datascts and Benchmarks.

---

on Earth monitoring tasks [55]. This is not surprising as it is trained mainly on RGB images taken from a ground perspective a single point in time.
While there are many similarities between Earth observation datasets and typical ML. image datasets, there also many important differences consider when designing effective MIL models. Earth observation images are taken from an overhead rather than ground perspective, usually from fixed distance from the Earth's surface (defined by a satellite's orbit). The satellite revisits provide a temporal axis that is sometimes irregular (e.g,, a few times per year) or regular (e.g.. every five days) with cloud coverage causing spurious occlusions. Images are acquired with sensors containing multiple spectral bands (e.g.. thirteen for Sentinel-2), or even with different kinds of sensors, e.g., synthetic aperture radar (SAR), which can penetrate cloud coverage. Moreover, the GPS coordinates and timestamp of each acquisition offer the opportunity to combine data from multiple sources, e.g., weather data, semantic maps, and elevation. This leads a rich mudti-modal signal with potentially missing information that can be inferred from other elements of the signal. There are currently petabytes of accessible satellite datasets containing images of the Earth under various modalities from the present day to as far back as the 1960s. Distilling this large amount of information into pre-trained models of various sizes offers the opportunity redistribute this information and make accessible to various labs for increasing the performances on large range of downstream tasks.
The fundamental goal of these large pre-trained models is to improve generalization performance on downstream tasks, Hence, to support the machine learning community in producing better pre-trained models, it is crucial to provide a benchmark with a wide variety of downstream tasks, covering a range of modalities and dataset shapes that are likely to be encountered in practice. At the moment, existing works on pre-training models from earth observations e.g., [13, 46, 69), evaluate on different sets of downstream tasks, making it impossible to directly compare performance. Moreover, the set of tasks often narrow in terms of diversity and the statistical methodologies do not adequately report the uncertainties in the evaluation.
The present work aims fill this void by providing a wide range of tasks across various countries with vari- ous modalities of sensors. Also, the transformed versions of the datasets are smaller than their original form, and all results can be replicated on single GPUs. This increases accessibility to research labs with limited resources and reduces overall energy consumption. Our proposed benchmark, GEO-Bench is composed of six image classification and six semantic segmentation tasks, which were curated by domain experts to ensure their diversity and relevance toward sustainable development. We expect this contribution to:
Stimulate and facilitate the development of foundation models for Earth monitoring
Provide a systematic way of measuring the quality of models for better scientific progress
Provide insights into which pre-trained models work best
Potentially reduces negative impacts of foundation models through open evaluation procedure.
In what follows, we start by discussing sources of data that can serve to train foundation models for earth monitoring (Sec. 2). We then present the details of GEO-Bench (Sec. 3) and how it can be used for the evaluation of foundation models (Sec. 4). Further, we review existing benchmark datasets for earth monitoring and discuss why GEO-Bench is complementary (Sec. 5). Finally, we present extensive set of experiments, showing the performance of 20 state-of-the-art models on the benchmark to lay down reference points and to gain valuable information on existing pre-trained models (Sec. 6).
2 Remote sensing data for self-supervision
The development of foundation models does not typically rely on specific dataset for the pre-training phase. The choice of data source part of the design of the model, e.g., a very large corpus of text from the internet [50] or pairs of text associated with images from the web [55]. As such, we do not provide data for training foundation models with this benchmark. However, for completeness, we outline potential sources of Earth observation data that could be used for pre- pre-training foundation models.
Multispectral images with revisits Satellite data sources such as Sentinel-2 [20, 23] and Landsat 8 [66] provide images in multiple spectral bands with periodic revisits. This yields a four-dimensional array of structured data (longitude, latitude, wavelength, time) which can be used to perform various forms of self- supervision, e.g., predicting adjacent tiles [30] or contrasting the different seasons for the same region [46].
'nttts:://zenon www.o/////o/ice///eeeeeeeeeeeeee bench

---

Segatti
Large
Penstrades
Melkase pluse
Fillin dail
Pretrained
Model
Fysle
Figure 1: Foundation models encapsulate multimodal data streams through self self-super training. The trained models can then fine-tuned for a variety of climate-related remote sensing tasks. Image sources: quantification [44], detection [32], generation [43], counting [36], segmentation [75], and multi-class classification [51].
Other sensors Synthetic Aperture Radar (SAR) and terrain elevation are also frequently available and can be matched to other sources of data through geolocalisation [54]. Such data are complementary to optical spectral bands and may encourage the model to learn higher-level semantic representations.
Semantic data Through geoneferencing, text-based data such as Wikipedia articles can be linked to satellite images [67]. is also possible join content from non-image data layers like OpenStreetMap [39]. By predicting or contrasting information from these sources, the model may learn useful and transferable semantic representations.
3 GEO-Bench
GEO-Bench is composed of 6 classification tasks and segmentation tasks. Detailed characteristics are presented in Table 1, examples are depicted in Figure and 3, and the spatial coverage on the world map is presented in Figure (supplementary material). what follows, we describe the procedure for collecting and transforming the datasets.
3,1 Design Principles
GEO-Bench was established by modifying and gathering geospatial datasets, adhering to principles that secure accessibility, usability, and effective model performance assessment across tasks.
Ease of Use fundamental goal was to create accessible, simple-to-use benchmark, and compact dataset assortment with code for loading the data a consistent schema. A key aim was to harmonize data reduce the engineering work needed tailor pre-trained architectures, while maintaining sensor type and resolution diversity.
Sector Experts and Steering Committee a team of six sector experts from fields such as forestry and climate science. A steering committee of respected scientists guides high-level benchmark decisions, assuring relevance and impact.
To align GEO-Bench with practical use-cases, assembled
The objective is evaluate model adaptability to varied geospatial sensors. Thus, the benchmark encompasses multispectral, SAR, hyperspectral, elevation, and cloud probability modalities, with spatial resolutions from 0.1 to 30 m/pixel.
Diversity of Modalities
Diversity of Tasks We ventured beyond image classification, incorporating object detection and semantic segmentation. To maintain of use, detection and counting tasks were transformed into semantic segmen- tation. This led to two task sets: six image classification tasks, and six semantic segmentation tasks [25, 38].
Original Train, Validation, and Test Splits Original dataset splits were preserved when available; otherwise, we generated validation and test sets from the train set while ensuring no spatial overlap.
Permissive License Most datasets needed be adapted from their original form to satisfy the above criteria and be included in the benchmark. Hence, we include only datasets with permissive licenses.

---

Classification
Name
Image Size
Classes
Tnin
Val
Test
# Bands
RGB res
Sensors
Cite
License
m-bigearthnet
120 120
43
20000
1000
1000
10.0
Sentinel-2
[14]
CDLA-P-1.0
m=s02sat
32
17
19992
986
986
18
10.0
Scatinel-2
CC-BY-00
brick. kiln
64 64
2
15063
999
999
13
10.0
Sentinel 2
SC 4.0
ne-forestnet
332
CCBBY-D
m-curosat
64 64
10
2000
1000
1000
13
10.0
Scatinel-2
MIT
m-pu-4ger
x 320
2
11814
3
0.1
RGB
MIT
Segmentation
Name
Irage Size
Classes
Trinn
Val
Test
# Bands
RGB res
Sensors
Cit
License
n--wegger-seg
320
2
3
RGB
[08]
m- coehheppeekee landonver
256 256
7
3000
1000
1000
4
1.0
RGBN
CDLA.P-1.
cashew plantation
Seminee-22
[1]
mn-A Ln-AA-crop-type
256.x 256
10
3000
1000
1000
13
10.0
Sentinel-2
liak
CC-BY-1O
500
2
524
3
RGB
CC-BY40
m-NeonTree
x00xx 400
2
270
5
0.1
RGB
[T]] CC
Hlpeerspectral
Elevation
Table GEO-Bench: Characteristics of datasets in the benchmark. Since datasets are modified, we prepend their name with "m-" to distinguish them from the original dataset.
m--forsstnct
m-bricckiln
BATTO
m-scZsat
m-pväger
bigearthnet
Figure 2: Representative samples of the classification benchmark.
m-chesapeake
m-NeonTres Treee
m-iz-cattle
pusggenseg
glanttoono
m-SAverop-tye type
Figure 3: Representative samples the segmentation benchmark.
3.2 Dataset Transformations
To produce benchmark that complies with the design choices of Section 3.1. applied the following transformations to each dataset. The procedure that was used to download and transform each dataset fully documented and open-sourced in the GEO-Bench GitHub repository?.
hhttps: htttpp///ggthaaccc//e//rrr/r///e

---

Subsampling Large Datasets To be more representative of typical downstream tasks, where data is usually scarce, datasets larger than 20000 samples were randomly subsampled. Avoiding large downstream tasks also comes with other benefits:
In Appendix A, we show that larger downstream datasets can decrease the ability to discriminate between two models that are similar in performance.
Downstream tasks with very large training sets will not usually benefit from pre- pre-training' Hence they are less useful for our evaluation purpose.
A smaller benchmark is faster download, yields results quicker and requires less energy for computation.
We can increase the variety of experiments and the number of seeds to improve the knowledge gained from experiments.
Removing Class Imbalance We randomly subsampled large classes to have near-uniform class sizes across datasets. This was done prevent users of the benchmark from increasing their score by using clever class imbalance techniques instead of making progress on better pre-trained models. While good performance on highly imbalanced (long tail of classes) datasets would be a desired property of a pre-trained model, we have not found a good dataset containing a large number of classes.
4 Using The Benchmark
Fine Tuning In the self-supervised learning literature, it is common to use the pre-trained model to encode a fixed representation of each image in the dataset and learn to classify images based on this representation [30]. While this works relatively well, this method highly depends on the pre-training task as it may not learn to encode information that is important for the downstream task [65. 53]. In practice, fine-tuning the pre- pre-trained model often mitigates this issue and is known to frequently yield a much higher generalization performance than a model trained from random weights [46, 1]. Since this more representative of practical usage, we encourage users of the benchmark to report the results of fine-tuned models. On the other hand, we do not discourage users from also reporting results with fixed backbones (pre-trained weights) as this can provide valuable information about the pre-trained model. In all cases, we ask users to report their fine-tuning methodology with enough details for reproducibility.
Hyperparameter Tuning Deep learning algorithms often require the adjustment of hyperparameters, especially when an architecture is fine-tuned on small dataset. For this reason, we recommend adjusting hyperparameters, but within a maximum budget of 16 trials per task". Early stopping based on validation metrics is also recommended.
Data Augmentation Data augmentation plays a cricial role in the training of deep learning models, especially with small training datasets. Hence, we consider it to be part of the fine-tuning process, As a guideline, we propose limiting the augmentations to 90" rotations and vertical and horizontal flips On the other hand, we also encourage users to study what are the best data augmentations for remote sensing as this could lead to useful findings for practitioners and the benchmark is well-suited for evaluating such findings.
Toolbox To facilitate the usage of the benchmark, we provide a collection of tools for various parts of the experimental pipeline as part of the open- open-sourced codebase This includes tools for loading datasets and visualising results. We also provide tools based on Py Ligghh-Light [24] facilitate model training.
4.1 Reporting Results
For reliable and comparable results across different publications, we recommend that users follow this pro- cedure report results. The aim is report results on individual tasks as well as aggregated across all tasks, with reliable confidence intervals (inspired by [2]) Code is provided to generate figures based on raw results.
FFrom Bayes rule, we know that the influence of the prior (pre-trained model) decreases as the size of the training data increases.
'While 16 is fairly small, we believe it's enough to adjust sensitive hyperparameters such as learning rate. Also, this favours models that are less sensitive to hyperparameter tuning.
'Random crop and resize are also common in vision, but remote sensing, this reduces spatial resolution, which is often cnucial for high performances.
'nttps: http///gttttut/t www.//orvicelo//geooooooooo
5

---

Random Seeds As demonstrated in [2], 3-5 seeds are not enough to obtain reliable confidence intervals. Since pre-training and hyperparameter search are usually the computationnl bottlenecks, we recommend retraining the selected hyperparameter configuration for at least 10 different seeds.
Interquartile Mean (IQM) We recommend using IQM. This metric removes the outliers by trimming the 25% highest values as well as the 25% lowest value and computing the average of the remaining values. The resulting finite sample estimator is less biased than the median and has less variance than the mean, often resulting in smaller confidence intervals [2].
To aggregate performance metrics across multiple tasks, one must first normalise their values. A common approach consists of applying a linear transformation based on reference points [4]. As such, we propose to use the lowest and highest metric values achieved by set of strong baselines (see Sec. 6) as official reference points. For each individual task, we scale the results such that the maximum score is I and the lowest one is O. Hence, if future model were to achieve a score superior to 1, it would imply that progress is being made on the benchmark. All reference points will be published alongside the benchmark.
Normalising Results
Bootstrapping To quantify uncertainty over observed IQMs, we use bootstrapping [21]. That is, we sample n times, with replacement, the results from training with n different seeds, and we compute IQM. Repeating this procedure n= 1000 times provides distribution over IQM results, from which confidence intervals can be extracted.
Aggregated Results After normalizing the results simply compute IQM across all datasets and all results of a given model. For confidence intervals, we use stratified bootstrap, where seeds are sampled with replacement individually for each dataset, but IQM is computed across all datasets.
Displaying the results In Figure 4, we show how compactly display results from wide range of baselines across the benchmark as well as aggregated results and statistical uncertainties. In Figure 5, we display the results for a growing training set size (with fixed validation and test set). This compactly reports the results of thousands of experiments.
Publishing the results We ask experimenters to publish the results of all seeds on all datasets for all models as a CSV file along with the open-sourced code of their experiments. This will allow future authors to incorporate existing results in their comparison figures.
5 Related Works
SustainBench consists of 15 public datasets covering 7 sustainable development goals [73]. Seven of these datasets are two-dimensional remote sensing. It includes a public leaderboard for tracking model performance. A featured task is the Brick Kiln classification, selected for its georeferenced, high-quality ground truth labels. SustainBench's purpose is monitoring progress in specified tasks, thus comprising diverse set of datasets. It doesn't aim for solution under single framework aggregate result tracking.
TorchGeo is a Python library designed to streamline the integration of remote sensing datasets into the PyTorch deep learning ecosystem [62]. TorchGeo currently features data loaders for 52 publicly available datasets of satellite, aerial, and drone imagery for classification, regression, change detection, semantic segmentation, instance segmentation, and object detection tasks. Our benchmark directly interfaces with TorchGeo and uses its data loaders for several datasets included in the benchmark.
NerthNets is a concurrently developed platform to evaluate deep learning methods on remote sensing datasets [72]. In their methodology, they analyse the metadata of 400 publicly available remote sensing datasets. Using meta-informatinn such as the number of samples, the size of each sample, and the type of annotations, they analyse the correlation between each dataset and identify a variety of clusters. Based on this analysis, they recommend two classification, two segmentation, and two detection datasets for benchmarking. In contrast, we provide collection of 12 datasets and we propose robust methodology for aggregating results and reporting statistical uncertainties of the evaluation process.
AÍTLAS recently proposed benchmark of 22 classification datasets [17]e 3 of which intersect with our classification benchmark. They proposed a standardised version of train, valid, test splits for existing datascts as well as fine-tuning procedure. By leveraging the overlap of labels across datasets, they also

---

provide more accurate test metric for real-world applications. Experiments are conducted using 10 different model families across the 22 datasets, using RGB images input.
Experiments
ResNet18-And
ResMet1B-MoCo-52 ResNet50-SECO-5
ResNet50-McCo-S2 ResNet50-timm
ConvNeXt VIT-T-imm
ResNet18 8-timm
SwinV2-TTtimm
0.50
0.40
0.0
0.00
aggregated
bigearthnet
brick-kklln
forestnet
m-pväger
m-so2sat
Classification Benchmark RGB Only: Normalised accuracies of various baselines (higher better). Violin plots are obtained from bootstrap samples normalized IQM (Section 4.1). The left plot reports the average across all tasks.
Figure 4:
ResNet18-Rnd
ConvNeet-SStimmn timm
VIT-S-timm
noma motric
1.0
0.5
0.0
-05
10
1
10"10"1
10
10"10"4
10"10
10"10
10
10"10
10
10"10
10
aggregated
m-bigearthnet
m-brick-kin
m-eurosat
m-forestnet
m-pv4ger
m-so2sat
Figure 5: Classification vs Train Size: Normalised accuracies of a subset of the baselines on Classification benchmark with growing size of the training set. The shaded region represents 80% confidence interval, obtained from bootstrap samples of normalized IQM (Section 4.1). The left plot reports the average across all tasks.
Reedeer Sp4imm S0ctiem R.Muh
MTS Emm V11-5-Emm 11:5-8mm-----0mm
VIT MoCo 52 mult 11-5-DNN--52-m-
Racker550.D100-522m
come bood bost meett
0.to
3.50
0.40
spgrogated bigcarthintm curozat
aggregated m-biggartheet m-curoset
Figure 6:
Effeet of Multispectral with ResNet50, Only Sentinel-2 tasks are reported. Normalised accuracies (Sec 4.1..
Figure 7: Effect of Multispectral with VIT-S. Only Sentinel-2 tasks are reported. Normalised accuracies (Sec 4.1).
In this section, provide range of baselines for the classification and segmentation benchmarks. These will serve as reference points for future evaluation?. We also seek to answer the following questions:
Which architecture performs best remote sensing data (Section 6.2.2)?
What is the effect of training set size on the performance of each model (Section 6.2.3y?
Can we leverage multispectral channels to improve performance (Section 6.2.4)?
Are smaller datasets better at discriminating the performance of different models (Section A.5)?
We recall that all datasets have been modified from their original version. Hence, our results are not directly
comparable to other published results.
7

---

6.1 Protocol
For each model, we replaced the last layer with a randomly initialised layer of the appropriate shape for the task at hand. We use different learning rates for the last layer (which starts from random weights) and for the hackhone (which starts from pre-trained weights). The best learning rates were selected using the highest accuracy or Intersection over Union (loU) the validation set over 16 trials 8 After choosing the hyperparameters, we repeated the training for 10 seeds. To minimize overfitting, we selected the best time step using accuracy (or IoU) on the validation set and we reported the test metrics the chosen time step. We use AdamW [42] to train convolution architectures and SGD to train transformer architectures.
6.2 Classification
6.2.1
Baselines Naming Schema
Each baseline name starts with the corresponding architecture: ResNet18 and ResNet50: standard ResNet architectures [26]; ConyNeXt-B: the base architecture of ConvNeXt [41]: ViT-T and VIT-S: VIT
architectures [19] of size tiny and small respectively; V2-T: a SwinV2-tiny architecture [40];
Then, keywords provide details about the training procedure: SeCo: a ResNer50 model trained on Sentinel 2 data with temporal contrastive loss acToss seasons [46]; MoCo-S2 and DINO-S2: model trained with self-supervision on Sentinel data [70] (RGB and Multispectral pre-trained weights); Rud: weights are randomly initialised: timm: pre- pre-trained weights are obtained from the timm library, usually from training on ImageNet; +R-Multi: we manually augment an RGB architecture by randomly initialising the weights of the missing channels in the 1st layer; multi: the pre- pre-trained model has multispectral channels.
6.2.2 Comparing Baselines on RGB only
In Figure 4, we report bootstrapped IQM of the normalized accuracy (Sec 4.1) for the six datasets of the classification benchmark, as well as aggregated results". this first experiment, all models can only see the RGB channels.
These results offer valuable information across 10 common baselines in the literature. We denote the outstanding performance of ConvNext and SwinV2 compared to other models. It is by a large margin the best models in aggregated results and almost systematically outperforms all models on all datasets. We can also observe the large difference between Scratch ResNet18 8 and ResNet18 on all datasets. This highlights the importance of using pre-trained model. Also, perhaps disappointingly. the existing model pre-trained on remote sensing data does not exhibit any improvement compared to their timm pre- trained weights, and ResNet50-SeCC
i.e., ResNet18 MosNeet8-Mooo-S2 MoCo--2, on the aggregated performance. On the other hand, in Section 6.2.4, we see that ResNet50-MoCCo S--mmii can leverage multispectral data to slightly surpass ResNet50-timm.
are all comparable to ResNet18
Another insight that can be gained from these results is how useful a dataset is at discriminating baselines, i.e., dataset where most baselines perform equally would have limited utility in our benchmark. To this end, we had to discard GeoLifeClet 2022 [12] as all models were performing equally badly m-eurosat also offers limited discriminativity as most models obtain very high accuracy (see Figure 9). To make this dataset harder, we subsample down to 2000 training samples. We can now see that smaller models tend to perform better on this dataset, but the discriminativity remains fairly low.
6.2.3 Accuracy vs training set size
As part of the benchmark, we also provide official subsets of the training sets with train ratios of (0.01, 0.02, 0.05, 0.1, 0.2, 0.5. 1) "!.
'The range of selected leaminng rates is different for each model and is selected based on early experiments, see appendix for details.
'W note that the variance of the results represents the uncertainty the mean (IQM) which is significantly smaller than the variance of the raw seeds presented in Figure 9 in Appendix. We suspect this dataset have high aleatoric uncertainties.
Reeprrting results on all 7 subsets increases the number of experiments by 7x. However, in Figure 13 (see Appendix), we show that the convergence time is proportional to the training set size. This means that training on all seven subsets takes on average about 1.88 times longer than just training on the full training set.
8

---

Figure 5 depicts a different perspective on the models. First, we can observe the noise due to the hyperparameter selection process that not accounted for by repeating 10 seeds with fixed hyperparametees. Also, we see that ConvNeXt often becomes better than SwinV2 as the training set decreases. This coincides with the common observations that transformer architectures tend be more data-hungry, but also tend to outperform convolution architectures in the high data regime [18]. We note also, that ConwNeXt-B-timi only requires 2% of the training set to obtain aggregated performances comparable to that ResNet18 18- 88Rnd. This impressive factor of 50x on data efficiency highlights the importance of developing new architectures and new pre-training methods. Finally, we can observe an increase in the discriminativity of the datasets as the training set decreases, specifically for m-eurosat, when the task becomes more difficult, the strong baselines stand out even more. The discriminativity of datasets is further studied in Section A.5.
6.2.4 Leveraging Multispectral Information
We now study the effect of leveraging multispectral information during the pre-training phase and during the fine-tuning phase. We do so by fixing the backbone to either ResNet50 (Fig. 6) or ViT-S (Fig. 7) and exploring various weight initialisation schema. Since we could only find pre-trained models for Sentinel-2, we limit this experiment to the four datasets satisfying this criterion.
We found that using a model pre-trained on RGB-only (timm pre-trained) and augmenting the architecture by randomly initialising the weights of the missing channels in the first layer (+RMulti) does not lead to systematic improvement. Moreover, the fine-tuning time is largely extended since we have to wait until the newly initialised weights on the first layer fully converge. On the other hand, the ResNet50 pre- preitrained on Sentinel-2 using DINO or MoCo [70] leads to a modest performance increase on average. When looking ViT-S (Fig. 7), incorporating multi multi-specctal only leads to a systematic performance decrease.
6.3
Segmentation
We defer experiments on the Segmentation benchmark to Appendix A.3, where we provide experiments on six baselines 8 ResNet50, ResNet101) x (U-Net, DeepLabV3) with pre-trained weights provided by the timm library. While ResNetl01-DeepL.ab5 performs best in aggregate, it still underperforms on some datasets.
6.4 Resource Usage
See Appendix A.6 for detailed resource usage of each algorithm evaluated in this section. We report the number of parameters, memory usage, the time required for a forward pass, and the convergence time for fine-tuning on downstream tasks. While memory footprint can increase by factor of 4x for a model like V2 and ConvNeXt-B compared to ResNet50, their forward pass is only twice as slow.
7 Conclusion
We developed new benchmark for evaluating pre-trained models remote sensing downstream tasks. This involves adapting a variety of remote sensing datasets to a more conventional machine learning pipeline and providing code for fine-tuning and evaluating individual tasks. We expect that this benchmark will stimulate the development of new foundation models that could lead to better generalization on a variety of earth monitoring downstream tasks and could open up opportunities for new applications.
Limitations Our benchmark does not extensively evaluate all desired features of a pre-trained model for earth monitoring. For example, it does not evaluate its ability to fine-tune temporal data nor perform fusion with other types of data such as text or weather. The spatial coverage of the benchmark covers most continents and improves coverage over individual datasets. However, the spatial coverage could still be largely improved to include a much wider range of countries and biomes. Finally, as pre-trained models become stronger, they will get closer to the theoretical limit of generalization performance, i.e. approaching the alcatoric uncertainty of the dataset. Under such a regime, we expect a bigger overlap between error bars when comparing 2 different models.
References
[1] Diab Abuaiadah and Alexander Switzer. Remote sensing dataset for detecting cows from high resolution aerial images. 2022.

---

[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
[3] Hamed Alemohammad. The case for open-access. ML-ready geospatial training data. International Geoscience and Remote Sensing Symposium. IEFE, 2021.
[4] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Reseurch, 47:253-279, 2013.
[5] Emily M Bender, Timnit Gebru, Angelina McMillan- -Mjjor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623, 2021.
[6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. arXiv preprint arXiv:2005. 14/65, 2020.
[8] Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon. Using satellite imagery to understand and promote sustainable development. Science, 371(6535), 2021.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sull-supervised vision transformers. arXiv preprint arXiv:2104. (1ti::2004..2244,,, 2021.
[10] Liang- Liang-Chich Chen, George Papandreou, Florian Schroff. and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint orXiv:/706.055550 2017.
[11] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pages 1597-1607, PMLR, 2020.
[12] Elijah Cole, Benjamin Deneu, Titouan Lorieul, Maximilien Servajean, Christophe Botella, Dan Moris, Nebojsa Jojic, Pierre Bonnet, and Alexis Joly. The geolifeclef 2020 dataset. arXiv preprint arXiv:2004.04/99 2020.
[13] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Manshall Burke, David B Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. arXiv preprint arXiv:2207.0800: 2022.
[14] Walter T Dado, Jillian M Deines, Rinkal Patel. Sang Zi Liang, and David B Lobell. High-resolution soybean yield mapping across the us midwest using subfield harvester data. Remote Sensing, 12(21))3471, 2020.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11X1v:1810.04805, 2018
[16] Sonu Dileep, Daniel Zimmerle, Ross Beveridge, and Timothy Vaughn. Automated identifcctionn of oil field features using cnns. 2020.
[17] Ivica Dimitrovski, Ivan Kitanovski, Dragi Kocev, and Nikola Simidjicvski. Current trends in deep learning for carth observation: An open-source benchmark arena for image classification. ISPRS Journal of Photogrammetry and Remote Sensing, 187:18-35, 2023.
[18] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathhas Minderer. Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint arXiv:2010. 11929, 2010.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikow, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[20] Mathias Drusch, Umberto Del Bello, Sébastien Carlier, Olivier Colin, Veronica Femanndez Ferran Gascon, Bianca Hoersch, Claudia Isola, Paolo Laberinti, Philippe Martimort, al. Sentinel-2: Esa's optical high-resolution mission for gmes operational services. Remote sensing of Environment, 120:25 36, 2012.
[21] Eforn. Bootstrap methods: another look at the jackknifee The Annals of Statistics, 7:1-26, 1979.
10

---

[22] EPA. Greenhouse Gas Emissions: Understanding Global Warming Potentials.
Technical report, US
Environmental Protection Agency, February 2017.
[23] ESA. Sentinel-2. Technical report, European Space Agency, Paris, France, 2021.
[24] William Falcon and The Py PyTorch Lightning team. PyTorch Lightning, 3 2019.
[25] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 32(9):1627-1645, 2010.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compiter vision and pattern recognition, pages 770-778, 2016.
[27] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[28] Dan Hendrycks. Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self superviiee learning can improve model robustness and uncertainty. arXiv preprint arXiv: 1906. 12340, 2019.
[29] Jeremy Irvin, Hao Sheng, Neel Ramachandran, Sonja Johnson- Yu, Sharon Zhou, Kyle Story, Rose Rustowicz, Cooper Elsworth, Kemen Austin, and Andrew Y Ng. Forestnet: Classifying drivers of deforestation in indonesia using deep leamiing on satellite imagery. arXiv preprint arXiv:201 1,05479, 2020.
[30] Ncal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tile2vee: Unsupervised representation leaming for spatially distributed data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3967-3974, 2019.
[31] Forrest Johnson, Andrew Wlazlo, Ryan Keys, Viren Desai, Frin Wetherley, Ryan Calvert, and Flena Berman. Airhome methane surveys pay for themselves: An economic case study of increased revenue from emissions control. preprint, Envinonmental Monitoring. July 2021.
[32] Siraput Jongaramrungruang. Christian Frankenberg, Andrew K. Thorpe, and Georgios Matheou. Methanet an di-driven approach to quantifying methane sounc-source emission from high resolutionn 2-d plume imagery. ICML Workshop on Tackling Climate Change with AI, 2021.
[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffiey Wu, and Dario Amodci. Scaling laws for neural language models. arXiv preprint arXiv:2001 08361, 2020.
[34] Hannah Kemer. Gabriel Tseng, Inbal Resker,Reshef, Catberine Nakalembe. Brian Barker, Blake Munshell. Madhava Paliyam, and Mehdi Hosseini. Rapid response crop maps in data sparse regions. arXiv preprint arXiv:2006. 166666 2020.
[35] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine leamiig. arXiv preprint arXiv:9910.097000 2019.
[36] Issam Laradji, Pau Rodriguez, Freddie Kalaitzis, David Vazquez, Ross Young, Ed Davey, and Alexandre Lacoste. Counting cows: Tracking illegal cattle ranching from high-resolution satellite imagery, arXiv preprint arXiv:2011.07369 2020.
[37] Jihyeon Lee, Nina R. Brooks, Fahim Tajwar, Marshall Burke, Stefano Ermon, David B. Lobell, Dehashish Biswas, and Stephen P. Luby. Scalable deep learning to identify brick kilns and aid regulatory capacity. Proceedings of the National Academy of Sciences, 118(17), 2021.
[38] Victor Lempitsky and Andrew Zisserman. Learning to count objects images. Advances in neural information processing systems, 23, 2010.
[39] Haifeng Li, Xin Dou, Chao Tao, Zhixiang Wu, Chen, Jian Peng. Min Deng, and Ling Zhao. Rsi-ch: large-scale remote sensing image classification benchmark using crowdsounced data. Sensors, 20(6):1594, 2020.
[40] Ze Liu. Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong. et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12009-12019 2022.
[41] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976 1776-11986 11966, 2022.
11

---

[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05001. 2017.
[43] Bjöm Lütjens, Brandon Leshchinskiy. Christian Requena-Mesa, Farrukh Chishtie, Natalia Diaz-Rodriguez, Océane Boulais, Aruna Sankaranarayanan. Aaron Pina, Yarin Gal. Chedy Raissi, Alexander Lavin, and Dava Newman. mnysically-coniistent generative adversarial networks for coastal flood visualization. ICML Workshop on Al for Modeling Oceans and Climate Change (AIMOCC), 2021.
[44] Björn Lütjens, Lucas Liebenwein, and Katharina Kramer. Machine leariin-basse estimation of forest carbon stocks increase transparency of forest preservation efforts. 2019 NeurlPS Workshop on Tackling Climate Change with (CCAI), 2019.
[45] Lei Ma, Yu Liu, Xueliang Zhang, Yuanxin Ye, Gaofei Yin, and Brian Alan Johnson. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS journal of photogrammetry and remote sensing, 152:166-177, 2019.
[46] Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Scasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9414-9423, 2021.
[47] M Maskey, Alemohammad, KJ Murphy, and R Ramachandran. Advancing ai for earth science: A data systems perspective. Ros, 101, 2020.
[48] Kevin Mayer, Benjamin Rausch, Maric-Louise Arlt, Gunther Gust, Zhecheng Wang, Dirk Neumann, and Ram Rajagopal, 3d-pv-locator: Large-scale detection of mounted photovoltaic systems in 3d. Applied Energy, 310:1 18469, 2022.
[49] Amy McGovem, Kimberly L. Elmore. David John Gagne, Sue Ellen Haupt, Christopher D. Karstens, Ryan Lagerquist, Travis Smith, and John K. Williams. Using artificial intelligence to improve real-time decision-making for high-impact weather. Bulletin of the American Meteomoogical Sociery, 98(10), 2017.
[50] T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. Never-ending learning. Commanications of the ACM, 1155):0031115, April 2018.
[51] Cassandra Pallai and Kathryn Wesson. Chesapeake bay program partnership high-resolution land cover classification accuracy assessment methodology, 2017.
[52] David Patterson, Joseph Gonzalez, Quoc Lc, Chen Liang, Lluis Mluis Munguia, Daniel Rothchild, David So. Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104. 10350, 2021.
[53] Otávio AB Penatti, Keiller Nogueira, and Jefersson A Dos Santos. Do deep features generalize from everyday objects to remote sensing and acrial scenes domains? In Proceedings of the IEFE conference on computer vision and pattern recognition workshops, pages 44-51, 2015.
[54] Karissa Pepin. Howard A. Zebker, and William Ellsworth. High-Pass Filters to Reduce the Effects Broad Atmospheric Contributions in Sbas Inversions: A Case Study in the Delaware Basin. In IGARSS 2020- 2020 IEEE International Geoscience and Remote Sensing Sympasium, pages 1030-1033, Waikoloa, HI, USA, September 2020. IEEE
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. arXiv preprint 1xXiv:2103.00022 2021.
[56] Caleb Robinson, Le Hou, Kolya Malkin, Rachel Soobitsky, Jacob Czawlytko, Bistra Dilkina, and Nebojsa Jojic. Large scale resouutinn land cover mapping with multi- restltiesolltion data. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pages 12726 12736,12255222 2019.
[57] David Rolnick, Priya L Donti Lynn Kaack, Kelly Kochanski, Alexandre Lacoste. Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic Dupont, Natasha Jaques, Anna Haldmann-rowwnr et al. Tackling climate change with machine learning. arXiv preprint at Xiv: 1906.05330 2019.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer intervention. pages 234 234-241. Springer, 2015.
12

---

[59] Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. CodeCarbon: Estimate and Track Carbon Emissions from Machine Leariing Computing. 2021.
[60] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 63(12):54-63, 2020.
Green ai. Communications of the ACM,
[61] Hao Sheng, Jeremy Irvin, Sasankh Munukutla, Shawn Zhang, Christopher Cross, Kyle Story, Rose Rustowicz, Cooper Elsworth, Zattao Yang. Mark Omara, et al. Ognet: Towards a glohal oil and gas infrastructure database using deep learning on remotely sensed imagery. arXiv preprint arXiv:201 1.07227, 2020.
[62] Adam Stewart, Caleb Robinson, Isaac A Corley, Anthony Ortiz, Juan M Lavista Ferres, and Arindam Banerjee. Torchgeo: deep leamingg with geospatial data, arXiv preprint arXiv:21 188872, 2021.
[63] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considcrations for deep learning in nlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-300 3650,, 2019.
[64] Gencer Sumbul, Ame De Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mario Caetano, Begüm Demir, and Volker Markl. Bigearthnet-mnn A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets]. IEEE Geoscience and Remote Sensing Magazine, 93):174--80, 2021.
[65] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839 2020.
[66] USGS. Landsat 8. Technical report, United States Geological Survey, Reston, Virginia, USA, 2021.
[67] Burak Uzkent, Evan Sheehan, Chenlin Meng. Zhongyi Tang, Marshall Burke, David Lobell, and Stefano Ennon. Learning to interpret satellite images using wikipedia. In Proceedings of the Twenty-Fäghth International Joint Conference on Artificial Intelligence, 2019.
[68] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[69] Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, and Dacheng Tao. An empirical study of remote sensing pretraining. IEEE Transactions on Geoscience and Remote Sensing, pages 1-1, 2022.
[70] Yi Wang. Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu. Ss14eo-s12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in earth observation.
[71] Ben G Weinstein, Sarah J Graves, Sergio Marconi, Aditya Singh, Alina Zare, Dylan Stewart, Stephanic A Bohlman, and Fthan P White. A benchmark dataset for canopy crown detection and delineation in co-registered airhome rgb, lidar and hyperspectral imagery from the national ccological observation network. PLoS computational biology, 17())xe0091180 2021.
[72] Zhitong Xiong, Fahong Zhang, Yi Wang, Yilei Shi, and Xiao Xiang Zhu, Earthnets: Empowering ai in carth observation. arXiv preprint arXiv:2210.04934 2022.
[73] Christopher Yeh, Chenlin Meng. Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B Lobell, and Stefano Ermon. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning. arXiv preprint arXiv:21 2021.
[74] Jin Z., Lin C., Weigl C., Obarowski J., and Hale D. Smallholder cashew plantations in benin, 2021.
[75] Valentina Zantedeschi, Fabrizio Falasca, Alyson Douglas, Richard Strange, Matt Kusner, and Duncan Watson-Parris, Cumulo: A dataset for leaminng cloud classes. arXiv preprint 191102:91,.04427, 2019.
[76] Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthiss Häberle, Yuansheng Hua, Rong Huang, al. So2sat lzz42: A benchmark dataset for global local climate zones classification. arXiv preprint 1912:11111.17771 2019.
[77] Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui- Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4):8-36, 2017.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix C
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.6.
4. If you are using existing assets (e.g., code, data, models) curating/releasig new assets...
(a) If your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] They are creative common datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Let's discuss it here. There is no text data and remote sensing data is unlikely contain PIT. The only high-resolution dataset NeonTree and it is collected over protected forests in the United States.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "F47ADJvHy5", "reviewer": "Reviewer_4soD", "summary": "", "questions": "", "limitations": "In the conclusion section, the authors list some aspects of pre-trained models that they have not covered. ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A standardized benchmark for evaluating pre-trained models. This allows for consistent comparisons between different models and facilitates the development of foundation models.\n\n2. The open evaluation procedure enables transparency and comparisons across new datasets and new models.\n\n3. A diverse set of image classification and semantic segmentation tasks that are curated by domain experts to ensure they are also relevant, A wide range of Earth monitoring applications can be tested on this benchmark.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TvS9d1570m", "reviewer": "Reviewer_PQwP", "summary": "", "questions": "", "limitations": "The authors reflect on limitations of the presented benchmark with regards to limited dataset modalities and model generalization power and discuss comprehensively on the potential societal impacts from a variety of angles. ", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The submission provides a very comprehensive benchmark covering a multitude of different model architectures as well as a variety of downstream tasks supported by the benchmark datasets. Its contribution to the broader research community is highly significant and timely considering both the rise of foundation models as the new AI paradigm and the urgency of Earth monitoring technology in the face of climate change. \n\nIn addition to that, the paper is very well structured and describes clearly and transparently the principles of evaluation and the experimental results which demonstrates the high quality of the benchmark and the underlying research.\n\nThe supplementary material provide additional insightful details on baseline results, dataset characteristics and resource usage. In addition, ethical and social implications are comprehensively discussed. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "MVS0KnG0hL", "reviewer": "Reviewer_HGjk", "summary": "", "questions": "", "limitations": "I would add data coverage as another limitation", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "GEO-Bench encompasses 6 classification tasks and 6 segmentation tasks. \nThe paper is novel, well written and does an ample set of analyses. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "YWsIBElVAn", "reviewer": "Reviewer_2NRk", "summary": "", "questions": "", "limitations": "Discuss more about the pros and cons of the datasets and the baseline methods.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset has a variety of image sizes, image classes, and bands for the classification datasets. Each classification dataset has over 6K training samples, around 1000 validation, and 1000 testing samples. The dataset has fewer images, image classes for the segmentation datasets. Each segmentation dataset has a few hundred to 3K training samples, and around tens to 1000 validation and testing samples. The baseline experiments are generally complete.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7ZaFY1TJCs", "reviewer": "Reviewer_xuaP", "summary": "", "questions": "", "limitations": "Yes the authors addressed the limitation and potential negative social impact of their work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The work is significantly relevant to many practitioners using earth observation remote sensing data to conduct research or even day-to-day work, including environmentalists, energy sector researchers, social science researcher,s etc.. The quality of the work is also very high as it not only provides great details of the benchmark and experimental results but also gave specific instructions for future users to store their results in csv and how to visualize them, which is rarely seen in other benchmark paper and frankly should be widely adopted. As this work provides great insight and opportunity for a lot of social work (using geospatial data), I think it has a positive social impact overall. Also, the authors limited the replication of their benchmark to single GPU usage, which is great for preventing this research from only benefiting the larger corporations and enabling smaller labs from less economically developed countries to build on this work and potentially benefit the local community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This is a timely paper and aligns well with the rise of foundation models and the need for strategies to process wide amounts of remote sensing data. Indeed, it relies on existing data, but I think it is a great example of things we can do with remote sensing and I am happy to recommend acceptance. 

The authors have replied to all questions and convinced the reviewers.
Decision: Accept (Poster)