# PANGAEA: Assessing Geospatial Foundation Models Capabilities through a Global and Inclusive Benchmark

Valerio Marsocci, Yuru Jia, Georges Le Bellier, David Kerekes, Liang Zeng, Sebastian Hafner, Sebastian Gerard, Eric Brune, Ritu Yadav, Ali Shibli, et al.

► To cite this version:

Valerio Marsocci, Yuru Jia, Georges Le Bellier, David Kerekes, Liang Zeng, et al. PANGAEA: Assessing Geospatial Foundation Models Capabilities through a Global and Inclusive Benchmark. IEEE geoscience and remote sensing magazine, 2025, pp.2-43. 10.1109/MGRS.2025.3628194. hal-05397884

HAL Id: hal-05397884

https://hal.science/hal-05397884v1

Submitted on 4 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PANGAEA: Assessing Geospatial Foundation Models Capabilities through a Global and Inclusive Benchmark

Valerio Marsocci<sup>*1,2*</sup>, Yuru Jia<sup>*1,2,3*</sup>, Georges Le Bellier<sup>4</sup>, David Kerekes<sup>3</sup>,
Liang Zeng<sup>2</sup>, Sebastian Hafner<sup>3,5</sup>, Sebastian Gerard<sup>3</sup>,
Eric Brune<sup>3</sup>, Ritu Yadav<sup>3</sup>, Ali Shibli<sup>3</sup>, Heng Fang<sup>3</sup>,
Yifang Ban<sup>3</sup>, Maarten Vergauwen<sup>2</sup>, Nicolas Audebert<sup>6,4</sup>, Andrea Nascetti<sup>3</sup>

¹ ESA Φ-lab

²Geomatics Section, Department of Civil Engineering, Faculty of Engineering Technology, KU Leuven, Ghent, Belgium

³KTH - Royal Institute of Technology Stockholm, Sweden

⁴Conservatoire national des arts et métiers, CEDRIC, F-75141 Paris, France,

⁵University of Glasgow, Glasgow, Scotland

⁶Univ. Gustave Eiffel, ENSG, IGN, LASTIG, F-94160 Saint-Mandé, France

*Equal contribution

**Abstract**—Geospatial Foundation Models (GFMs) have emerged as powerful tools for extracting representations from Earth observation data, but their evaluation remains inconsistent and narrow. Existing works often evaluate on suboptimal downstream datasets and tasks, that are often too easy or too narrow, limiting the usefulness of the evaluations to assess the real-world applicability of GFM s. Additionally, there is a distinct lack of diversity in current evaluation protocols, which fail to account for the multiplicity of image resolutions, sensor types, and temporalities, which further complicates the assessment of GFM performance. In particular, most existing benchmarks are geographically biased towards North America and Europe, questioning the global applicability of GFM s. To overcome these challenges, we introduce PANGAEA, a standardized evaluation protocol that covers a diverse set of datasets, tasks, resolutions, sensor modalities, and temporalities. It establishes a robust and widely applicable benchmark for GFM s. We evaluate the most popular GFM s openly available on this benchmark and analyze their performance across several domains. In particular, we compare these models to supervised baselines (e.g. UNet and vanilla ViT), and assess their effectiveness when faced with limited labeled data. Our findings highlight the limitations of GFM s, under different scenarios, showing that they do not consistently outperform supervised models. PANGAEA is designed to be highly extensible, allowing for the seamless inclusion of new datasets, models, and tasks in future research. By releasing the evaluation code and benchmark, we aim to enable other researchers to replicate our experiments and build upon our work, fostering a more principled evaluation protocol for large pre-trained geospatial models. The code is available at https://github.com/VMarsocci/pangaea-bench.

**Keywords** geospatial foundation models Earth observation evaluation benchmark self-supervised learning

## I. INTRODUCTION

Geospatial foundation models (GFM s) have emerged as a powerful tool for extracting representations from Earth observation (EO) data [1], [2]. These models are trained on large-scale EO datasets and aim to capture fundamental geospatial features that can be leveraged for downstream tasks [3], [4]. However, the rapid development of GFM s has outpaced

the establishment of a standardized and robust evaluation methodology.

Current evaluation practices often rely on a limited set of downstream tasks and datasets. For example, EuroSAT [5] is often used to evaluate downstream performance on land cover classification, despite being virtually solved (MTP [6] reports > 99% accuracy) and despite the community's shift toward segmentation-based tasks for land cover and land use prediction, given the real-world importance. Reported performance on such datasets is, therefore, not representative of the performance of GFM s across a broader range of applications. Additionally, current evaluation protocols often lack diversity, overlooking differences in image resolution, multiplicity of sensor types and modalities, geographical diversity and other factors crucial to model performance [7], [8]. For instance, the PhilEO Bench [9], though a valuable contribution, evaluates models only through the lens of Sentinel-2 data, limiting the breadth of the conclusions that can be drawn from this benchmark. In comparison, GEO-Bench [10] and SustainBench [11], include data from multiple sensors; however, they cover only land observations, ignore marine environments, and focus on mono-temporal applications. Finally, FoMo-Bench [12] and Crop Type Bench [13] restrict their scope respectively to forest and agriculture environments.

This lack of a standardized comprehensive evaluation protocol poses a significant challenge for the remote sensing community. It limits the ability to effectively compare models, assess their operational capabilities, and identify areas for improvement. Consequently, researchers and practitioners struggle to make informed decisions about which GFM s to utilize for their specific needs. To address this critical gap, we propose PANGAEA, a global and inclusive benchmark for GFM s. We aim to establish a robust, standardized and broadly applicable benchmark by curating a diverse and comprehensive collection of downstream datasets and tasks. PANGAEA covers a wide spectrum of tasks, ranging from marine debris segmentation to building damage mapping through change detection. It spans
<figure><img src="image_2.png" /><figcaption>Fig. 1: **Normalized performance comparison of different models across various datasets and training conditions.** The y-axis represents the normalized performance across the 11 PANGAEA's datasets, where the best-performing model for each dataset is assigned a value of 1 and the worst-performing model is assigned a value of 0. (a) **Full Labels:** Models trained on downstream tasks with access to the full labeled dataset. We observe that supervised baselines – especially UNet – outperform most of the models; (b) **Limited Labels (10%):** Models trained on downstream tasks with only 10% of labeled data. In this scenario, some GFM's – e.g. CROMA – excel and outperform both other GFM's and supervised baselines, although still competitive; (c) **Multi-Spectral Data:** Results for datasets made exclusively of multi-spectral images, i.e. HLS Burn Scars, MADOS, PASTIS-R, Sen1Floods11, Crop Type Mapping-SS, and BioMassters. Multi-spectral datasets tend to use low-resolution images, on which UNet excels. GFM's trained on high-resolution images, e.g. Scale-MAE, underperform in this setting that requires both spatial and spectral information; (d) **High-Resolution Data:** Results for high-resolution datasets, i.e. xView2, FiveBillionPixels, DynamicEarthNet, and SpaceNet 7. They are made of either RGB or RGB-NIR imagery. In this setting, most GFM's pre-trained on lower-resolution imagery fall behind, except CROMA and DOFA. On the other hand, GFM's pre-trained on high-resolution images and UNet perform well.</figcaption></figure>

multiple application domains, including urban, agricultural, marine, and forest environments. It also incorporates datasets with diverse temporal configurations (uni-temporal, bi-temporal, and multi-temporal, i.e. satellite image time series), extensive geographical coverage (all over the Earth), various spatial resolutions (from 1.5 m/pixel to 30 m/pixel resolution), and multiple sensor types, including natural colors, multi-spectral and synthetic aperture radar (SAR).

To gain a deeper understanding of the GFM landscape, we select a set of highly representative models and put them to the test on PANGAEA. Specifically, our selection criteria focus on approach, impact, and reproducibility. Regarding the different approaches, we include models employing diverse strategies, such as multi-modal contrastive learning (e.g. [14]), masked image modeling (MIM) (e.g. [15]), and supervised training (e.g. [16]). With impactful contributions, we consider models featured in top-tier computer vision conferences, specialized journals, or recognized within the research community. For reproducibility, the availability of open-source code and pre-trained models is a pivotal consideration. This comparative analysis allows us to assess the strengths and weaknesses of different models, providing insights and fostering advancements in the development of GFM's.

Through the development of a standardized evaluation protocol and comprehensive benchmarking effort, we examine a set of key research questions:

* **Generalization across domains:** How effectively do GFM's generalize to diverse downstream task domains? How do different model designs contribute to performance variations across these domains?

* **Comparison to supervised baselines:** Do GFM's consistently beat supervised task-specific baselines?

* **Performance with limited labels:** Do GFM's demonstrate tangible advantages in scenarios with limited labeled data?

Our experiments yield new insights into the applicability of GFM's. Pre-training dataset characteristics, with a specific focus on the sensor spectral and spatial resolutions, have a significant impact on the downstream performance of different GFM's. GFM's pre-trained on datasets with richer spectral information or higher spatial resolutions tend to perform better in tasks requiring such features Fig. 1. However, while GFM's generally perform well across a wide range of tasks, they do not consistently outperform supervised baselines. Task-specific models, i.e. those trained end-to-end on the task at hand, can leverage the spectral and spatial details of the dataset more effectively. This is especially true when there is a notable gap between the characteristics of the pre-training dataset and the downstream dataset (e.g. RGB vs multi-spectral). Similar behavior has been observed in recent works, spanning from time series [17], to crop monitoring [13] and canopy height prediction [18]. Similar findings are also in the data scarcity scenario, where some GFM's excel also w.r.t.
supervised baselines (Table VII), but the average performance
confirms what is observed when much more labels are available
(Fig. 1). Finally, we observe no strong downstream performance
advantage between a frozen encoder and end-to-end fine-tuning.
While fine-tuning does improve performance in some cases,
this is not systematic. The impact of fine-tuning depends on
the GFM's architecture and the type of downstream task. Many
other factors were tested in several ablations studies, ranging
from the impact of normalization to hyperparameter fine-tuning.

In addition to evaluating model performance, PANGAEA
emphasizes the importance of results reproducibility. By releas-
ing the full evaluation code alongside our proposed benchmark,
we ensure that other researchers can replicate our experiments
and compare models transparently thanks to an efficient and
robust evaluation protocol. This reproducibility fosters greater
transparency and collaboration within the research community,
promoting trust and consistency in GFM evaluations. Our
benchmark is designed to be extensible and modular, enabling
the easy inclusion of additional datasets, tasks, and models
in future research. This flexibility ensures that the benchmark
remains adaptable to the evolving needs of the remote sensing
community, accommodating new sensor types, more complex
tasks, and emerging model architectures.

In summary, our contributions are as follows:

* Open-source codebase for reproducible and transparent benchmarking. We provide an **easy-to-use and robust evaluation protocol**, to ensure comparability among different GFM s pre-trained under varying conditions.

* **Diverse benchmark** covering a wide range of datasets, tasks, and sensor types, fostering a more comprehensive evaluation of GFM s.

* **Highlighting the gaps in current GFM capabilities** and showcasing the potential for future advancements in model generalization, task specialization, and fine-tuning strategies.

By establishing this framework illustrated in Fig. 2, we aim
to facilitate a comprehensive and robust benchmark evaluation,
making it easier to understand and analyze the performance
and generalization capabilities of newly developed GFM s.

## II. RELATED WORK

### A. *Datasets for Earth Observation*

In recent years, the availability of geospatial and EO datasets
has expanded significantly, motivating systematic efforts to
catalog and organize them [19], [20]. These datasets can be
classified according to various criteria, including task, modality,
geographic region, domain, and temporality. From a task-based
perspective, research has branched into distinct areas, from
scene classification to pixel-level regression.

**Scene classification**, often focusing on land use or land
cover (LULC), is one of the most prevalent tasks due to its
alignment with computer vision methodologies. Early examples
include UC-Merced [21] and WHU-RS19 [22], [23], which
consist of high-resolution RGB imagery. MillionAID [24],
[25] is among the largest datasets in this category. Beyond
RGB images, some datasets like EuroSAT [5] rely on Sentinel-
2 images, while BigEarthNet [26], [27] incorporates both

Sentinel-1 and Sentinel-2 data while introducing a multi-
label classification task. Beyond LULC scene classification,
many specialized datasets address specific tasks with domain-
specific annotations. For example, METER-ML [28] is designed
for methane source mapping, CV4A Kenya [29] for crop
type identification, BrazilDAM [30] for dam detection, and
ForestNet [31] for deforestation driver classification. We do not
include classification tasks in PANGAEA, despite these tasks
being widespread in academic datasets, because patch-level
classification is often unsuitable for real-world geospatial tasks.
Indeed, patch-level predictions lose critical spatial detail and
fail to provide the dense information required for real-world
applications. This has been the main driver of the community
shift towards dense classification.

In addition to classification, **object detection** in EO imagery
has gained considerable attention. This task involves locating
and identifying specific objects, such as airplanes or ships,
with applications closely tied to the annotated object categories.
Notable datasets in this domain include Functional Map of
the World (fMoW) [32], a high-resolution global dataset with
62 object categories; DIOR [33], a large-scale 20-classes
multi-resolution object detection benchmark; and DOTA [34]
and iSAID [35], which are curated to include many small-
size objects and unbalanced classes. Furthermore, specialized
datasets cater to specific object categories, e.g. to detect ships
[36], [37], [38], oil storage tanks [39] and vehicles [40], [41].
To date, object detection in EO doesn't have a universally
established protocol for annotation and for robustly using
the GFM features; additionally, semantic segmentation often
achieves similar results by providing pixel-level information
that can be aggregated to detect and delineate objects more
flexibly. For this reason, we didn't include object detection
in PANGAEA, following the same approach as in other
benchmarks [10].

**Semantic segmentation** is another common task in EO
image analysis. Unlike classification task which focuses on
image-level labels, segmentation aims at assigning labels at
the pixel level. This dense labeling is essential to produce
higher-resolution semantic maps. Several notable semantic seg-
mentation datasets exist for LULC classes. ISPRS 2D Vaihingen
and Potsdam [42] provide an early example of multi-modal
data for semantic segmentation. However, the community has
moved towards datasets that capture multiple areas of interest,
such as LoveDA [43] and Five-Billion Pixels [44], which
consider aerial and Gaofen-2 satellite imagery respectively,
covering several Chinese cities. DeepGlobe [45] is notable as
one of the first global-scale semantic segmentation for LULC.
FLAIR-1 [46] integrates temporal and spectral information from
optical satellite time series to capture the diverse landscapes
of France. Beyond land cover mapping, semantic segmentation
datasets also tackle specialized applications, such as cloud cover
segmentation [47], [48], [49], building footprint segmentation
[50], [51], agricultural parcel mapping [52], [53], and beyond.

Another important EO analysis task is **change detection**,
which extends single image classification to multiple acquisi-
tions to identify and map alterations in the Earth's surface
over time. This technique is crucial in applications such
as monitoring floods [54], tracking urban development [55],
<figure><img src="image_3.png" /><figcaption>Fig. 2: PANGAEA aims for robust evaluation across diverse downstream datasets and applications.</figcaption></figure>

[56], and assessing the impacts of natural disasters [57], [58]. Change detection is most often addressed as a classification or segmentation problem, taking 2 or more images as inputs. Among change detection datasets, SZTAKI AirChange [59] is an early example focused on identifying significant changes in aerial image pairs taken years apart and across seasons. SECOND [60] consists of multi-temporal aerial images with pixel-level labels for land-cover variations and semantic change types. OSCD [61] includes multi-spectral Sentinel-2 images of various global cities for urban changes, while Levir-CD [56] is a very high-resolution (VHR) dataset designed for building change detection.

Beyond the aforementioned tasks, regression tasks are also becoming increasingly prominent in EO analysis, especially in estimating biophysical indicators from remote sensing imagery. Patch-level regression datasets like SatBird [62], for bird species distribution modeling, and ClimSim [63], for climate forecasting, showcase the diverse applications in this domain. Similarly, pixel-level regression tasks are explored by datasets such as BioMassters [64] for above-ground forest biomass estimation, and 3DCD [65], [66] for elevation change prediction.

Some useful references for a comprehensive list of currently available datasets include [19], [20], along with resources like the Satellite Image Deep Learning Datasets repository<fnref n="1" /> and the IEEE GRSS Earth Observation Dataset Search<fnref n="2" />.

## B. Geospatial Foundation Models

a) About the Naming: "Geospatial Foundation Models" refer to large-scale, self-supervised pre-trained deep learning models specifically designed to process and analyze geospatial

data [67]. Instead of targeting specific tasks and datasets, GFM should aim to support a wide array of downstream applications by training on large amounts of data and leveraging their learned representations to generalize across various domains.

These large pre-trained geospatial models deviate from the standard vision-language foundation models from computer vision and NLP, such as CLIP [68] and LLaVa [69], in several key aspects. First, vision-language models are multi-modal and integrate texts and images across diverse contexts. In comparison, large pre-trained geospatial models mainly rely on satellite imagery as their primary modality and more rarely incorporate other modalities, especially metadata or text. Self-supervised models for geospatial data are mostly limited to a single image modality, and scarcely consider vector data or temporal information. Second, vision-based modalities in geospatial are dominated by multi-spectral and SAR imagery. Broader integration of geospatial data remains underdeveloped, with little sensor diversity in the pre-training datasets. This is in part due to the unique challenges of geospatial data, especially size – ESA's Copernicus archives alone contain over 66 petabytes of satellite data as of 2024 – and non-stationarity, with significant variability across geographic regions, sensors, and time. This combination of incomplete multi-modal integration and non-stationary data distribution further complicates the development of robust, generalizable geospatial models. Third, most “geospatial foundation models” are yet to compete with their classical counterparts, both in parameter count and in pre-training data size. As shown in Tables III and IV, the largest currently available “geospatial foundation models” are CROMA [14] and Scale-MAE [15], with ≈ 350M parameters, which is smaller than the largest CLIP (ViT-L/14, ≈430M parameters). The commonly used LAION-5B dataset [70] is comprised of more than 5 billion images with an on-disk size greater than 200TB, compared to the 1TB of Prithvi [71] or

<footnote n="1">¹https://github.com/satellite-image-deep-learning/datasets</footnote>

<footnote n="2">²https://eod-grss-ieee.com/dataset-search</footnote>
30TB of Satlas [16].

These limitations are closely tied to issues with generalization. Many geospatial models are tested primarily on narrow tasks and datasets, leaving their ability to handle domain shifts—such as varying geographic regions, sensors, or temporal conditions—largely unexplored. The term “geospatial foundation model” is increasingly used in the remote sensing community, although still inconsistently. Some models still refer to themselves as large self-supervised models, although they are evaluated on downstream tasks for their representation abilities. In this work, we unify all these models under the same umbrella of “geospatial foundation models” for simplification. Nonetheless, it is important to recognize that significant gaps remain to achieve parity with vision-language foundation models, especially regarding multi-modal integration, generalization, and robustness. These challenges must be addressed for these models to fully realize the potential implied by the “foundation model” denomination.

In this section, we review these general-purpose models, including those not explicitly labeled as “foundation models” but that have still been developed to address various downstream applications.

b) Geospatial Foundation Models: The GFM framework has advanced significantly through both supervised and unsupervised approaches for visual tasks. A prominent example of a supervised model is SatlasNet [16], which employs a Swin Transformer backbone trained on a large annotated dataset with seven label types. However, full supervision requires extensive expert annotations, which are time-consuming and costly. Given the vast quantities of EO data and the expense of labels, many GFM s now adopt self-supervised learning (SSL) to learn representative features without labeled supervision. Studies have shown that SSL can enhance downstream performance on diverse tasks [72], [73]. Among the first attempts, SSL4EO-L [74] and SSL4EO-S12 [75] introduce globally distributed datasets based on Landsat-8 and Sentinel-1/2 imagery, respectively. These datasets have been used to train various off-the-shelf computer vision models, such as masked autoencoders (MAE) [76], data2vec [77], DINO [78], and MoCo [79].

Within the SSL framework, several works [80], [81], [82] focus on the techniques of masked image modeling (MIM), which first masks a proportion of the image and then learn to reconstruct masked parts. RingMO [83] falls into this category and was one of the first to mention "foundation models" in the title explicitly. RingMo-Sense [84] and RingMo-SAM [85] extend it, respectively making use of spatiotemporal evolution disentangling for spatiotemporal prediction tasks and segment anything model (SAM) [86] for image segmentation tasks. CtxMIM [87] combines a MIM approach with a Siamese network to give better context to the feature extractor. GFM [88]<sup>3</sup> proposes a continual pre-training strategy using MIM. msGFM [89] extends it, including multi-sensor images through a conditional computation approach (i.e. MoE). SatMAE [90] was one of the first frameworks based on masked autoencoders (MAE) [76] to tackle EO peculiarities, including multi-spectral

and spatiotemporal location embeddings. SatMAE++ [91] enhances it through multi-level feature extraction. Cross-scale MAE [92], also enforces cross-scale consistency through contrastive and generative losses. Scale-MAE [15] introduces resolution-aware positional encodings to learn features at different scales. USat [93] modifies the patch projection layers by embedding each spectral band separately before feeding them into MAE. RVSA [94] adapts the original full attention in MAE blocks with rotated varied-size window attention. FG-MAE [95] learns to reconstruct the image features instead of reconstructing the raw pixels. SatVision-TOA [96] is an MIM, trained on MODIS data.

Leveraging the MAE approach, some research has been oriented toward directly dealing with multi-modality characteristics of EO data [97], [98]. Prithvi [71] includes 3D positional embedding and 3D patch embedding modifications to learn from massive Harmonized Landsat-Sentinel (HLS) multi-temporal data. OmniSAT [99] proposes a multi-modality fusion technique by including spatially aligned VHR images and optical and SAR time series. SpectralGPT [100] extends MAEs to multi-spectral data. DOFA [101] leverages wavelength as a unifying parameter across various EO modalities to achieve a multi-modal representation. MMEarth [102] utilizes 12 modalities including pixel-level and image-level modalities as a pretext task for MAE reconstruction to learn representations for Sentinel-2 images. PRESTO [103] operates on pixel time series data and learns to reconstruct data when bands or timesteps are incomplete.

Another line of work investigates **contrastive learning** [104], which captures the relationships within the data by contrasting similar and dissimilar data points. GASSL [105] leverages spatially aligned images over time to construct temporal positive pairs, while SeCo [106] and CACo [107] model also the seasonality. MATTERS [108] learns invariance to illumination and viewing angle to achieve consistency in material and texture representation. SkySense [109], [110], [111] encodes multi-modal temporal remote sensing imagery and utilizes contrastive learning to learn features at various levels of granularities. CROMA [112] combines contrastive learning and MAE with aligned radar and optical images to learn both unimodal and multi-modal representations. GeRSP [113] combines contrastive learning and supervised learning to learn from unlabeled remote sensing imagery and labeled natural images jointly. Swim-diff [114] introduces a scene-wide matching approach that combines contrastive learning with diffusion models.

As part of the SSL learning paradigm, various studies have examined **self-distillation** based approaches, like in the DINO [78], where two different views are fed into two encoders, known as the student and the teacher, and the student learns to predict the teacher’s output. For example, DINO-MM [115] takes concatenated SAR-optical images as raw inputs and feeds the transformed augmented views to the student-teacher network. X-STARS [116] adds a cross-sensor alignment objective based on the DINO framework, for both pre-training from scratch and continual pre-training. There are also works [6], [117], [118], [119], [120], [121] contributing to the importance of different pre-training steps. For instance, MTP [6] conducts a

<footnote n="3">³Which we will refer to as GFM-Swin in this paper to disambiguate from the general term GFM.</footnote>
multi-task pre-training pipeline that supports both convolutional neural networks and vision transformer foundation models. DeCUR [122] decouples representations, distinguishing inter and intra-modal embeddings based on Barlow Twins [119], while RSMamba dense [120] and RSMamba [121] are two approaches based on Mamba [123], using selective state spaces. [124] surveys models and trends in **vision-language modeling**, where RemoteCLIP [125] was one of the first vision-language foundation models to learn aligned text embeddings and visual features for remote sensing. Other works explored the use of large language models (LLMs) for spatio-temporal modeling and prediction tasks [126], [127]. In addition, **generative** foundation models have also advanced within the EO field, examples include DiffusionSAT [128], CRS-Diff [129] and MetaEarth [130], which generate realistic EO images all over the Earth based on the diffusion models. Recently, it was shown that generative models can have great discriminative capabilities [131].

### c) Specialized Geospatial Foundation Models:
The unique challenges of **hyperspectral imagery** (HSI) have spurred the development of specialized foundation models, moving beyond architectures designed for multi-spectral data. One approach is the HSI-native, billion-parameter HyperSIGMA [132], trained from scratch on the massive HyperGlobal-450K dataset using a custom vision transformer architecture with sparse sampling attention to manage data redundancy. In contrast, the SpectralEarth [133] project pragmatically adapts classical vision backbones for HSI data by integrating a lightweight spectral adapter, pre-training on the large-scale, multi-temporal EnMAP satellite dataset. A different paradigm is introduced by HyperFree [134], a tuning-free model that leverages prompt engineering for various downstream tasks without requiring fine-tuning. Concerning **SAR** images, several models were considered precursors for strong GFM s, based on different approaches like MAEs [135], self-distillation [136], and DINO [137]. Research is also progressing on **parameter-efficient fine-tuning** (PEFT) methods [138], [139], [140], [141], [142], focusing on the efficiency to adapt existing Geospatial Foundation Models to spectral data with minimal trainable parameters.

Some models inherit characteristics of general-purpose GFM s while targeting domain-specific applications [12], [143], [144]. For example, FoMo-Net [12] is shaped to solve any forest monitoring task based on the MAE framework. Hydro-FM <fnref n="4" /> is a foundation model trained to solve water-related tasks. Other domain-oriented efforts include disaster management [143] and building segmentation [144]. Other works focus on more narrow aspects of GFM s, such as optimizing high-performance computing (HPC) performance [145], [146], [147], also preserving privacy [148]. Finally, various studies employ pre-trained GFM s for specialized applications, including, transfer learning [112], parameter forecasting [149], radar image target recognition [150], disaster management [143], image classification [151], domain generalization [152], [143], and sensor-agnostic image retrieval [153].

## C. Benchmarks in Geospatial Tasks

The emergence of GFM s has further driven the development of benchmarks designed to evaluate model performance across various tasks; while several valuable benchmarks have been proposed in recent years, they still exhibit certain limitations. A key issue is that many of these benchmarks focus on highly specific **domains**, making them valuable for targeting specific performance metrics but limiting their applicability from a more generalist perspective.

For example, PhilEO Bench [9] proposed a benchmark dataset comprising Sentinel-2 images, annotated for three downstream semantic segmentation tasks: building density estimation, road segmentation, and land cover classification. FoMo-Bench [154] targets forest monitoring with datasets covering classification, segmentation, and detection tasks, whereas [13] focuses on agriculture, selecting six crop classification datasets sourced from five continents.

Other benchmarks, however, overlook certain critical **tasks** (e.g. change detection) and fail to account for a pivotal characteristic of EO data: **multi-temporality**. For instance, EarthNets [19] provides a benchmark after analyzing published datasets, including fMoW [32] and BigEarthNet [27] for classification; DIOR [33] and fMoW [32] for object detection; and GeoNRW [155] and SEASONET [156] for semantic segmentation. While these datasets contribute valuable benchmarks for various tasks, they overlook the temporal aspect that is crucial for many geospatial models. Also, GEO-Bench [10] adapts and curates six classification and six segmentation datasets with diverse modalities and spatial resolutions, yet still does not fully incorporate the temporal dimension of EO data. Finally, SustainBench [11], though one of the most comprehensive benchmarks, focusing on 15 public datasets across 7 Sustainable Development Goals (SDGs) and addressing tasks such as economic development, agriculture, health, education, water and sanitation, climate action, and biodiversity, still lacks other important vision modalities, such as Synthetic Aperture Radar (SAR) in addition to optical imagery. These considerations are summarized in Table I, highlighting differences in application domains, tasks, modalities, and temporalities.

Several benchmarks also evaluate vision-language models (VLMs) for EO applications. For example, VRSBench [157] datasets set a benchmark for image captioning, visual grounding, and visual question answering using remote sensing images with paired captions, object references, and question-answers. UrBench [158] evaluates multi-modal capabilities in urban environments, with questions spanning geo-localization, scene understanding, and object understanding. VLEO-Bench [159] presents 11 real-world scenarios to test VLMs' abilities in scene understanding, localization and counting, and change detection. However, evaluating VLMs falls outside the scope of this work. Our benchmark focuses on dense-label tasks crucial for fine-grained spatial understanding, whereas vision-language tasks often center around image captioning, high-level scene interpretation, or object recognition, which differ from the dense prediction tasks we emphasize.

In the next section, we outline the parameters we have selected to evaluate the significance of a benchmark and

<footnote n="4">⁴https://github.com/isaaccorley/hydro-foundation-model</footnote>
<table><caption>TABLE I: Comparison of the PANGAEA Benchmark with existing geospatial vision benchmarks. We categorized each benchmark by domain (LULC stands for Land Use and Land Cover; SDG for Sustainable Development Goals; Agri for Agricultural), tasks (classification, semantic segmentation, object detection, and regression), modality and temporality.</caption><thead><tr><th>Benchmark</th><th>Domain</th><th>Tasks</th><th>Modality</th><th>Temporality</th></tr></thead><tbody><tr><td>EarthNets</td><td>LULC</td><td>Class, SemSeg, OD</td><td>MSI</td><td>Single</td></tr><tr><td>SustainBench</td><td>SDG</td><td>Class, SemSeg, Regr</td><td>MSI, StreetView</td><td>Multi</td></tr><tr><td>GEO-Bench</td><td>Urban, Agri, Forest</td><td>Class, SemSeg</td><td>MSI, SAR, DSM</td><td>Single</td></tr><tr><td>PhilEO Bench</td><td>Urban</td><td>SemSeg</td><td>MSI</td><td>Single</td></tr><tr><td>FoMO Bench</td><td>Forest</td><td>Class, SemSeg, OD</td><td>MSI</td><td>Multi</td></tr><tr><td><strong>PANGAEA</strong></td><td>Urban, Agri, Forest<br>Marine, Disaster</td><td>SemSeg, CD, Regr</td><td>MSI, SAR</td><td>Multi</td></tr></tbody></table>

describe the methodology we employed to identify the most effective one.

### III. THE PANGAEA BENCHMARK

#### A. Datasets Selection

To establish a new-generation benchmark aligned with real-world applications across diverse scenarios, we select datasets based on five key parameters: i) **application domain**, ii) **type of task**, iii) **multi-modality**, iv) **temporality**, and v) **geographical diversity**.

Evaluating the effectiveness of vision GFM requires assessing their performance across diverse **application domains** to understand their adaptability and generalization capabilities. Each domain presents unique challenges: models pre-trained on agricultural datasets may perform well in crop classification but struggle with marine tasks like pollutant detection, while those optimized for urban environments may underperform in rural or forested areas. To identify a model's generalization ability and versatility, it is essential to consider datasets tailored to various domains. For instance, datasets such as SpaceNet 7 [51] and DynamicEarthNet [55] are used for tasks like **urban** land cover or building footprint segmentation. Crop Type Mapping-South Sudan [52] (referred to as Crop Type Mapping-SS) and PASTIS-R [161], [53] are employed to identify and map crop types in **agricultural** landscapes. Datasets like MADOS [160] and BioMassters [64] [166] support the development of models for tasks in **marine** and **forest** environments. For natural **disaster** damage assessment, datasets such as xView2 [57] and Sen1Floods11 [167] are invaluable, addressing scenarios from earthquakes to floods. In addition, it is essential to assess the versatility of models not only across domains but also across **downstream tasks**. For example, within the urban domain, GFM optimized for semantic segmentation of land cover may struggle with change detection (e.g. building damage assessment) or regression tasks (e.g. building height estimation). To address this, we include a diverse set of downstream tasks, including semantic segmentation, change detection, and regression tasks, with a focus on dense, pixel-wise problems that hold significant real-world relevance in remote sensing. As previously stated, we exclude classification tasks, as they are less relevant for real-world application, as also shown by the community shift towards more dense tasks. We also excluded object detection, following other benchmarks [10], for two main reasons: there is no consensus on how to evaluate GFM for detection, making it out of scope for the moment; dense

tasks like semantic segmentation can achieve similar real-world results (i.e. segmenting one class).

Another key consideration in shaping the benchmark is incorporating **multi-modality** (i.e. data from different sensors or sources such as LiDAR, SAR, optical, or text). Different satellites are equipped with sensors of varying resolutions and spectral properties. Incorporating multi-modal datasets ensures the benchmark evaluates a GFM's ability to adapt to diverse sensing conditions, enhancing robustness for real-world applications. Currently, we focus on optical and SAR modalities due to the limited availability of datasets for other modalities, such as hyperspectral data.

EO data is inherently **multi-temporal** (i.e. data from the same location captured at different timeframes). However, not all models leverage this dimension. Temporal information is crucial for tasks such as biomass tracking, urban expansion monitoring, and seasonal crop assessment. Evaluating GFM on temporal data provides a more comprehensive assessment of their capabilities to determine whether these models can interpret dynamic processes.

Finally, concerning **geographic diversity**, [20] reports that, in their vast sampling of the available EO datasets, only 25% cover locations all over the world and almost 40% of the available datasets focus on "first-world countries" (Europe 21% and North America 18%). Asia is covered by 10% of the datasets, while Africa (5%), South America (4%), and Australia (1%) are significantly underrepresented. Many GFM perform poorly in underrepresented regions such as Africa or Southeast Asia, due to biases in the pre-training datasets. By building a geographically diverse dataset selection on a global scale, we aim to alleviate the biases in the evaluation datasets, thus enabling more robust assessments of models and their susceptibility to pre-training biases.

To evaluate GFM across all these dimensions, we curate a diverse and comprehensive set of datasets to compose the PANGAEA benchmark, spanning various domains, tasks, locations, temporalities, and satellite sources, detailed in Table II. We include PASTIS-R [161], [53] and AI4SmallFarms [165] ensuring that the benchmark evaluates models on the high-impact applications that are crop mapping and field boundary segmentation, across different regions, leveraging different modalities (i.e. Sentinel-1 and Sentinel-2) and multi-temporal data. The inclusion of MADOS [160] in the marine domain adds a global perspective on marine pollutant mapping, addressing worldwide environmental challenges. Disaster mapping datasets like xView2 [57], Sen1Floods11 [162], and HLS
<table><caption>TABLE II: Datasets included in the PANGAEA benchmark, highlighting their domains, tasks, sensor types, geographic coverage and temporality. This selection ensures diverse evaluation of GFM across a wide range of applications.</caption><thead><tr><th>Dataset</th><th>Domain</th><th>Task</th><th>Sensors</th><th>Location</th><th>Dates</th></tr></thead><tbody><tr><td>HLS Burn Scars [71]</td><td>Wildfire</td><td>Segmentation</td><td>HLS</td><td>USA</td><td>Mono</td></tr><tr><td>MADOS [160]</td><td>Marine</td><td>Segmentation</td><td>S2</td><td>Global</td><td>Mono</td></tr><tr><td>PASTIS-R [53], [161]</td><td>Agriculture</td><td>Segmentation</td><td>S1, S2, SPOT-6</td><td>France</td><td>Multi</td></tr><tr><td>Sen1Floods11 [162]</td><td>Flood</td><td>Segmentation</td><td>S1, S2</td><td>Global</td><td>Mono</td></tr><tr><td>xView2 [57], [58]</td><td>HADR</td><td>Change det.</td><td>Maxar</td><td>Global</td><td>Bi</td></tr><tr><td>Five Billion Pixels [163]</td><td>(Urban) Land Cover</td><td>Segmentation</td><td>Gaofen-2</td><td>China</td><td>Mono</td></tr><tr><td>DynamicEarthNet [55]</td><td>(Urban) Land Cover</td><td>Segmentation</td><td>PlanetFusion</td><td>Global</td><td>Mono</td></tr><tr><td>Crop Type Mapping-SS [11], [164]</td><td>Agriculture</td><td>Segmentation</td><td>S1, S2, Planet</td><td>South Sudan</td><td>Multi</td></tr><tr><td>SpaceNet 7 [51]</td><td>Urban</td><td>Change det.</td><td>Planet</td><td>Global</td><td>Bi</td></tr><tr><td>AI4SmallFarms [165]</td><td>Agriculture</td><td>Segmentation</td><td>S2</td><td>Southeast Asia</td><td>Mono</td></tr><tr><td>BioMassters [64]</td><td>Forest</td><td>Regression</td><td>S1, S2</td><td>Finland</td><td>Multi</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 3: Geographical distribution of PANGAEA benchmark dataset across different domains.</figcaption></figure>

Burn Scars [71] test GFM's abilities to detect and respond to sudden changes, such as natural disasters, across various global locations and temporalities. Datasets such as SpaceNet 7 [51] and DynamicEarthNet [55] challenge models with tasks like urban change detection and multi-temporal land cover analysis, ensuring robustness in dynamic urban and land cover scenarios. In the forest domain, BioMassters [64] provides a multi-modal, multi-temporal dataset for forest biomass estimation, a critical task for monitoring forest health and carbon sequestration. By incorporating BioMassters, we also add robust and long-overdue evaluation of GFM on regression tasks. Geographic diversity is further enhanced by datasets such as Five Billion Pixels [44] and AI4SmallFarms [165] (Asia), as well as Crop Type Mapping-SS [11], [164] (Africa), ensuring that the benchmark evaluates GFM in underrepresented regions with diverse satellite data, including Gaofen and Sentinel-1/2. The spatial distribution of the PANGAEA benchmark datasets across domains is shown in Fig. 3.

This extensive coverage of PANGAEA across domains, tasks, modalities, temporalities, and locations ensures that the benchmark is robust, enabling a thorough evaluation of model generalizability, adaptability, and performance in real-world remote sensing applications.

## B. Models Selection

To efficiently benchmark the performance of different GFM, PANGAEA includes nine models selected based on three criteria: **reproducibility**, **approach** and **impact**. The models are listed alphabetically in Table III, along with their corresponding image sources and the data volume used for pre-training.

To ensure reproducibility, we require each model to have open-source code and publicly available weights. With this condition met, we select a diverse set of models, each representing a unique approach to highlighting their strengths in remote sensing and geospatial analysis.

SSL4EO-S12 [74] is selected as a GFM baseline. This includes four vision-transformer (ViT)-based models, each pre-trained with different self-supervised learning approaches, such as masked image modeling (S12-MAE), contrastive learning (S12-MoCo) and knowledge distillation (S12-DINO, S12-Data2Vec). Since these models employ original computer vision (CV) algorithms without EO-specific modifications but are pre-trained on large-scale remote sensing imagery, they serve as a baseline to provide insights into the performance of CV models without adjustments for the peculiarities of EO data.

Several models are selected based on the widespread
<table><caption>TABLE III: Overview of the pretraining datasets and the number of patches used by the selected GFM s. For Prithvi, the data volume is reported.</caption><thead><tr><th>Model</th><th>Pretraining Images</th><th>Patches/Volume</th></tr></thead><tbody><tr><td>CROMA</td><td>Sentinel-1, Sentinel-2</td><td>3M</td></tr><tr><td>DOFA</td><td>Sentinel-1, Sentinel2, Gaofen-2, NAIP, EnMAP</td><td>8.08M</td></tr><tr><td>GFM-Swin</td><td>NAIP, RSD46-WHU, MLRSNet, RESISC45, PatternNet</td><td>600K</td></tr><tr><td>Prithvi</td><td>Harmonized Landsat Sentinel-2 (HLS)</td><td>1TB</td></tr><tr><td>RemoteCLIP</td><td>SEG-4, DET-10, RET-3</td><td>165K</td></tr><tr><td>SatlasNet</td><td>Sentinel-2, NAIP</td><td>856K</td></tr><tr><td>Scale-MAE</td><td>FMoW-RGB</td><td>363.6K</td></tr><tr><td>SpectralGPT</td><td>fMoW-S2, BigEarthNet</td><td>1.47M</td></tr><tr><td>SSL4EO-S12</td><td>Sentinel-1, Sentinel-2</td><td>3M</td></tr></tbody></table>

<table><caption>TABLE IV: Complexity of the selected GFM s. Please consider that all models include a UPerNet [168] decoder, except UNet [169] which uses its original decoder.</caption><thead><tr><th></th><th>Model</th><th>Complexity (GMacs)</th><th>Trainable Params (M)</th><th>Total Params (M)</th></tr></thead><tbody><tr><td rowspan="10">Foundation models</td><td>CROMA</td><td>139.23</td><td>46.95</td><td>350.0</td></tr><tr><td>DOFA</td><td>65.02</td><td>39.35</td><td>150.9</td></tr><tr><td>GFM-Swin</td><td>19.05</td><td>33.71</td><td>120.4</td></tr><tr><td>Prithvi</td><td>64.85</td><td>39.34</td><td>125.7</td></tr><tr><td>RemoteCLIP</td><td>17.57</td><td>39.35</td><td>16.8</td></tr><tr><td>SatlasNet</td><td>16.70</td><td>33.25</td><td>121.2</td></tr><tr><td>Scale-MAE</td><td>103.77</td><td>46.95</td><td>350.1</td></tr><tr><td>SpectralGPT</td><td>380.69</td><td>164.40</td><td>249.8</td></tr><tr><td>S12-MoCo</td><td>46.90</td><td>30.89</td><td>53.5</td></tr><tr><td>S12-DINO</td><td>47.93</td><td>30.89</td><td>53.5</td></tr><tr><td>S12-MAE</td><td>46.90</td><td>30.89</td><td>53.5</td></tr><tr><td>S12-Data2Vec</td><td>46.89</td><td>30.89</td><td>53.5</td></tr><tr><td rowspan="2">Backlines</td><td>UNet</td><td>27.38</td><td>14.79</td><td>14.79</td></tr><tr><td>ViT-B/16</td><td>64.74</td><td>125.1</td><td>125.1</td></tr></tbody></table>

masked image modeling framework, each with a different focus. **Scale-MAE** [15] is included for its ability to handle input images under varying resolutions, demonstrating strong performance and widespread applicability. **SpectralGPT** [100] is selected due to its explicit design for spectral satellite images, addressing the unique characteristics of spectral data. Similarly, to effectively deal with spectral bands, **DOFA** [101] employs a dynamic network weights generator based on the central wavelengths of each spectral band, allowing generalization to different modalities **Prithvi** [71] has made a significant impact by training an MAE on NASA's massive Harmonized Landsat Sentinel-2 (HLS) data. It is also one of the few models to incorporate the temporal dimension in its pre-training strategy, expanding the 2D MAE to a 3D version to utilize 3D spatiotemporal data. Among other MIM models, we select **GFM-Swin** [170], which leverages multi-modality data for its pre-training to also learn the cross-sensor reconstruction.

In addition to MIM-based approaches, we also include models that employ contrastive learning strategies. While the former methods are faster to train, several studies [116], [170] demonstrate the advantages of contrastive learning which relies less on large quantities of high-resolution images and more on auxiliary data, such as text or additional modalities. In this context, **RemoteCLIP** [171] and **CROMA** [14] are included. **RemoteCLIP** is trained to align vision-language representations and learns robust visual features with rich semantics of satellite imagery visual concepts. **CROMA**, on the other hand, aligns geographically and temporally matched radar and optical image pairs while also combining unimodality

MIM training objectives, making it a valuable inclusion.

Furthermore, **SatlasNet** [16], one of the few GFM s based on fully supervised learning, is included to demonstrate the potential and limitations of this approach in geospatial tasks.

By including these models in PANGAEA, we aim to investigate how different model designs contribute to variations in performance across various tasks and conditions and determine which strategies are best suited for advancing the field of geospatial analysis. The pre-training data sources leveraged by GFM s play a key role in achieving robust representations for geospatial applications. This factor is considered in Table III, which showcases the data sources and scales utilized by the selected models in PANGAEA, serving as a critical aspect in evaluating the capabilities of these GFM s.

## IV. EXPERIMENTAL PROTOCOL

PANGAEA aims to extensively evaluate GFM s more robustly by introducing a standardized evaluation protocol on the 11 benchmark datasets. To do so, we integrate all selected models and datasets, alongside task-specific fine-tuning and evaluation code into a unified framework. The protocol incorporates standardized approaches to ensure compatibility between downstream tasks and models, such as consistent band adaptation alongside multi-modal and multi-temporal evaluation strategies.

a) GFM Evaluation on Downstream Task: As commonly done to evaluate self-supervised models [15], [75], we evaluate the representation abilities of the models using frozen encoders. We treat the encoders of various foundation models as feature extractors by freezing their pre-trained weights. Because
PANGAEA relies on dense pixel-level tasks and not all GFM encoders produce dense features, we then plug a trainable decoder to upsample the features to match the input image resolution. This decoder performs either regression or pixel-level classification depending on the task. Following existing works that evaluate GFM on dense tasks [15], [100], we implement a UPerNet [168] decoder, that takes as input four intermediate feature levels from the foundation model encoders, as shown in Fig. 4a. We select features from evenly spaced layers depending on the specific GFM architecture. The decoder is then trained on the datasets under various scenarios, including fine-tuning with limited labeled data (10%, 50%, and 100% of each dataset). We compare these fine-tuned GFMs against two common fully supervised baselines: a UNet [169], representing a convolutional neural network backbone, and a Vision Transformer [172], representing a transformer backbone (ViT-B/16). Both these baselines are trained from scratch using all available bands from the downstream dataset.

Based on this evaluation protocol, we compute the complexity and parameter count for each model, as shown in Table IV. These metrics provide insights into model size and computational efficiency, helping to assess the trade-offs between performance and resource requirements.

b) *Data Preprocessing:* All models are trained using the same preprocessed dataset. Because different foundation models expect different image sizes, we either crop (for larger images) or resize (for smaller images) the input images as needed. We normalize the images by removing the mean and dividing by the standard deviation, computed for each band across every dataset. This ensures consistent scaling across all images from every dataset. In data-scarce scenarios, we implement stratified sampling to preserve the overall distribution of the dataset. For segmentation tasks, we first calculate the category distributions for the dataset, organizing samples into bins that represent unique combinations of category proportions. By grouping samples into these bins, we can then draw a proportionate number (e.g. 10%, 50%) of samples from each bin, ensuring that the original category distribution is maintained. For regression, the mean value of each image is computed and binned into intervals, enabling proportional sampling from each bin.

c) *Bands Matching and Adaptation:* Foundation models are pre-trained on diverse sensors and spectral bands, which often do not align precisely with those available in downstream datasets (e.g. a GFM trained on RGB images cannot directly process multi-spectral Sentinel-2 imagery). To address this, we adopt a common band adaptation strategy for all models. If the bands used to pre-train the GFM are included in bands from the downstream dataset, then we match those. For example, GFMs trained on RGB imagery will only use the red, green and blue bands from datasets like PASTIS-R, which is comprised of Sentinel-2 images. If certain bands are unavailable in the dataset, we apply zero-padding for those missing bands, as proposed by [173]. For example, GFMs trained on Sentinel-2 data are still evaluated on RGB-only datasets, such as xView2 (which uses color imagery from Maxar), however the missing bands are filled with zeros.

This strategy also enables the models to address multi-modal

data by ensuring band matching across all sensor modalities. Among the selected models, only CROMA and DOFA can jointly handle optical and SAR data. We experiment with these two models using either unimodal data or both modalities for datasets containing both optical and SAR images.

d) *Multi-temporal Strategy:* Although we have selected multi-temporal datasets as downstream tasks, not all considered GFMs have the capability to handle image time series. In our selection, only SatlasNet and Prithvi natively handle temporal information. SatlasNet employs a dedicated aggregation network that processes individual images through a shared encoder and aggregates features via a pooling strategy. Prithvi incorporates the temporal dimension directly within its patch embedding process. For these models, we directly feed the decoder with the multi-temporal representations extracted from the encoder. As illustrated in Fig. 4b, for bi-temporal datasets such as change detection, we pass both images to the same GFM encoder and then concatenate the features before passing them to the UPerNet decoder. To enable models that only handle single images to be trained on multi-temporal datasets, we introduce and compare two feature aggregation strategies: a naive linear mapping and a lightweight spatial-temporal encoder (L-TAE) [174]. For models supporting only single-timeframe input, each image in the time series from the multi-temporal dataset is passed independently through the encoder. The linear mapping then applies a layer to compress features across the temporal dimension, while L-TAE uses a temporal attention module to capture relationships between frames over time. The temporally aggregated features are then fed through the UPerNet decoder as before, as shown in Fig. 4c.

e) *Hyperparameters:* For a fair comparison across models, the decoder is fine-tuned with an identical set of hyperparameters. For every model and dataset combination, the decoder is fine-tuned for 80 epochs using the Adam optimizer with an initial learning rate of 1e-4, a batch size of 8, and a weight decay of 0.05. The learning is decreased by a factor of 0.1 using a multi-step schedule at 60% and 90% of the training. Some specific edge cases are detailed in Appendix A for runs that necessitate adjustments due to memory constraints. The best-performing checkpoint on the validation set is kept and used to evaluate the model on the test set. For evaluation, to process larger images, we employ a sliding window inference strategy, dividing the input image into evenly distributed smaller crops. A stride is used to ensure the crops collectively cover the entire image with the fewest patches, and the overlapping areas are averaged.

## V. EXPERIMENTAL RESULTS

The code is available at https://github.com/VMarsocci/pangaea-bench, where an extensive documentation is provided to reproduce results, to customize the experimental setup for specific applications, and to add new datasets and models.

### A. Main Comparison

Table V presents the experimental results of GFM evaluation using the full dataset across all 11 benchmark datasets, listed alphabetically. For multi-temporal datasets, the reported results
<figure><img src="image_5.png" /><figcaption>Fig. 4: Fine-tuning strategies of the foundation models depending on the input temporality.</figcaption></figure>

reflect the use of the L-TAE feature aggregation, except for a few models that inherently support multi-temporal data. We include two fully supervised baselines for comparison, UNet and ViT, trained from scratch in an end-to-end manner without any pre-trained weights. These results provide valuable insights from multiple perspectives. Extended results are presented in Appendix A.

**Impact of Resolution on Model Performance.** GFM pre-trained on lower-resolution data tend to perform worse on high-resolution datasets with limited spectral information, such as Five Billion Pixels (FBP) and SpaceNet 7 (see Fig. 1). GFM like Scale-MAE are often among the best-performing models, reaching e.g. 67.19% mIoU on FBP and 62.96% mIoU on SpaceNet 7. In contrast, models pre-trained on lower resolutions, such as S12-DINO (51.15% on FBP) and S12-Data2Vec (48.82% on FBP), show lower performance. This trend is especially evident in tasks without extensive spectral information, where spatial detail becomes essential. Observe, for example, MADOS, where GFM-Swin achieves a high 64.71% mIoU, but models like SSL4EO12-MAE (49.90% mIoU) and SSL4EO12-DINO (49.37% mIoU) lag behind, showing the advantage of pre-training on high-resolution data. On the other hand, in some cases, when there is a match in the pre-training data and the downstream task data we can observe an average increase in the performance (e.g. Prithvi pre-trained on Harmonized Landsat Sentinel-2 data performs strongly on HLS Burn Scars). When compared to supervised baselines, we can observe that GFM struggle especially on low-resolution tasks, probably because features useful for these downstream tasks are easier to learn from not specialized architecture (i.e. UNet), due to their reduced available dense information. On the other hand, on high-resolution tasks, GFM, pre-trained on similar resolution data, still offer strong performance (Fig. 1).

**Role of Spectral Information in Task-Specific Performance.** Spectral richness significantly impacts tasks that require fine spectral detail (e.g. marine, agricultural, and forest domains). In datasets such as MADOS, Crop Type Mapping-SS, and BioMassters, GFM trained with a wider spectral range perform best. CROMA reaches 67.55% mIoU on MADOS and 49.38% mIoU on Crop Type Mapping-SS. Similarly, SpectralGPT along with CROMA are among the top-performing

models with low RMSE on BioMassters, suggesting that models that use diverse spectral data capture essential distinctions, whereas models without this range (e.g. Scale-MAE reaches 47.15 RMSE on BioMassters) underperform. The trend on HLS Burn Scars illustrates this: trained on a wider set of bands reach better performance on average. On simpler tasks where spectral information is less crucial, models with fewer spectral inputs are competitive. For example, GFM-Swin excels in FBP and SpaceNet 7, but struggles on BioMassters, indicating that spatial features can compensate for lower spectral diversity for urban land cover tasks, especially as the downstream datasets use RGB imagery only. Similar observations hold for the supervised baselines that can leverage all of the spectral information and deliver strong performances on low-resolution tasks (e.g. UNet reaches 84.51% and ViT 81.58% on HLS Burn Scars).

**Effectiveness of Text-Based Models for Multi-Class Semantic Segmentation Datasets.** The text-augmented RemoteCLIP model shows advantages in complex, multi-class tasks where class boundaries are subtle, such as MADOS (where it reaches third place with 60.00% mIoU) and FBP (first place with 69.19% mIoU). This success likely comes from RemoteCLIP's ability to leverage text-based information during training to learn features that can differentiate visually similar classes. However, RemoteCLIP underperforms on simpler tasks, e.g. scores only 74.26% mIoU on Sen1Floods11, compared to S12-MoCo (89.26% mIoU) and CROMA (90.89% mIoU). This limitation may be due to its RGB-only training, which restricts its potential in tasks that benefit from a broader spectral range.

**Strong Convolutional Baseline.** For simpler datasets with fewer classes, the convolutional UNet baseline excels. UNet achieves the best results on HLS Burn Scars (84.51 % mIoU), Sen1Floods11 (91.42 % mIoU), and AI4SmallFarms (46.34 % mIoU), showing that such a simple convolutional model is well-suited for simpler datasets. Overall, our results indicate that learning the features through convolution is the best approach for one-class datasets. On the other hand, when there is a domain gap between pre-training and downstream task (i.e. regression), performance is so poor that even supervised baselines do better than GFM (35.67 RMSE w.r.t. second best 36.11 RMSE with SpectralGPT). Thus, UNet is the model that reaches top-2 performance (across all models) the biggest
<table><caption>TABLE V: Performance evaluation of Geospatial Foundation Models across 11 benchmark datasets using 100% of the data. For semantic segmentation and change detection tasks, the mIoU ↑ is reported. For regression task, RMSE ↓ is reported. #Top2 indicates the number of datasets where the models achieve top-2 performance across all evaluated datasets.</caption><thead><tr><th>Model</th><th>HLS</th><th>Burns</th><th>MADOS</th><th>PASTIS</th><th>Sen1Floods11</th><th>xView2</th><th>FBP</th><th>DynEarthNet</th><th>CropMap</th><th>SN7</th><th>AI4Farms</th><th>BioMassters</th><th>#Top2</th></tr></thead><tbody><tr><td>CROMA</td><td>82.42</td><td><b>67.55</b></td><td>32.32</td><td><u>90.89</u></td><td>53.27</td><td>51.83</td><td>38.29</td><td>49.38</td><td>59.28</td><td>25.65</td><td>36.81</td><td></td><td>2</td></tr><tr><td>DOFA</td><td>80.63</td><td>59.58</td><td>30.02</td><td>89.37</td><td><u>59.64</u></td><td>43.18</td><td><u>39.29</u></td><td>51.33</td><td>61.84</td><td>27.07</td><td>42.81</td><td></td><td>2</td></tr><tr><td>GFM-Swin</td><td>76.90</td><td><u>64.71</u></td><td>21.24</td><td>72.60</td><td>59.15</td><td>67.18</td><td>34.09</td><td>46.98</td><td>60.89</td><td>27.19</td><td>46.83</td><td></td><td>1</td></tr><tr><td>Prithvi</td><td><u>83.62</u></td><td>49.98</td><td>33.93</td><td>90.37</td><td>49.35</td><td>46.81</td><td>27.86</td><td>43.07</td><td>56.54</td><td>26.86</td><td>39.99</td><td></td><td>1</td></tr><tr><td>RemoteCLIP</td><td>76.59</td><td>60.00</td><td>18.23</td><td>74.26</td><td><u>57.41</u></td><td><b>69.19</b></td><td>31.78</td><td><u>52.05</u></td><td>57.76</td><td>25.12</td><td>49.79</td><td></td><td>2</td></tr><tr><td>SatlasNet</td><td>79.96</td><td>55.86</td><td>17.51</td><td>90.30</td><td>52.23</td><td>50.97</td><td>36.31</td><td>46.97</td><td>61.88</td><td>25.13</td><td>41.67</td><td></td><td>0</td></tr><tr><td>Scale-MAE</td><td>76.68</td><td>57.32</td><td>24.55</td><td>74.13</td><td><b>60.72</b></td><td><u>67.19</u></td><td>35.11</td><td>25.42</td><td><b>62.96</b></td><td>21.47</td><td>47.15</td><td></td><td>3</td></tr><tr><td>SpectralGPT</td><td>80.47</td><td>57.99</td><td>35.44</td><td>89.07</td><td>48.40</td><td>33.42</td><td>37.85</td><td>46.95</td><td>58.86</td><td>26.75</td><td><u>36.11</u></td><td></td><td>1</td></tr><tr><td>S12-MoCo</td><td>81.58</td><td>51.76</td><td>34.49</td><td>89.26</td><td>51.59</td><td>53.02</td><td>35.44</td><td>48.58</td><td>57.64</td><td>25.38</td><td>40.21</td><td></td><td>0</td></tr><tr><td>S12-DINO</td><td>81.72</td><td>49.37</td><td><u>36.18</u></td><td>88.61</td><td>50.56</td><td>51.15</td><td>34.81</td><td>48.66</td><td>56.47</td><td>25.62</td><td>41.23</td><td></td><td>1</td></tr><tr><td>S12-MAE</td><td>81.91</td><td>49.90</td><td>32.03</td><td>87.79</td><td>50.44</td><td>51.92</td><td>34.08</td><td>45.8</td><td>57.13</td><td>24.69</td><td>41.07</td><td></td><td>0</td></tr><tr><td>S12-Data2Vec</td><td>81.91</td><td>44.36</td><td>34.32</td><td>88.15</td><td>51.36</td><td>48.82</td><td>35.90</td><td><b>54.03</b></td><td>58.23</td><td>24.23</td><td>41.91</td><td></td><td>1</td></tr><tr><td>UNet Baseline</td><td><b>84.51</b></td><td>54.79</td><td>31.60</td><td><b>91.42</b></td><td>58.68</td><td>60.47</td><td><b>39.46</b></td><td>47.57</td><td><u>62.09</u></td><td><b>46.34</b></td><td><b>35.67</b></td><td></td><td>6</td></tr><tr><td>ViT Baseline</td><td>81.58</td><td>48.19</td><td><b>38.53</b></td><td>87.66</td><td>57.43</td><td>59.32</td><td>36.83</td><td>44.08</td><td>52.57</td><td><u>38.37</u></td><td>38.55</td><td></td><td>2</td></tr></tbody></table>

number of times (5 times w.r.t to CROMA's 3 and Scale-MAE 3 times).

**Challenges with Multi-Temporal Tasks.** Most GFM s do not consider satellite image time series in their design, and the few GFM s that are designed to leverage the temporal dimension of EO data (i.e. Prithvi and SatlasNet) performed comparatively low on multi-temporal tasks. Prithvi scored the lowest mIoU on DynamicEarthNet (27.86%) and SatlasNet scored the lowest mIoU on PASTIS-R (17.51%). Both Prithvi and SatlasNet were outperformed by several uni-temporal GFM s when combined with a temporal module (i.e. L-TAE). This is stressed by the results on all multi-temporal datasets we evaluated; PASTIS-R, DynamicEarthNet, Crop Type Mapping-SS, and BioMassters. Most notably, the uni-temporal models such as CROMA achieved better scores (32.32% mIoU on PASTIS-R, 42.12% mIoU on DynamicEarthNet, and 36.81 RMSE on BioMassters) than both Prithvi and SatlasNet. Therefore, our results highlight the need for dedicated multi-temporal foundation models capable of effectively capturing temporal patterns in satellite image time series.

**Challenges with Multi-Modal Datasets.** While some models like S12-MAE, CROMA, and DOFA incorporate SAR data through specialized encoders or dynamic wavelength adjustments, their performance on multi-modal datasets often falls short of expectations. As demonstrated in Tables A3, A12 and A18, utilizing optical data alone generally yields superior results compared to incorporating SAR information, whether solely or in combination with optical data. This performance gap indicates that effectively leveraging multi-modal data, particularly the fusion of optical and SAR imagery, remains a significant challenge for GFM s.

## B. Data Scarcity

We further evaluate the performance of the selected GFM s by reducing the percentage of labeled data used for training to

50% and 10%, while preserving the original data distribution. The results are summarized in Tables VI and VII, respectively, providing insights into the models' performance under limited supervision, simulating real-world scenarios where labeled data is often scarce.

A general trend of performance degradation is observed across various EO tasks. This observation aligns with the well-established principle that models typically benefit from larger training sets, which provide a more comprehensive representation of the data distribution and thereby enable better generalization.

Nevertheless, with limited training samples, a compelling trend emerges where several GFM s demonstrate notable strengths. We report the number of times each model achieves a top-2 performance across all datasets. While the fully supervised UNet baseline performs best in this score, this metric shifts in favor of GFM s as the amount of labeled data decreases. For instance, UNet achieves a score of 6 for this metric with fully labeled datasets, but its score decreases to 4 and 2 for the 50% and 10% labeled data scenarios, respectively. In contrast, some GFM s show improved performance, with CROMA achieving a score of 6 in the 10% labeled data scenario. This trend is also evident in datasets such as HLS Burn Scars and SpaceNet 7.

This phenomenon can be attributed to the better generalization capabilities of foundation models compared to fully supervised baselines, which stem from their exposure to a vast and diverse range of data during pre-training. In situations where labeled data is scarce, foundation models can effectively leverage the knowledge acquired during pre-training to compensate for the lack of task-specific labeled data, highlighting the potential of foundation models to address challenges in EO tasks where acquiring labeled data is often costly and time-consuming.
<table><caption>TABLE VI: Performance evaluation of Geospatial Foundation Models across 11 benchmark datasets using 50% of the data. For semantic segmentation and change detection tasks, the mIoU ↑ is reported. For regression task, RMSE ↓ is reported. #Top2 indicates the number of datasets where the models achieve top-2 performance across all evaluated datasets.</caption><thead><tr><th>Model</th><th>HLS Burns</th><th>MADOS</th><th>PASTIS</th><th>Sen1Floods11</th><th>xView2</th><th>FBP</th><th>DynEarthNet</th><th>CropMap</th><th>SN7</th><th>AI4Farms</th><th>BioMassters</th><th># Top-2</th></tr></thead><tbody><tr><td>CROMA</td><td><u>81.52</u></td><td>57.68</td><td>32.33</td><td><u>90.57</u></td><td>51.44</td><td>48.01</td><td><u>38.30</u></td><td><u>42.20</u></td><td>59.31</td><td>28.19</td><td>38.50</td><td>4</td></tr><tr><td>DOFA</td><td>78.02</td><td>55.21</td><td>28.60</td><td>88.39</td><td><u>58.91</u></td><td>36.90</td><td><u>39.20</u></td><td>30.93</td><td>47.06</td><td>26.69</td><td>42.81</td><td>2</td></tr><tr><td>GFM-Swin</td><td>74.36</td><td><u>63.37</u></td><td>20.41</td><td>71.61</td><td>57.81</td><td><u>63.14</u></td><td>31.25</td><td>31.42</td><td>59.83</td><td>28.43</td><td>48.19</td><td>2</td></tr><tr><td>Prithvi</td><td>80.89</td><td>40.79</td><td>33.13</td><td>89.69</td><td>45.79</td><td>40.27</td><td>33.43</td><td><u>42.51</u></td><td>49.45</td><td>29.27</td><td>41.03</td><td>1</td></tr><tr><td>RemoteCLIP</td><td>74.28</td><td>53.26</td><td>17.46</td><td>71.67</td><td>57.43</td><td><u>65.92</u></td><td>30.91</td><td>36.3</td><td>50.83</td><td>25.11</td><td>50.09</td><td>1</td></tr><tr><td>SatlasNet</td><td>75.97</td><td>52.24</td><td>16.78</td><td>89.45</td><td>50.74</td><td>46.04</td><td>36.34</td><td>35.29</td><td><u>60.74</u></td><td>27.08</td><td>42.23</td><td>1</td></tr><tr><td>Scale-MAE</td><td>75.47</td><td>46.87</td><td>23.26</td><td>72.54</td><td><u>59.45</u></td><td>62.11</td><td>32.60</td><td>20.32</td><td><u>61.24</u></td><td>26.40</td><td>46.74</td><td>2</td></tr><tr><td>SpectralGPT</td><td>76.40</td><td><u>58.00</u></td><td>34.61</td><td>87.52</td><td>45.94</td><td>21.71</td><td>36.52</td><td>32.09</td><td>56.28</td><td>27.46</td><td><u>37.34</u></td><td>2</td></tr><tr><td>S12-MoCo</td><td>79.79</td><td>42.90</td><td>32.59</td><td>89.22</td><td>49.66</td><td>46.92</td><td>34.45</td><td>41.32</td><td>56.21</td><td>28.38</td><td>41.08</td><td>0</td></tr><tr><td>S12-DINO</td><td>80.12</td><td>40.42</td><td><u>35.71</u></td><td>88.93</td><td>48.46</td><td>44.85</td><td>32.76</td><td>31.13</td><td>55.14</td><td>25.68</td><td>41.47</td><td>1</td></tr><tr><td>S12-MAE</td><td>80.13</td><td>44.29</td><td>31.15</td><td>88.43</td><td>47.09</td><td>45.63</td><td>33.29</td><td>28.07</td><td>55.55</td><td>27.50</td><td>41.66</td><td>0</td></tr><tr><td>S12-Data2Vec</td><td>79.82</td><td>41.22</td><td>33.42</td><td>86.58</td><td>48.84</td><td>46.73</td><td>32.61</td><td>28.53</td><td>56.94</td><td>25.84</td><td>42.82</td><td>0</td></tr><tr><td>UNet Baseline</td><td><u>82.39</u></td><td>43.87</td><td>30.25</td><td><u>90.91</u></td><td>56.58</td><td>55.42</td><td>35.14</td><td>36.30</td><td>46.82</td><td><u>45.02</u></td><td><u>36.72</u></td><td>4</td></tr><tr><td>ViT Baseline</td><td>78.17</td><td>28.77</td><td><u>38.71</u></td><td>86.08</td><td>54.82</td><td>57.32</td><td>37.33</td><td>39.53</td><td>49.21</td><td><u>38.37</u></td><td>39.56</td><td>2</td></tr></tbody></table>

<table><caption>TABLE VII: Performance evaluation of Geospatial Foundation Models across 11 benchmark datasets using 10% of the data. For semantic segmentation and change detection tasks, the mIoU ↑ is reported. For regression task, RMSE ↓ is reported. #Top2 indicates the number of datasets where the models achieve top-2 performance across all evaluated datasets.</caption><thead><tr><th>Model</th><th>HLS Burns</th><th>MADOS</th><th>PASTIS</th><th>Sen1Floods11</th><th>xView2</th><th>FBP</th><th>DynEarthNet</th><th>CropMap</th><th>SN7</th><th>AI4Farms</th><th>BioMassters</th><th>#Top2</th></tr></thead><tbody><tr><td>CROMA</td><td>76.44</td><td><strong>32.44</strong></td><td>32.80</td><td><strong>87.22</strong></td><td>46.54</td><td>37.39</td><td><strong>36.08</strong></td><td><strong>36.77</strong></td><td>42.15</td><td><strong>38.48</strong></td><td><strong>40.25</strong></td><td>6</td></tr><tr><td>DOFA</td><td>71.98</td><td>23.77</td><td>27.68</td><td>82.84</td><td><u>55.60</u></td><td>27.82</td><td><strong>39.15</strong></td><td>29.91</td><td>46.10</td><td>27.74</td><td>46.03</td><td>2</td></tr><tr><td>GFM-Swin</td><td>67.23</td><td>28.19</td><td>21.47</td><td>62.57</td><td>53.45</td><td>55.58</td><td>28.16</td><td>27.21</td><td>39.48</td><td>32.88</td><td>49.30</td><td>0</td></tr><tr><td>Prithvi</td><td>77.73</td><td>21.24</td><td>33.56</td><td>86.28</td><td>35.08</td><td>29.98</td><td>32.28</td><td>27.71</td><td>36.78</td><td>35.04</td><td>41.19</td><td>0</td></tr><tr><td>RemoteCLIP</td><td>69.40</td><td>20.57</td><td>17.19</td><td>62.22</td><td>53.75</td><td><u>56.23</u></td><td>34.43</td><td>19.86</td><td>43.11</td><td>23.85</td><td>53.32</td><td>1</td></tr><tr><td>SatlasNet</td><td>74.79</td><td><u>29.87</u></td><td>16.76</td><td>83.92</td><td>44.07</td><td>37.86</td><td>34.64</td><td>29.08</td><td><strong>49.78</strong></td><td>13.91</td><td>44.38</td><td>2</td></tr><tr><td>Scale-MAE</td><td>75.47</td><td>21.47</td><td>22.86</td><td>64.74</td><td><strong>56.06</strong></td><td>48.75</td><td>35.27</td><td>13.44</td><td><u>49.68</u></td><td>26.66</td><td>54.16</td><td>2</td></tr><tr><td>SpectralGPT</td><td><strong>83.35</strong></td><td>20.29</td><td>34.53</td><td>83.12</td><td>35.81</td><td>39.51</td><td>35.33</td><td>31.06</td><td>36.31</td><td>37.35</td><td><strong>39.44</strong></td><td>2</td></tr><tr><td>S12-MoCo</td><td>73.11</td><td>19.47</td><td>32.51</td><td>79.58</td><td>41.15</td><td>35.57</td><td>32.24</td><td>36.54</td><td>49.46</td><td>37.97</td><td>44.83</td><td>0</td></tr><tr><td>S12-DINO</td><td>75.93</td><td>23.47</td><td><u>36.62</u></td><td>84.95</td><td>41.02</td><td>34.63</td><td>32.78</td><td><strong>38.44</strong></td><td>41.15</td><td>37.91</td><td>42.74</td><td>2</td></tr><tr><td>S12-MAE</td><td>76.60</td><td>18.44</td><td>31.06</td><td>84.81</td><td>39.84</td><td>35.56</td><td>30.59</td><td>35.29</td><td>40.51</td><td>23.60</td><td>43.76</td><td>0</td></tr><tr><td>S12-Data2Vec</td><td>74.38</td><td>17.86</td><td>33.09</td><td>81.91</td><td>41.60</td><td>37.27</td><td>33.63</td><td>34.11</td><td>40.66</td><td>22.85</td><td>46.52</td><td>0</td></tr><tr><td>UNet Baseline</td><td><u>79.46</u></td><td>24.30</td><td>29.53</td><td><strong>88.55</strong></td><td>46.77</td><td>52.58</td><td>35.59</td><td>13.88</td><td>46.08</td><td>34.84</td><td>40.39</td><td>2</td></tr><tr><td>ViT Baseline</td><td>75.92</td><td>10.18</td><td><strong>38.44</strong></td><td>81.85</td><td>44.85</td><td><strong>56.53</strong></td><td>35.39</td><td>27.76</td><td>36.01</td><td><strong>39.20</strong></td><td>44.89</td><td>3</td></tr></tbody></table>

### C. Normalization Impact

In this ablation study, we examine the impact of the normalization strategy during the decoder fine-tuning stage. While our experiments normalized images using the mean and standard deviation of each band (following most of the original papers), this step was omitted in this ablation study to evaluate its effects. The analysis was conducted on both low-resolution datasets (HLS Burn Scars and MADOS) and a high-resolution dataset (SpaceNet 7).

The results in Table VIII compare evaluation performance with and without the normalization strategy. Generally, normalization affects low-resolution datasets more significantly than high-resolution ones. In the high-resolution dataset SpaceNet 7,

removing normalization has a minimal impact on performance; for example, the largest percentage difference is observed with RemoteCLIP, which shows only a 4% drop in mIoU. However, removing normalization causes quite large effects on low-resolution datasets. Particularly, models pre-trained on high-resolution data, are more susceptible to the absence of normalization in this setting. For instance, the performance of models like GFM-Swin, Scale-MAE, and RemoteCLIP drops significantly when applied to unnormalized data on both the HLS Burn Scars and MADOS datasets.
<table><caption>TABLE VIII: Ablation study on the impact of normalization strategies across three benchmark datasets.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">HLS Burns</th><th colspan="3">MADOS</th><th colspan="3">SpaceNet 7</th></tr><tr><th>w/</th><th>w/o</th><th>% diff</th><th>w/</th><th>w/o</th><th>% diff</th><th>w/</th><th>w/o</th><th>% diff</th></tr></thead><tbody><tr><td>CROMA</td><td>82.42</td><td>83.76</td><td>2%</td><td>67.55</td><td>54.21</td><td>-20%</td><td>59.28</td><td>58.09</td><td>-2%</td></tr><tr><td>DOFA</td><td>80.63</td><td>80.47</td><td>0%</td><td>59.58</td><td>51.55</td><td>-13%</td><td>61.84</td><td>60.41</td><td>-2%</td></tr><tr><td>GFM-Swin</td><td>76.90</td><td>68.94</td><td>-10%</td><td>64.71</td><td>50.39</td><td>-22%</td><td>60.89</td><td>58.87</td><td>-3%</td></tr><tr><td>Prithvi</td><td>83.62</td><td>81.47</td><td>-3%</td><td>49.98</td><td>40.31</td><td>-19%</td><td>56.54</td><td>56.80</td><td>0%</td></tr><tr><td>RemoteCLIP</td><td>76.59</td><td>66.04</td><td>-14%</td><td>60.00</td><td>32.82</td><td>-45%</td><td>57.76</td><td>55.41</td><td>-4%</td></tr><tr><td>SatlasNet</td><td>79.96</td><td>81.79</td><td>2%</td><td>55.86</td><td>62.66</td><td>12%</td><td>61.88</td><td>61.29</td><td>-1%</td></tr><tr><td>Scale-MAE</td><td>76.68</td><td>61.68</td><td>-20%</td><td>57.32</td><td>16.81</td><td>-71%</td><td>62.96</td><td>61.07</td><td>-3%</td></tr><tr><td>SpectralGPT</td><td>80.47</td><td>80.85</td><td>0%</td><td>57.99</td><td>53.50</td><td>-8%</td><td>58.86</td><td>58.06</td><td>-1%</td></tr><tr><td>S12-MoCo</td><td>81.58</td><td>82.61</td><td>1%</td><td>51.76</td><td>60.18</td><td>16%</td><td>57.64</td><td>56.73</td><td>-2%</td></tr><tr><td>S12-DINO</td><td>81.72</td><td>81.25</td><td>-1%</td><td>49.37</td><td>62.31</td><td>26%</td><td>56.47</td><td>55.39</td><td>-2%</td></tr><tr><td>S12-MAE</td><td>81.91</td><td>82.44</td><td>1%</td><td>49.90</td><td>60.48</td><td>21%</td><td>57.13</td><td>55.55</td><td>-3%</td></tr><tr><td>S12-Data2Vec</td><td>81.91</td><td>80.96</td><td>-1%</td><td>44.36</td><td>51.64</td><td>16%</td><td>58.23</td><td>56.53</td><td>-3%</td></tr></tbody></table>

### D. Matching Training Resolution of the Model

In this ablation study, we examine how adjusting the input resolution of test data to more closely match the resolution on which the GFM's were trained during the pre-training stage affects model performance. Specifically, we select models pre-trained on low-resolution data (i.e., S12-DINO and CROMA, pre-trained on 10m Sentinel-2 data, and Prithvi, pre-trained on 30m Harmonized Landsat and Sentinel-2 (HLS) data) and perform ablations on the high-resolution dataset FBP. For the FBP dataset, instead of cropping the images (as in previous experiments), we resized them using bilinear interpolation, effectively reducing the spatial resolution of the input data from 2 m/pixel to match the training resolution of the models.

The results in Table IX show that all models experience a significant performance drop when their input resolution is reduced to match the training resolutions. For instance, Prithvi's performance decreases from 46.81% mIoU to 30.99% mIoU, and CROMA's score drops from 51.83% mIoU to 21.10% mIoU. This highlights that fine-grained spatial details are essential for complex datasets like FBP, which involve many visually similar classes, even for models trained on lower-resolution data.

### E. Temporal Capabilities and Aggregation Strategies

We employ two distinct feature aggregation strategies for multi-temporal datasets (PASTIS-R, Crop Type Mapping-SS, BioMassters and DynamicEarthNet): a naive linear mapping and L-TAE. A comparative analysis, presented in Table X, reveals that L-TAE consistently enhances performance compared to the linear approach in all the datasets but DynamicEarthNet. Notably, L-TAE yields improvements ranging from 10% to 20% for the top-performing models on the PASTIS-R dataset. On DynamicEarthNet, however, L-TAE may lead to overfitting due to the minor temporal window spanned by the time series (6 days). See Appendix A for more insights. Generally speaking, these findings underscore the importance of effectively encoding temporal information in remote sensing tasks and highlight the potential of developing more sophisticated temporal aggregation strategies within GFM's.

### F. Domain Adaptation

The cross-region adaptation task on FBP presents a great challenge, primarily due to spatial autocorrelation. The results

are reported in Table XI. Here, the class distribution differs significantly from the in-domain test, with the absence of five minor classes that are present in the training data. This discrepancy in class representation greatly impacts performance. For instance, CROMA, which performs competitively in cross-sensor tests, sees a reduction to 18.38% in cross-region tests, while other strong GFM's like RemoteCLIP and GFM-Swin score 36.53% mIoU and 35.39% mIoU, respectively.

### G. Consideration on the Training Convergence

Fig. 5 illustrates the training loss trajectories of some of the most illustrative GFM's – Scale-MAE, RemoteCLIP, UNet, CROMA, and DOFA – across four datasets: FBP, MADOS, HLS Burn Scars, and Sen1Floods11. The trends reveal important insights into the GFM's' behavior depending on dataset difficulty and pre-training.

For complex datasets like FBP and MADOS, the pre-trained GFM's – with frozen encoders – tend to begin with lower loss values. This suggests that these models possess beneficial pre-learned features, enabling them to adapt more readily to challenging datasets without requiring significant fine-tuning. By contrast, UNet, which is end-to-end fine-tuned, starts with a higher loss but reduces it steadily, showing that while it initially lags, it progressively learns the data patterns, even if not as efficiently as models with strong initialization from pre-training.

In contrast, on easier datasets like HLS Burn Scars and Sen1Floods11, the training loss for all models, including the pre-trained ones, starts at relatively low values, indicating that the data may be less complex or more straightforward for feature extraction. In these cases, pre-trained models do not appear to retain a distinct advantage in initial loss values over UNet, which starts on nearly the same level. This behavior aligns with the fact that UNet, with end-to-end fine-tuning, performs well on simpler datasets, as it can optimize its parameters fully to match the dataset characteristics. Interestingly, on Sen1Floods11, UNet's stronger performance despite higher training loss values hints at potential overfitting by the foundation models, as they may overly adapt to training data without capturing the generalization needed for evaluation.
<table><caption>TABLE IX: Ablation study on the effect of downsampling the input resolution to match the model's training resolution on the FiveBillionPixels dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">FiveBillionPixels</th></tr><tr><th>w/o matching</th><th>w/ matching</th><th>% diff</th></tr></thead><tbody><tr><td>CROMA</td><td>51.83</td><td>21.10</td><td>-59%</td></tr><tr><td>Prithvi</td><td>46.81</td><td>30.99</td><td>-34%</td></tr><tr><td>S12-DINO</td><td>51.15</td><td>36.75</td><td>-28%</td></tr></tbody></table>

<table><caption>TABLE X: Comparison of different multi-temporal strategies on multi-temporal datasets. L-TAE generally demonstrates an advantage over linear temporal mapping.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">BioMassters</th><th colspan="2">Crop Type Mapping-SS</th><th colspan="2">PASTIS-R</th><th colspan="2">DynamicEarthNet</th></tr><tr><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th></tr></thead><tbody><tr><td>CROMA</td><td>39.99</td><td>36.81</td><td>47.02</td><td>49.38</td><td>32.06</td><td>38.51</td><td>38.25</td><td>38.29</td></tr><tr><td>DOFA</td><td>43.25</td><td>42.81</td><td>49.81</td><td>51.33</td><td>24.49</td><td>30.02</td><td>40.84</td><td>39.29</td></tr><tr><td>GFM-Swin</td><td>47.96</td><td>46.83</td><td>39.72</td><td>46.98</td><td>15.13</td><td>21.24</td><td>35.78</td><td>34.09</td></tr><tr><td>Prithvi</td><td>42.90</td><td>39.99</td><td>39.92</td><td>43.07</td><td>27.40</td><td>33.93</td><td>28.42</td><td>27.86</td></tr><tr><td>RemoteCLIP</td><td>49.60</td><td>49.79</td><td>46.50</td><td>52.05</td><td>13.31</td><td>18.23</td><td>34.07</td><td>31.78</td></tr><tr><td>Scale-MAE</td><td>52.24</td><td>47.15</td><td>21.39</td><td>25.42</td><td>11.74</td><td>24.55</td><td>35.47</td><td>35.11</td></tr><tr><td>SpectralGPT</td><td>38.55</td><td>36.11</td><td>53.50</td><td>46.95</td><td>27.52</td><td>35.44</td><td>39.27</td><td>37.85</td></tr><tr><td>S12-MoCo</td><td>41.82</td><td>40.21</td><td>44.22</td><td>48.58</td><td>26.09</td><td>34.49</td><td>36.53</td><td>35.44</td></tr><tr><td>S12-DINO</td><td>41.54</td><td>41.23</td><td>46.56</td><td>48.66</td><td>32.45</td><td>36.18</td><td>33.80</td><td>34.81</td></tr><tr><td>S12-MAE</td><td>42.54</td><td>41.07</td><td>46.28</td><td>45.80</td><td>27.06</td><td>32.03</td><td>35.23</td><td>34.08</td></tr><tr><td>S12-Data2Vec</td><td>42.02</td><td>41.91</td><td>54.01</td><td>54.03</td><td>26.94</td><td>34.32</td><td>36.41</td><td>35.90</td></tr><tr><td>ViT Baseline</td><td>40.94</td><td>38.55</td><td>50.24</td><td>51.78</td><td>36.57</td><td>38.53</td><td>36.83</td><td>36.38</td></tr></tbody></table>

<table><caption>TABLE XI: Domain Adaptation experiments for FiveBillionPixels, comparing model performance on dataset splits without and with regional domain gaps.</caption><thead><tr><th>Model</th><th>w/o regional domain gap</th><th>w/ regional domain gap</th><th>% diff</th></tr></thead><tbody><tr><td>CROMA</td><td>51.83</td><td>18.38</td><td>-65%</td></tr><tr><td>DOFA</td><td>43.18</td><td>16.56</td><td>-62%</td></tr><tr><td>GFM-Swin</td><td>67.18</td><td>35.39</td><td>-47%</td></tr><tr><td>Prithvi</td><td>46.81</td><td>16.75</td><td>-64%</td></tr><tr><td>RemoteCLIP</td><td>69.19</td><td>36.53</td><td>-47%</td></tr><tr><td>SatlasNet</td><td>50.97</td><td>21.13</td><td>-59%</td></tr><tr><td>Scale-MAE</td><td>67.19</td><td>34.88</td><td>-48%</td></tr><tr><td>SpectralGPT</td><td>33.42</td><td>18.34</td><td>-45%</td></tr><tr><td>S12-MoCo</td><td>53.02</td><td>17.30</td><td>-67%</td></tr><tr><td>S12-DINO</td><td>51.15</td><td>18.64</td><td>-64%</td></tr><tr><td>S12-MAE</td><td>51.92</td><td>17.78</td><td>-66%</td></tr><tr><td>S12-Data2Vec</td><td>48.82</td><td>18.23</td><td>-63%</td></tr><tr><td>UNet Baseline</td><td>60.47</td><td>30.14</td><td>-50%</td></tr><tr><td>ViT Baseline</td><td>59.32</td><td>22.77</td><td>-62%</td></tr></tbody></table>

## H. Impact of Encoder Fine-tuning

In Table XII, we analyzed the effect of end-to-end fine-tuning on the performance of one of the best and worst-performing GFM s across all the datasets. From the results, we can observe that the impact of fine-tuning is not consistent across the GFM s and datasets. In some cases, fine-tuning improves the performance, while in others, it results in a decline. For instance, RemoteCLIP shows a noticeable drop in performance on HLS BurnsScars, with the model's performance dropping from 76.59% (without fine-tuning) to 65.13% (with fine-tuning) mIoU. This suggests that fine-tuning might have caused the model to lose some of the encoded information from the pre-trained encoder, possibly related to text features. On the other hand, GFM s pre-trained on low-resolution data tend to exhibit more positive results, with constant or better performance. For example, CROMA, shows stability in performance after fine-

tuning on MADOS, moving from 67.55% to 66.98% mIoU. DOFA shows improved results after fine-tuning on FBP, from 43.82% to 63.74% mIoU, suggesting that fine-tuning may help refine models trained on lower-resolution data to adapt more effectively to task-specific features. Overall, the lack of a clear, consistent pattern highlights that fine-tuning does not always guarantee an improvement and that the success of fine-tuning depends on the GFM's pre-training, dataset characteristics, and the specific task at hand. Moreover, fine-tuning requires more resources, given the need for backpropagating the gradient on all the parameters.

## I. Sensitivity to Hyperparameters

For PANGAEA, we aim to adopt a standardized protocol that avoids the need for extensive hyperparameter fine-tuning for every new GFM tested. To achieve this, we set up an
<figure><img src="image_6.png" /><figcaption>Fig. 5: Convergence of training losses for four GFM s – one pre-trained on high-resolution (Scale-MAE), one on text-image (RemoteCLIP), one on different modalities and resolution (DOFA) and one on low-resolution Sentinel data – and a baseline (UNet), on different datasets (i.e. different domains and resolutions).</figcaption></figure>

<table><caption>TABLE XII: Impact of encoder fine-tuning on the performance of the best and worst-performing GFM s for each dataset. The table shows results without (w/o) and with (w/) fine-tuning, along with the relative percentage difference (% diff). Positive values indicate improvement, while negative values denote performance degradation. Results highlight the variability in fine-tuning effects across different models and datasets.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Model</th><th colspan="3">Best model</th><th colspan="3">Worst model</th></tr><tr><th>w/o</th><th>w/</th><th>% diff</th><th>w/o</th><th>w/</th><th>% diff</th></tr></thead><tbody><tr><td>HLS Burns</td><td>Prithvi</td><td>83.62</td><td>85.52</td><td>+2%</td><td>RemoteCLIP</td><td>76.59</td><td>65.13</td><td>-15%</td></tr><tr><td>MADOS</td><td>CROMA</td><td>67.55</td><td>66.98</td><td>-1%</td><td>S12-DINO</td><td>49.38</td><td>56.98</td><td>+15%</td></tr><tr><td>PASTIS</td><td>S12-DINO</td><td>36.18</td><td>44.03</td><td>+22%</td><td>SatlasNet</td><td>17.51</td><td>29.05</td><td>+66%</td></tr><tr><td>Sen1Floods11</td><td>CROMA</td><td>90.89</td><td>90.87</td><td>0%</td><td>GFM-Swin</td><td>72.60</td><td>75.70</td><td>+4%</td></tr><tr><td>xView2</td><td>ScaleMAE</td><td>60.72</td><td>62.16</td><td>+2%</td><td>Prithvi</td><td>49.35</td><td>57.33</td><td>+16%</td></tr><tr><td>FBP</td><td>RemoteCLIP</td><td>69.19</td><td>55.58</td><td>-20%</td><td>DOFA</td><td>43.82</td><td>63.74</td><td>+45%</td></tr><tr><td>DynEarthNet</td><td>DOFA</td><td>39.29</td><td>37.76</td><td>-4%</td><td>Prithvi</td><td>27.86</td><td>29.21</td><td>+5%</td></tr><tr><td>CropMap</td><td>S12-Data2Vec</td><td>54.03</td><td>56.27</td><td>+4%</td><td>Scale-MAE</td><td>25.42</td><td>20.17</td><td>-21%</td></tr><tr><td>SN7</td><td>Scale-MAE</td><td>62.96</td><td>53.98</td><td>-14%</td><td>S12-DINO</td><td>56.47</td><td>56.55</td><td>0%</td></tr><tr><td>AI4Farms</td><td>CROMA</td><td>25.65</td><td>24.78</td><td>-3%</td><td>S12-MAE</td><td>24.69</td><td>23.73</td><td>-4%</td></tr><tr><td>BioMassters</td><td>SpectralGPT</td><td>36.11</td><td>35.86</td><td>-1%</td><td>RemoteCLIP</td><td>49.79</td><td>46.34</td><td>-7%</td></tr></tbody></table>

environment where learning rate tuning should have the least possible impact. The ADAM optimizer used in our experiments is known to be robust to changes in learning rate. In addition, the datasets are normalized, the decoder architecture is the same for each experiment, and it includes batch normalization layers. Because of this, we expect the magnitude of gradients to be fairly even between models.

To verify this assumption, we optimize the learning rate on the MADOS and HLS Burn Scars datasets by evaluating seven possible learning rates over a log scale, from $10^{-2}$ down to $10^{-5}$. Table XIII shows the results with the default learning rate of $10^{-4}$, versus the best result of the optimization

for the specific model-dataset combination, and their relative difference.

The results indicate that in general our assumption stands, and the experiments are fairly stable under learning rate tuning. In the specific case of MADOS and the S12 family, the results do improve by tuning the learning rate, but even with this improvement, the overall ranking of models doesn't change significantly. The S12 family of models is still in close competition with Prithvi for the worst-performing GFM on this dataset.

This learning rate agnosticism is even more pronounced on the HLS Burn Scars experiment, where the results change
even less. One interesting detail is that some models lose
performance in the tuning experiments, meaning that the
stochasticity of running multiple training runs is in the same
order of magnitude as the gains achievable by learning rate
tuning.

## VI. LIMITATIONS

Despite its contributions, this work has some limitations. First, the focus is predominantly on computer vision-based EO tasks, leaving out non-vision applications such as climate modeling or weather forecasting. Second, the computational resources required for training and fine-tuning GFM s may limit accessibility for resource-constrained environments. Additionally, while Pangaea includes diverse datasets, there is room for expanding geographical and application domain coverage. Future work should also explore multi-sensor data integration beyond satellite imagery, incorporating aerial and drone datasets to further enhance GFM applicability. Finally, from a protocol standardization point of view, we foresee releasing new features that can help the fast and consistent reproducibility of our results and the benchmark of newly added GFM s. These features include the release of a fixed stratified subsample for each dataset, and an arena-mode benchmarking, with fixed command lines and randomness to launch experiments and obtain final scores.

## VII. FUTURE DIRECTIONS

Building on the foundation laid by PANGAEA, we plan
several key directions to further advance the benchmarking
and development of GFM s. First, we plan to expand the
range of downstream tasks beyond the current set to in-
clude object detection, image-level classification, and kNN,
some of which are already supported in https://github.com/
VMarsocci/pangaea-bench. These tasks offer diverse evalua-
tion perspectives and reflect practical needs in operational
RS workflows. Second, we aim to develop thematic sub-
benchmarks focused on specific application domains — such
as agriculture, disaster response and damage assessment, and
biodiversity monitoring—to enable a more targeted assessment
of GFM performance in specialized contexts. Additionally,
we will investigate representation quality through embedding-
based analyses, assessing how well GFM s capture semantic
similarities and structure in EO. Lastly, we are interested in
exploring model robustness and carbon footprint issues, to
monitor both security and environmental aspects of GFM s.
Together, these extensions aim to solidify PANGAEA not only
as a static benchmark but as a dynamic and evolving platform
for guiding the next generation of GFM s.

## VIII. CONCLUSIONS

The Pangaea Benchmark addresses critical gaps in the
evaluation of GFM s. By incorporating diverse downstream tasks
(i.e. semantic segmentation, change detection, and regression)
and employing datasets that encompass various resolutions,
sensor types, and application domains (e.g. urban, agriculture,
forestry), it provides a robust framework for assessing GFM
generalization and performance under real-world conditions.

We advocate for surveying and testing a representative set
of GFM approaches, including vision language-based models
and fully-supervised learning paradigms, to gain a deeper
understanding of their strengths and weaknesses. Our analysis
demonstrates the strengths and weaknesses of GFM s under
different scenarios (e.g. label scarcity, domain shift, etc...),
especially w.r.t. specialized baseline, that most of the time
can still be a strong counterpart. Furthermore, the open-source
nature of this benchmark fosters transparency and collaboration,
driving progress in the development of adaptable and effective
GFM s.

## CONTRIBUTORS

**Conceptualization.** VM AN NA Methodology. VM AN NA Codebase prototype. VM YJ GL DK LZ Semantic Segmentation. YJ VM DK GL LZ EB AS Change Detection. SH SG HF Regression. RY EB Ablation Studies. VM HF DK YJ RY GL Manuscript Revision. AN MV NA Supervision. AN NA MV YB Fundings. AN NA MV YB

## ACKNOWLEDGMENTS

Georges Le Bellier and Nicolas Audebert's work was
conducted as part of the research project MAGE (ANR-22-
CE23-0010) funded by the *Agence Nationale de la Recherche*.
KTH researchers were supported by the EO-AI4GlobalChange
project, funded by Digital Futures, Stockholm, Sweden, and by
Andrea Nascetti's startup grant funded by KTH Royal Institute
of Technology.

## REFERENCES

[1] Gengchen Mai, Chris Cundy, Kristy Choi, Yingjie Hu, Ni Lao, and Stefano Ermon. Towards a foundation model for geospatial artificial intelligence (vision paper). In *Proceedings of the 30th International Conference on Advances in Geographic Information Systems*, SIGSPATIAL '22, New York, NY, USA, 2022. Association for Computing Machinery. 1

[2] Alexandre Lacoste, Evan David Sherwin, Hannah Kerner, Hamed Alemohammad, Björn Lütjens, Jeremy Irvin, David Dao, Alex Chang, Mehmet Gunturkun, Alexandre Drouin, Pau Rodriguez, and David Vazquez. Toward foundation models for earth monitoring: Proposal for a climate change benchmark, 2021. 1

[3] Devis Tuia, Konrad Schindler, Begüm Demir, Gustav Camps-Valls, Xiao Xiang Zhu, Mrinalini Kochupillai, Sašo Džeroski, Jan N. van Rijn, Holger H. Hoos, Fabio Del Frate, Mihai Datcu, Jorge-Arnulfo Quiané-Ruiz, Volker Markl, Bertrand Le Saux, and Rochelle Schneider. Artificial intelligence to advance earth observation: a perspective, 2023. 1

[4] Hao Zhang, Jin-Jian Xu, Hong-Wei Cui, Lin Li, Yaowen Yang, Chao-Sheng Tang, and Niklas Boers. When geoscience meets foundation models: Towards general geoscience artificial intelligence system, 2024. 1

[5] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7):2217-2226, 2019. 1, 3

[6] Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, and Liangpei Zhang. Mtp: Advancing remote sensing foundation model via multi-task pretraining, 2024. 1, 5

[7] Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, Chris Cundy, Ziyuan Li, Rui Zhu, and Ni Lao. On the opportunities and challenges of foundation models for geospatial artificial intelligence, 2023. 1
<table><caption>TABLE XIII: Performance comparison of various models on MADOS and HLS Burn Scars with a learning rate tuning. The table reports mIoU scores and the relative percentage difference (% diff) between untuned (w/o) and tuned (w/) learning rate configurations.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">MADOS</th><th colspan="3">HLSBurnScars</th></tr><tr><th>w/o</th><th>w/</th><th>% diff</th><th>w/o</th><th>w/</th><th>% diff</th></tr></thead><tbody><tr><td>CROMA</td><td><strong>67.55</strong></td><td><strong>66.98</strong></td><td>-1%</td><td>82.42</td><td><strong>83.25</strong></td><td>+1%</td></tr><tr><td>DOFA</td><td>59.58</td><td>61.56</td><td>+3%</td><td>80.63</td><td>82.04</td><td>+2%</td></tr><tr><td>GFM-Swin</td><td><strong>64.71</strong></td><td><strong>66.48</strong></td><td>+3%</td><td>76.90</td><td>76.48</td><td>-1%</td></tr><tr><td>Prithvi</td><td>49.98</td><td>53.83</td><td>+8%</td><td><strong>83.62</strong></td><td>82.83</td><td>-1%</td></tr><tr><td>RemoteCLIP</td><td>60.00</td><td>60.93</td><td>+2%</td><td>76.59</td><td>76.65</td><td>0%</td></tr><tr><td>SatlasNet</td><td>55.86</td><td>58.31</td><td>+4%</td><td>79.96</td><td>80.88</td><td>+1%</td></tr><tr><td>Scale-MAE</td><td>57.32</td><td>57.63</td><td>+1%</td><td>76.68</td><td>76.90</td><td>0%</td></tr><tr><td>SpectralGPT</td><td>57.99</td><td>58.13</td><td>0%</td><td>80.47</td><td>80.65</td><td>0%</td></tr><tr><td>S12-Data2Vec</td><td>44.36</td><td>54.29</td><td>+22%</td><td>81.91</td><td>81.74</td><td>0%</td></tr><tr><td>S12-DINO</td><td>49.37</td><td>55.75</td><td>+13%</td><td>81.72</td><td>81.67</td><td>0%</td></tr><tr><td>S12-MAE</td><td>49.90</td><td>51.66</td><td>+4%</td><td>81.91</td><td>81.18</td><td>-1%</td></tr><tr><td>S12-MoCo</td><td>51.76</td><td>52.15</td><td>+1%</td><td>81.58</td><td>81.14</td><td>-1%</td></tr><tr><td>UNet Baseline</td><td>54.79</td><td>54.79</td><td>0%</td><td><strong>84.51</strong></td><td><strong>85.68</strong></td><td>+1%</td></tr><tr><td>ViT Baseline</td><td>48.19</td><td>48.52</td><td>+1%</td><td>81.58</td><td>80.76</td><td>-1%</td></tr></tbody></table>

[8] Esther Rolf, Konstantin Klemmer, Caleb Robinson, and Hannah Kerner. Mission critical – satellite data is a distinct modality in machine learning, 2024. 1

[9] Casper Fibaek, Luke Camilleri, Andreas Luyts, Nikolaos Dionelis, and Bertrand Le Saux. Phileo bench: Evaluating geo-spatial foundation models, 2024. 1, 6

[10] Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, and Xiao Xiang Zhu. Geo-bench: Toward foundation models for earth monitoring, 2023. 1, 3, 6, 7

[11] Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B. Lobell, and Stefano Ermon. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning, 2021. 1, 6, 8

[12] Nikolaos Ioannis Bountos, Arthur Ouaknine, and David Rolnick. Fomo-bench: a multi-modal, multi-scale and multi-task forest monitoring benchmark for remote sensing foundation models. arXiv preprint arXiv:2312.10114, 2023. 1, 6

[13] Yi-Chia Chang, Adam J. Stewart, Favyen Bastani, Piper Wolters, Shreya Kannan, George R. Huber, Jingtong Wang, and Arindam Banerjee. On the generalizability of foundation models for crop type mapping. ArXiv, abs/2409.09451, 2024. 1, 2, 6

[14] Anthony Fuller, Koreen Millard, and James R. Green. Croma: Remote sensing representations with contrastive radar-optical masked autoencoders, 2023. 2, 4, 9

[15] Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning, 2023. 2, 4, 5, 9, 10, 33

[16] Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. Satlas: A large-scale, multi-task dataset for remote sensing image understanding. arXiv preprint arXiv:2211.15660, 2022. 2, 5, 9

[17] Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle to beat supervised baselines, 2024. 2

[18] Esther Rolf, Lucia Gordon, Milind Tambe, and Andrew Davies. Contrasting local and global modeling with machine learning and satellite data: A case study estimating tree canopy height in african savannas, 2024. 2

[19] Zhitong Xiong, Fahong Zhang, Yi Wang, Yilei Shi, and Xiao Xiang Zhu. Earthnets: Empowering ai in earth observation, 2024. 3, 4, 6

[20] Michael Schmitt, Seyed Ali Ahmadi, Yonghao Xu, Gülşen Taşkin, Ujjwal Verma, Francescopaolo Sica, and Ronny Hänsch. There are no data like more data: Datasets for deep learning in earth observation. IEEE Geoscience and Remote Sensing Magazine, 11(3):63–97, September 2023. 3, 4, 7

[21] Yi Yang and Shawn Newsam. Bag-of-visual-words and spatial

extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270–279, 2010. 3

[22] Gui-Song Xia, Wen Yang, Julie Delon, Yann Gousseau, Hong Sun, and Henri Maître. Structural high-resolution satellite image indexing. In Symposium: 100 Years ISPRS - Advancing Remote Sensing Science, Vienna, Austria, 2010. 3

[23] Dengxin Dai and Wen Yang. Satellite image classification via two-layer sparse coding with biased image representation. IEEE Transactions on Geoscience and Remote Sensing, 8(1):173–176, 2011. 3

[24] Yang Long, Gui-Song Xia, Liangpei Zhang, Gong Cheng, and Deren Li. Aerial scene parsing: From tile-level scene classification to pixel-wise semantic labeling, 2022. 3

[25] Yang Long, Gui-Song Xia, Shengyang Li, Wen Yang, Michael Ying Yang, Xiao Xiang Zhu, Liangpei Zhang, and Deren Li. On creating benchmark dataset for aerial image interpretation: Reviews, guidances and million-aid. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:4205–4230, 2021. 3

[26] Gencer Sumbul, Marcela Charfuelan, Begüm Demir, and Volker Markl. Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pages 5901–5904. IEEE, 2019. 3

[27] Gencer Sumbul, Arne De Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mario Caetano, Begüm Demir, and Volker Markl. Bigearthnet-mm: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets]. IEEE Geoscience and Remote Sensing Magazine, 9(3):174–180, 2021. 3, 6

[28] Bryan Zhu, Nicholas Lui, Jeremy Irvin, Jimmy Le, Sahil Tadwalkar, Chenghao Wang, Zutao Ouyang, Frankie Y. Liu, Andrew Y. Ng, and Robert B. Jackson. Meter-ml: A multi-sensor earth observation benchmark for automated methane source mapping, 2022. 3

[29] Dan M. Kluger, Sherrie Wang, and David B. Lobell. Two shifts for crop mapping: Leveraging aggregate crop statistics to improve satellite-based maps in new regions. Remote Sensing of Environment, 262:112488, 2021. 3

[30] Edemir Ferreira, Matheus Brito, Remis Balaniuk, Mário S Alvim, and Jefersson A dos Santos. Brazildam: A benchmark dataset for tailings dam detection. In 2020 IEEE Latin American GRSS & ISPRS Remote Sensing Conference (LAGIRS), pages 339–344. IEEE, 2020. 3

[31] Jeremy Irvin, Hao Sheng, Neel Ramachandran, Sonja Johnson-Yu, Sharon Zhou, Kyle Story, Rose Rustowicz, Cooper Elsworth, Kemen Austin, and Andrew Y. Ng. Forestnet: Classifying drivers of deforestation in Indonesia using deep learning on satellite imagery, 2020. 3

[32] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172–6180, 2018. 3, 6

[33] Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and a new
benchmark. *ISPRS journal of photogrammetry and remote sensing*, 159:296-307, 2020. 3, 6

[34] J Ding, N Xue, GS Xia, X Bai, W Yang, MY Yang, S Belongie, J Luo, M Datcu, M Pelillo, et al. Object detection in aerial images: A large-scale benchmark and challenges. *arXiv preprint arXiv:2102.12219*, 2021. 3

[35] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 28–37, 2019. 3

[36] Fernando Paolo, Tsu ting Tim Lin, Ritwik Gupta, Bryce Goodman, Nirav Patel, Daniel Kuster, David Kroodsma, and Jared Dunnmon. xview3-sar: Detecting dark fishing activity using synthetic aperture radar imagery. 2022. 3

[37] Lanqing Huang, Bin Liu, Boying Li, Weiwei Guo, Wenhao Yu, Zenghui Zhang, and Wenxian Yu. Opensarship: A dataset dedicated to sentinel-1 ship interpretation. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 11(1):195–208, 2017. 3

[38] Jianwei Li, Changwen Qu, and Jiaqi Shao. Ship detection in sar images based on an improved faster r-cnn. In *2017 SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA)*, pages 1–6. IEEE, 2017. 3

[39] Jakaria Rabbi, Nilanjan Ray, Matthias Schubert, Subir Chowdhury, and Dennis Chao. Small-object detection in remote sensing images with end-to-end edge-enhanced gan and object detector network. *Remote Sensing*, 12(9):1432, 2020. 3

[40] Sebastien Razakarivony and Frederic Jurie. Vehicle detection in aerial imagery: A small target detection benchmark. *Journal of Visual Communication and Image Representation*, 34:187–203, 2016. 3

[41] Geremy Heitz and Daphne Koller. Learning spatial context: Using stuff to find things. In *Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part 1 10*, pages 30–43. Springer, 2008. 3

[42] Franz Rottensteiner, Gunho Sohn, Jaewook Jung, Markus Gerke, Caroline Baillard, Sebastien Benitez, and Uwe Breitkopf. The isprs benchmark on urban object classification and 3d building reconstruction. *ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences I-3 (2012)*, Nr. 1, 1(1):293–298, 2012. 3

[43] Junjie Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. *Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation*. 2022. 3

[44] Xin-Yi Tong, Gui-Song Xia, and Xiao Xiang Zhu. Enabling country-scale land cover mapping with meter-resolution satellite imagery. *ISPRS Journal of Photogrammetry and Remote Sensing*, 196:178–196, 2023. 3, 8

[45] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge to parse the earth through satellite images. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops*, pages 172–181, 2018. 3

[46] Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sebastien Giordano, and Boris Watrelos. FLAIR: a country-scale land cover semantic segmentation dataset from multi-source optical imagery. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023. 3

[47] Steve Foga, Pat L Scaramuzza, Song Guo, Zhe Zhu, Ronald D Dilley Jr, Tim Beckmann, Gail L Schmidt, John L Dwyer, M Joseph Hughes, and Brady Laue. Cloud detection algorithm comparison and validation for operational landsat data products. *Remote sensing of environment*, 194:379–390, 2017. 3

[48] Sorour Mohajerani and Parvaneh Saeedi. Cloud-net: An end-to-end cloud detection algorithm for landsat 8 imagery. In *IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium*, pages 1029–1032. IEEE, 2019. 3

[49] Zhiwei Li, Huanfeng Shen, Qing Cheng, Yuhao Liu, Shucheng You, and Zongyi He. Deep learning based cloud detection for remote sensing images by the fusion of multi-scale convolutional features. *arXiv preprint arXiv:1810.05801*, page 7, 2018. 3

[50] Wojciech Sirko, Emmanuel Asiedu Brempong, Juliana T. C. Marcos, Abigail Annkah, Abel Korme, Mohammed Alewi Hassen, Krishna Sapkota, Tomer Shekel, Abdoulaye Diack, Sella Nevo, Jason Hickey, and John Quinn. High-resolution building and road detection from sentinel-2. *ArXiv*, abs/2310.11622, 2023. 3

[51] Adam Van Etten, Dave Lindenbaum, and Todd M. Bacastow. *Spacenet: A remote sensing dataset and challenge series*, 2019. 3, 7, 8

[52] Rose M Rustowicz, Robin Cheong, Lijing Wang, Stefano Ermon, Marshall Burke, and David Lobell. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In *Proceedings of the IEEE/cvf conference on computer vision and pattern recognition workshops*, pages 75–82, 2019. 3, 7

[53] Vivien Sainte Fare Garnot, Loic Landrieu, and Nesrine Chehata. Multi-modal temporal attention models for crop mapping from satellite time series. *ISPRS Journal of Photogrammetry and Remote Sensing*, 2022. 3, 7, 8

[54] Luigi T Luppino, Filippo M Bianchi, Gabriele Moser, and Stian N Anfinsen. Unsupervised image regression for heterogeneous change detection. *arXiv preprint arXiv:1909.05948*, 2019. 3

[55] Aysim Toker, Lukas Kondmann, Mark Weber, Marvin Eisenberger, Andrés Camero, Jingliang Hu, Ariadna Pregel Hoderlein, Çaglar Şenaras, Timothy Davis, Daniel Cremers, et al. Dynamicearthnet: Daily multi-spectral satellite dataset for semantic change segmentation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 21158–21167, 2022. 4, 7, 8

[56] Hao Chen and Zhenwei Shi. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection. *Remote Sensing*, 12(10), 2020. 4

[57] Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery, 2019. 4, 7, 8, 25

[58] Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. Creating xbd: A dataset for assessing building damage from satellite imagery. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops*, June 2019. 4, 8

[59] Csaba Benedek and Tamás Szirányi. Change detection in optical aerial images by a multilayer conditional mixed markov model. *IEEE Transactions on Geoscience and Remote Sensing*, 47(10):3416–3430, 2009. 4

[60] Kunping Yang, Gui-Song Xia, Zicheng Liu, Bo Du, Wen Yang, Marcello Pelillo, and Liangpei Zhang. Asymmetric siamese networks for semantic change detection in aerial images. *IEEE Transactions on Geoscience and Remote Sensing*, 60:1–18, 2021. 4

[61] Rodrigo Caye Daudt, Bertr Le Saux, and Alexandre Boulch. Fully convolutional siamese networks for change detection. In *2018 25th IEEE international conference on image processing (ICIP)*, pages 4063–4067. IEEE, 2018. 4

[62] Mélisande Teng, Amna Elmustafa, Benjamin Akera, Yoshua Bengio, Hager Radi, Hugo Larochelle, and David Rolnick. Satbird: a dataset for bird species distribution modeling using remote sensing and citizen science data. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023. 4

[63] Sungduk Yu, Walter Hannah, Liran Peng, Jerry Lin, Mohamed Aziz Bhouri, Ritwik Gupta, Björn Lütjens, Justus Christopher Will, Gunnar Behrens, Julius Busecke, Nora Loose, Charles I Stern, Tom Beucler, Bryce Harrop, Benjamin R Hillman, Andrea Jenney, Savannah Ferretti, Nana Liu, Anima Anandkumar, Noah D Brenowitz, Veronika Eyring, Nicholas Geneva, Pierre Gentine, Stephan Mandt, Jaideep Pathak, Akshay Subramaniam, Carl Vondrick, Rose Yu, Laure Zanna, Tian Zheng, Ryan Abernathey, Fiaz Ahmed, David C Bader, Pierre Baldi, Elizabeth Barnes, Christopher Bretherton, Peter Caldwell, Wayne Chuang, Yilun Han, Yu Huang, Fernando Iglesias-Suarez, Sanket Jantre, Karthik Kashinath, Marat Khairoutdinov, Thorsten Kurth, Nicholas Lutsko, Po-Lun Ma, Griffin Mooers, J. David Neelin, David Randall, Sara Shamekh, Mark A Taylor, Nathan Urban, Janni Yuval, Guang Zhang, and Michael Pritchard. Climsim: A large multi-scale dataset for hybrid physics-ml climate emulation, 2024. 4

[64] Andrea Nascetti, RITU YADAV, Kirill Brodt, Qixun Qu, Hongwei Fan, Yuri Shendryk, Isha Shah, and Christine Chung. Biomasters: A benchmark dataset for forest biomass estimation using multi-modal satellite time-series. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023. 4, 7, 8

[65] V Coletta, V Marsocci, and R Ravanelli. 3dcd: A new dataset for 2d and 3d change detection using deep learning techniques. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 43:1349–1354, 2022. 4

[66] Valerio Marsocci, Virginia Coletta, Roberta Ravanelli, Simone Scardapane, and Mattia Crespi. Inferring 3d change detection from bitemporal optical images. *ISPRS Journal of Photogrammetry and Remote Sensing*, 196:325–339, 2023. 4
[67] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Erman, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021. 4

[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021. 4

[69] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, *Advances in Neural Information Processing Systems*, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. 4

[70] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, *Advances in Neural Information Processing Systems*, volume 35, pages 25278–25294. Curran Associates, Inc., 2022. 4

[71] Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurung, Sam Khallaghi, Hanxi, Li, Michael Cecil, Maryam Ahmadi, Fatemeh Kordi, Hamed Alemohammad, Manil Maskey, Raghu Ganti, Kommy Weldemariam, and Rahul Ramachandran. Foundation models for generalist geospatial artificial intelligence. 2023. 4, 5, 8, 9, 23

[72] Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham, Lichao Mou, and Xiao Xiang Zhu. Self-supervised learning in remote sensing: A review. arXiv preprint arXiv:2206.13188, 2022. 5

[73] Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, and Dacheng Tao. An empirical study of remote sensing pretraining. *IEEE Transactions on Geoscience and Remote Sensing*, 2022. 5

[74] Adam Stewart, Nils Lehmann, Isaac Corley, Yi Wang, Yi-Chia Chang, Nassim Ait Ait Ali Braham, Shradha Sehgal, Caleb Robinson, and Arindam Banerjee. Ssl4eo-1: Datasets and foundation models for landsat imagery. *Advances in Neural Information Processing Systems*, 36, 2024. 5, 8

[75] Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M. Albrecht, and Xiao Xiang Zhu. Ssl4eo-s12: A large-scale multimodal, multitemporal dataset for self-supervised learning in earth observation [software and data sets]. *IEEE Geoscience and Remote Sensing Magazine*, 11(3):98–106, 2023. 5, 9

[76] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 16000–16009, 2022. 5

[77] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, *Proceedings of the 39th International Conference on Machine Learning*, volume 162 of *Proceedings of Machine Learning Research*, pages 1298–1312. PMLR, 17–23 Jul 2022. 5

[78] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294, 2021. 5

[79] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In *Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)*, pages 9729–9738, Seattle, WA, USA, 2020. 5

[80] Zhitong Xiong, Yi Wang, Fahong Zhang, and Xiao Xiang Zhu. One for all: Toward unified foundation models for earth vision, 2024. 5

[81] Keumgang Cha, Junghoon Seo, and Taekyung Lee. A billion-scale foundation model for remote sensing images. arXiv preprint arXiv:2304.05215, 2023. 5

[82] Dilxat Muhtar, Xueliang Zhang, Pengfeng Xiao, Zhenshi Li, and Feng Gu. Cmid: A unified self-supervised learning framework for remote sensing image understanding. *IEEE Transactions on Geoscience and Remote Sensing*, 2023. 5

[83] Xian Sun, Peijin Wang, Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong, Zhujun Yang, Hao Chang, et al. Ringmo: A remote sensing foundation model with masked image modeling. *IEEE Transactions on Geoscience and Remote Sensing*, 2022. 5

[84] Fanglong Yao, Wanxuan Lu, Heming Yang, Liangyu Xu, Chenglong Liu, Leiyi Hu, Hongfeng Yu, Nayu Liu, Chubo Deng, Deke Tang, Changshuo Chen, Jiaqi Yu, Xian Sun, and Kun Fu. Ringmo-sense: Remote sensing foundation model for spatiotemporal prediction via spatiotemporal evolution disentangling. *IEEE Transactions on Geoscience and Remote Sensing*, 61:1–21, 2023. 5

[85] Zhiyuan Yan, Junxi Li, Xuexue Li, Ruixue Zhou, Wenkai Zhang, Yingchao Feng, Wenhui Diao, Kun Fu, and Xian Sun. Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images. *IEEE Transactions on Geoscience and Remote Sensing*, 61:1–16, 2023. 5

[86] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. 5

[87] Mingming Zhang, Qingjie Liu, and Yunhong Wang. Ctxmim: Context-enhanced masked image modeling for remote sensing image understanding. arXiv preprint arXiv:2310.00022, 2023. 5

[88] Matias Mendieta, Boran Han, Xingjian Shi, Yi Zhu, and Chen Chen. Towards geospatial foundation models via continual pretraining. 2023. 5

[89] Boran Han, Shuai Zhang, Xingjian Shi, and Markus Reichstein. Bridging remote sensors with multisensor geospatial foundation models. 2024. 5

[90] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. arXiv preprint arXiv:2207.08051, 2022. 5

[91] Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwar, Salman Khan, and Fahad Shahbaz Khan. Rethinking transformers pre-training for multi-spectral satellite imagery, 2024. 5

[92] Maofeng Tang, Andrei Cozma, Konstantinos Georgiou, and Hairong Qi. Cross-scale mae: A tale of multiscale exploitation in remote sensing. *Advances in Neural Information Processing Systems*, 36, 2024. 5

[93] Jeremy Irvin, Lucas Tao, Joanne Zhou, Yuntao Ma, Langston Nashold, Benjamin Liu, and Andrew Y. Ng. Usat: A unified self-supervised encoder for multi-sensor satellite imagery, 2023. 5

[94] Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang. Advancing plain vision transformer towards remote sensing foundation model. *IEEE Transactions on Geoscience and Remote Sensing*, pages 1–1, 2022. 5

[95] Yi Wang, Hugo Hernández Hernández, Conrad M Albrecht, and Xiao Xiang Zhu. Feature guided masked autoencoder for self-supervised learning in remote sensing. arXiv preprint arXiv:2310.18653, 2023. 5

[96] Caleb S. Spradlin, Jordan A. Caraballo-Vega, Jian Li, Mark L. Carroll, Jie Gong, and Paul M. Montesano. Satvision-toa: A geospatial foundation model for coarse-resolution all-sky remote sensing imagery, 2024. 5

[97] Yingchao Feng, Peijin Wang, Wenhui Diao, Qibin He, Huiyang Hu, Hanbo Bi, Xian Sun, and Kun Fu. A self-supervised cross-modal
remote sensing foundation model with multi-domain representation and cross-domain fusion. In *IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium*, pages 2239–2242, 2023. 5

[98] Xuyang Li, Danfeng Hong, and Jocelyn Chanussot. S2mae: A spatial-spectral pretraining foundation model for spectral remote sensing data. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 24088–24097, 2024. 5

[99] Guillaume Astruc, Nicolas Gonthier, Clement Mallet, and Loic Landrieu. Omnisat: Self-supervised modality fusion for earth observation, 2024. 5

[100] Danfeng Hong, Bing Zhang, Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, Hao Li, Pedram Ghamisi, Xiuping Jia, Antonio Plaza, Paolo Gamba, Jon Atli Benediktsson, and Jocelyn Chanussot. Spectralgpt: Spectral remote sensing foundation model. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, page 1–18, 2024. 5, 9, 10

[101] Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu. Neural plasticity-inspired foundation model for observing the earth crossing modalities, 2024. 5, 9

[102] Vishal Nedungadi, Ankit Kariryaa, Stefan Oehmcke, Serge Belongie, Christian Igel, and Nico Lang. Mmearth: Exploring multi-modal pretext tasks for geospatial representation learning. *arXiv preprint arXiv:2405.02771*, 2024. 5

[103] Gabriel Tseng, Ruben Cartuyvels, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner. Lightweight, pre-trained transformers for remote sensing timeseries, 2024. 5

[104] Michail Tarasiou and Stefanos Zafeiriou. Embedding earth: Self-supervised contrastive pre-training for dense land cover classification, 2022. 5

[105] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-aware self-supervised learning. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 10181–10190, 2021. 5

[106] Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 9414–9423, 2021. 5

[107] Utkarsh Mall, Bharath Hariharan, and Kavita Bala. Change-aware sampling and contrastive learning for satellite images. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 5261–5270, June 2023. 5

[108] Peri Akiva, Matthew Purri, and Matthew Leotta. Self-supervised material and texture representation learning for remote sensing tasks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8203–8215, June 2022. 5

[109] Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, Huimei He, Jian Wang, Jingdong Chen, Ming Yang, Yongjun Zhang, and Yansheng Li. Sky sense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery, 2024. 5

[110] Yingying Zhang, Lixiang Ru, Kang Wu, Lei Yu, Lei Liang, Yansheng Li, and Jingdong Chen. Sky sense v2: A unified foundation model for multi-modal remote sensing. *arXiv preprint arXiv:2507.13812*, 2025. 5

[111] Kang Wu, Yingying Zhang, Lixiang Ru, Bo Dang, Jiangwei Lao, Lei Yu, Junwei Luo, Zifan Zhu, Yue Sun, Jiahao Zhang, et al. A semantic-enhanced multi-modal remote sensing foundation model for earth observation. *Nature Machine Intelligence*, pages 1–15, 2025. 5

[112] Anthony Fuller, Koreen Millard, and James R Green. Transfer learning with pretrained remote sensing transformers. *arXiv preprint arXiv:2209.14969*, 2022. 5, 6

[113] Ziyue Huang, Mingming Zhang, Yuan Gong, Qingjie Liu, and Yunhong Wang. Generic knowledge boosted pre-training for remote sensing images. *IEEE Transactions on Geoscience and Remote Sensing*, 2024. 5

[114] Jiayuan Tian, Jie Lei, Jiaqing Zhang, Weiying Xie, and Yunsong Li. Swimdiff: Scene-wide matching contrastive learning with diffusion constraint for remote sensing image. *IEEE Transactions on Geoscience and Remote Sensing*, 2024. 5

[115] Yi Wang, Conrad M Albrecht, and Xiao Xiang Zhu. Self-supervised vision transformers for joint sar-optical representation learning. In *IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium*, pages 139–142. IEEE, 2022. 5

[116] Valerio Marsocci and Nicolas Audebert. Cross-sensor self-supervised training and alignment for remote sensing, 2024. 5, 9

[117] Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, Kurt Keutzer, and Trevor Darrell. Self-supervised pretraining improves self-supervised pretraining. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 2584–2594, January 2022. 5

[118] Tong Zhang, Peng Gao, Hao Dong, Yin Zhuang, Guanqun Wang, Wei Zhang, and He Chen. Consecutive pre-training: A knowledge transfer learning strategy with relevant unlabeled data for remote sensing domain. *Remote Sensing*, 14(22):5675, 2022. 5

[119] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In *International conference on machine learning*, pages 12310–12320. PMLR, 2021. 5, 6

[120] Sijie Zhao, Hao Chen, Xueliang Zhang, Pengfeng Xiao, Lei Bai, and Wanli Ouyang. Rs-mamba for large remote sensing image dense prediction, 2024. 5, 6

[121] Keyan Chen, Bowen Chen, Chenyang Liu, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsmamba: Remote sensing image classification with state space model, 2024. 5, 6

[122] Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham, Chenying Li, Zhitong Xiong, and Xiao Xiang Zhu. Decur: decoupling common & unique representations for multimodal self-supervision, 2023. 6

[123] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. 6

[124] Xiang Li, Congcung Wen, Yuan Hu, Zhenghang Yuan, and Xiao Xiang Zhu. Vision-language models in remote sensing: Current progress and future trends, 2024. 6

[125] Utkarsh Mall, Cheng Perng Phoo, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, and Kavita Bala. Remote sensing vision-language foundation models without annotations via ground remote alignment, 2023. 6

[126] Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Urbangpt: Spatio-temporal large language models. In *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 5351–5362, 2024. 6

[127] Susen Guo, Qiming Zhang, Junyue Jiang, Mingxing Peng, Meixin Zhu, and Hao Frank Yang. Towards explainable traffic flow prediction with large language models. *Communications in Transportation Research*, 4:100150, 2024. 6

[128] Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David Lobell, and Stefano Ermon. Diffusionsat: A generative foundation model for satellite imagery, 2023. 6

[129] Datao Tang, Xiangyong Cao, Xingsong Hou, Zhongyuan Jiang, and Deyu Meng. Crs-diff: Controllable generative remote sensing foundation model, 2024. 6

[130] Zhiping Yu, Chenyang Liu, Liqin Liu, Zhenwei Shi, and Zhengxia Zou. Metaearth: A generative foundation model for global-scale remote sensing image generation, 2024. 6

[131] Yuru Jia, Valerio Marsocci, Ziyang Gong, Xue Yang, Maarten Ver-gauwen, and Andrea Nascetti. Can generative geospatial diffusion models excel as discriminative geospatial foundation models?, 2025. 6

[132] Di Wang, Meiqi Hu, Yao Jin, Yuchun Miao, Jiaqi Yang, Yichu Xu, Xiaolei Qin, Jiaqi Ma, Lingyu Sun, Chenxing Li, Chuan Fu, Hongruixuan Chen, Chengxi Han, Naoto Yokoya, Jing Zhang, Minqiang Xu, Lin Liu, Lefei Zhang, Chen Wu, Bo Du, Dacheng Tao, and Liangpei Zhang. Hypersigma: Hyperspectral intelligence comprehension foundation model. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2025. 6

[133] Nassim Ait Ali Braham, Conrad M Albrecht, Julien Mairal, Jocelyn Chanussot, Yi Wang, and Xiao Xiang Zhu. Spectralearth: Training hyperspectral foundation models at scale. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 2025. 6

[134] Jingtao Li, Yingyi Liu, Xinyu Wang, Yunning Peng, Chen Sun, Shaoyu Wang, Zhendong Sun, Tian Ke, Xiao Jiang, Tangwei Lu, et al. Hyperfree: A channel-adaptive and tuning-free foundation model for hyperspectral remote sensing imagery. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 23048–23058, 2025. 6

[135] Matt Allen, Francisco Dorr, Joseph A. Gallego-Mejia, Laura Martínez-Ferrer, Anna Jungbluth, Freddie Kalaitzis, and Raúl Ramos-Pollán. Large scale masked autoencoding for reducing label requirements on sar data, 2023. 6

[136] Laura Martínez-Ferrer, Anna Jungbluth, Joseph A. Gallego-Mejia, Matt Allen, Francisco Dorr, Freddie Kalaitzis, and Raúl Ramos-Pollán. Exploring generalisability of self-distillation with no labels for sar-based vegetation prediction, 2023. 6
[137] Joseph A. Gallego-Mejia, Anna Jungbluth, Laura Martínez-Ferrer, Matt Allen, Francisco Dorr, Freddie Kalaitzis, and Raúl Ramos-Pollán. Exploring dino: Emergent properties and limitations for synthetic aperture radar imagery. 2023. 6

[138] Zhe Dong, Yanfeng Gu, and Tianzhu Liu. Upetu: A unified parameter-efficient fine-tuning framework for remote sensing foundation model. *IEEE Transactions on Geoscience and Remote Sensing*, 2024, 6

[139] Keyan Chen, Chengyang Liu, Wenyuan Li, Zili Liu, Hao Chen, Haotian Zhang, Zhengxia Zou, and Zhenwei Shi. Time travelling pixels: Bitemporal features integration with foundation model for remote sensing image change detection, 2023. 6

[140] Kaiyu Li, Xiangyong Cao, and Deyu Meng. A new learning paradigm for foundation model-based remote sensing change detection, 2024. 6

[141] Xin He, Yushi Chen, Lingbo Huang, Danfeng Hong, and Qian Du. Foundation model-based multimodal remote sensing data classification. *IEEE Transactions on Geoscience and Remote Sensing*, 62:1–17, 2024. 6

[142] Romain Thoreau, Valerio Marsocci, and Dawa Derksen. Parameter-efficient adaptation of geospatial foundation models through embedding deflection, 2025. 6

[143] Edoardo Arnaudo, Jacopo Lungo Vaschetti, Lorenzo Innocenti, Luca Barco, Davide Lisi, Vanina Fissore, and Claudio Rossi. Fmars: Annotating remote sensing images for disaster management using foundation models, 2024. 6

[144] Mingze Wang, Keyan Chen, Lili Su, Cilin Yan, Sheng Xu, Haotian Zhang, Pengcheng Yuan, Xiaolong Jiang, and Baochang Zhang. Rs-building: Towards general remote sensing image building extraction and change detection with foundation model, 2024. 6

[145] Xiao Wang, Aristeidis Tsaris, Siyan Liu, Jong-Youl Choi, Ming Fan, Wei Zhang, Junqi Yin, Moetasim Ashfaq, Dan Lu, and Prasanna Balaprakash. Orbit: Oak ridge base foundation model for earth system predictability, 2024. 6

[146] Aristeidis Tsaris, Philipe Ambrozio Dias, Abhishek Potnis, Junqi Yin, Feiyi Wang, and Dalton Lunga. Pretraining billion-scale geospatial foundational models on frontier, 2024. 6

[147] Yansheng Li, Jeyi Tan, Bo Dang, Mang Ye, Sergey A Bartalev, Stanislav Shinkarenko, Linlin Wang, Yingying Zhang, Lixiang Ru, Xin Guo, et al. Unleashing the potential of remote sensing foundation models via bridging data and computability islands. *The Innovation*, 6(6), 2025. 6

[148] Jeyi Tan, Chengwei Zhang, Bo Dang, and Yansheng Li. Towards privacy-preserved pre-training of remote sensing foundation models with federated mutual-guidance learning. *arXiv preprint arXiv:2503.11051*, 2025. 6

[149] Michael J. Smith, Luke Fleming, and James E. Geach. Earthpt: a time series foundation model for earth observation, 2024. 6

[150] Weijie L., Wei Yang, Yuenan Hou, Li Liu, Yongxiang Liu, and Xiang Li. SARATR-X: a foundation model for synthetic aperture radar images target recognition. *arXiv*, 2405.09365, 2024. 6

[151] Junyan Lin, Feng Gao, Xiaocheng Shi, Junyu Dong, and Qian Du. Ss-mae: Spatial-spectral masked auto-encoder for multi-source remote sensing image classification, 2023. 6

[152] Ziyang Gong, Zhixiang Wei, Di Wang, Xianzheng Ma, Hongruixuan Chen, Yuru Jia, Yupeng Deng, Zhenming Ji, Xiangwei Zhu, Naoto Yokoya, Jing Zhang, Bo Du, and Liangpei Zhang. Crossearth: Geospatial vision foundation model for domain generalizable remote sensing semantic segmentation, 2024. 6

[153] Jakob Hackstein, Gencer Sumbul, Kai Norman Clasen, and Begüm Demir. Exploring masked autoencoders for sensor-agnostic image retrieval in remote sensing, 2024. 6

[154] Nikolaos Ioannis Bountos, Arthur Ouaknine, and David Rolnick. Fomo-bench: a multi-modal, multi-scale and multi-task forest monitoring benchmark for remote sensing foundation models, 2024. 6

[155] Gerald Baier, Antonin Deschemps, Michael Schmitt, and Naoto Yokoya. Synthesizing optical and sar imagery from land cover maps and auxiliary raster data. *IEEE Transactions on Geoscience and Remote Sensing*, 60:1–12, 2021. 6

[156] Dominik Koßmann, Viktor Brack, and Thorsten Wilhelm. Seasonet: A seasonal scene classification, segmentation and retrieval dataset for satellite imagery overgermany. In *IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium*, pages 243–246. IEEE, 2022. 6

[157] Li Xiang, Ding Jian, and Elhoseiny Mohamed. Vrsbench: A versatile vision-language benchmark dataset for remote sensing image understanding. *arXiv:2406.12384*, 2024. 6

[158] Baichuan Zhou, Haote Yang, Dairong Chen, Junyan Ye, Tianyi Bai, Jinhua Yu, Songyang Zhang, Dahua Lin, Conghui He, and Weijia Li. Urbench: A comprehensive benchmark for evaluating large multimodal

models in multi-view urban scenarios. *arXiv preprint arXiv:2408.17267*, 2024. 6

[159] Zhang Chenhui and Wang Sherrie. Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data. *arXiv preprint arXiv:2401.17600*, 2024. 6

[160] Katerina Kikaki, Ioannis Kakogeorgiou, Ibrahim Hoteit, and Konstantinos Karantzalos. Detecting marine pollutants and sea surface features with deep learning in sentinel-2 imagery. *ISPRS Journal of Photogrammetry and Remote Sensing*, 210:39–54, 2024. 7, 8, 23

[161] Vivien Sainte Fare Garnot and Loic Landrieu. Panoptic segmentation of satellite image time series with convolutional temporal attention networks. *ICCV*, 2021. 7, 8

[162] Clément Rambour, Nicolas Audebert, E Koeniguer, Bertrand Le Saux, M Crucianu, and Mihai Datcu. Flood detection in time series of optical and sar images. *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 43(B2):1343–1346, 2020. 7, 8

[163] Xin-Yi Tong, Gui-Song Xia, and Xiao Xiang Zhu. Enabling country-scale land cover mapping with meter-resolution satellite imagery. *ISPRS Journal of Photogrammetry and Remote Sensing*, 196:178–196, 2023. 8

[164] Rose Rustowicz, Robin Cheong, Lijing Wang, Stefano Ermon, Marshall Burke, and David Lobell. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops*, June 2019. 8, 29

[165] Claudio Persello, Jeroen Grift, Xinyan Fan, Claudia Paris, Ronny Hänsch, Mila Koeva, and Andrew Nelson. Ai4smallfarms: A dataset for crop field delineation in southeast asian smallholder farms. *IEEE Geoscience and Remote Sensing Letters*, 20:1–5, 2023. 7, 8

[166] Panagiotis Agraflotis, Łukasz Janowski, Dimitrios Skarlatos, and Begüm Demir. Magicbathynet: A multimodal remote sensing dataset for bathymetry prediction and pixel-based classification in shallow waters, 2024. 7

[167] Derrick Bonafilia, Beth Tellman, Tyler Anderson, and Erica Issenberg. Sen1floods11: A georeferenced dataset to train and test deep learning flood algorithms for sentinel-1. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops*, June 2020. 7, 25

[168] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In *Proceedings of the European conference on computer vision (ECCV)*, pages 418–434, 2018. 9, 10

[169] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical image computing and computer-assisted intervention*, pages 234–241. Springer, 2015. 9, 10

[170] Matias Mendieta, Boran Han, Xingjian Shi, Yi Zhu, Chen Chen, and Mu Li. Gfm: Building geospatial foundation models via continual pretraining. *arXiv preprint arXiv:2302.04476*, 2023. 9

[171] Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing, 2024. 9

[172] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929*, 2020. 10

[173] Chia-Yu Hsu, Wenwen Li, and Sizhe Wang. Geospatial foundation models for image analysis: evaluating and enhancing nasa-ibm prithvi's domain adaptability. *International Journal of Geographical Information Science*, pages 1–30, 2024. 10

[174] Vivien Sainte Fare Garnot and Loic Landrieu. Lightweight temporal self-Attention for classifying satellite images time series. In *Workshop on Advanced Analytics and Learning on Temporal Data*, AALTD, 2020. 10

[175] Victor Durnov. xview2 1st place solution. 27

[176] Adam Van Etten, Daniel Hogan, Jesus Martinez Manso, Jacob Shermeyer, Nicholas Weir, and Ryan Lewis. The multi-temporal urban development spacenet dataset. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 6398–6407, 2021. 33

[177] Sebastian Hafner, Yifang Ban, and Andrea Nascetti. Urban change detection using a dual-task siamese network and semi-supervised learning. In *IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium*, pages 1071–1074. IEEE, 2022. 33
# APPENDIX

Below, we provide a detailed description of the benchmark datasets, along with the extensive experimental results including additional metrics.

## A. HLS Burn Scars

HLS Burn Scars [71] contains Harmonized Landsat and Sentinel-2 imagery of burn scars and the associated masks for 2018-2021 over the contiguous United States. There are 804 512x512 scenes. Scenes contain six bands (Blue - B02, Green - B03, Red - B04, NIR - B8A, SW 1 - B11, SW 2 - B12), and masks have one band (with values 1 = Burn scar, 0 = Not burned, -1 = Missing data). For satellite scenes, each band has already been converted to reflectance. The class distribution is 11% for Burn Scar, 88% for Not Burned, and 1% for No Data. The 804 files have been randomly split into training (2/3) and validation (1/3) directories, each containing the masks, scenes, and index files. To create the dataset, after co-locating the shapefile and HLS scene, the 512x512 chip was formed by taking a window with the burn scar in the center. Burn scars near the edges of HLS tiles are offset from the center. Images were manually filtered for cloud cover and missing data to provide as clean a scene as possible and burn scar presence was also manually verified. Some example images and labels are displayed in Fig. A1.

1) *Extensive Results:* Table A1 presents a comprehensive comparison of models on the HLS Burn Scars dataset using stratified sampling with 10%, 50%, and 100% labeled data.

The UNet Baseline achieves the best performance across all metrics for the 100% label setting, with the highest mIoU (84.51%), m-f1 (91.11%), and m-Accuracy (97.19%). This demonstrates the robustness of the UNet architecture when sufficient labeled data is available.

The Prithvi model achieves the second-best results in the 100% label setting, with mIoU (82.67%), m-f1 (89.90%), and m-Accuracy (96.55%). These results closely follow the UNet Baseline, indicating its competitiveness as an alternative model.

At 10% labels, SpectralGPT excels with the highest mIoU (83.35%) and m-f1 (90.67%), outperforming all other models. Its strong results in this low-label regime suggest robustness in low-supervision scenarios. However, its performance decreases at higher label proportions, which could indicate limitations in scalability or overfitting to specific data distributions.

Across all models, in average, performance improves significantly with increased labeled data, reflecting the importance of supervision for these tasks. The most substantial performance gains occur between 10% and 50% labels, indicating that moderate amounts of labeled data can significantly boost model effectiveness.

The ViT Baseline, added for comparison, demonstrates competitive performance at 100% labels with mIoU (81.58%), m-f1 (89.16%), and m-Accuracy (96.26%). While it does not surpass the UNet Baseline, its results highlight the potential of Vision Transformers for remote sensing tasks.

## B. MADOS

The Marine Debris and Oil Spill (MADOS) [160] dataset contains marine pollutants under a diverse set of locations

and sea surface features. The imagery contains multi-spectral (Bands 1-8A, 11 & 12) Sentinel-2 data from 174 scenes, with approximately 1.5M annotated pixels and a resolution of 10 to 60 meters, depending on the band. After processing, the dataset contains 2803 tiles, each 240x240 pixels in size. The authors define train, validation, and test splits based on location and date, with an approximate ratio of 50% / 25% / 25%.

Four major classes are assigned to different types of backgrounds: Marine Water (34.8%), Turbid Water (11.7%), Sediment-Laden Water (20.6%), and Shallow Water (9.2%). These and the Oil Spill (18.5%) category represent around 95% of all labeled pixels in the data, with the remaining 5% of labels distributed among the remaining 10 classes. The 15 classes have a sparse distribution over scenes, and examples of multiple classes in a tile are exceedingly rare.

Most pixels in the dataset are unlabeled, causing some batches of randomly selected data to have no valid labels and consequently an invalid loss value. To avoid this problem we employ focus cropping, which only selects crops with at least one valid label. Some examples are shown in Fig. A2

1) *Extensive Results:* Table A2 presents a more detailed breakdown of results on the MADOS dataset. The mean accuracy metric places more emphasis on the more abundant categories, and is therefore useful for identifying GFM s that do not capture the rare classes well.

CROMA achieves outstanding results in both m-f1 and mIoU scores while coming in 4th on accuracy, and our evaluation of the per-class results confirms that this is because of its great performance in the rarer classes. We conjecture that the large variety of bands used by the model, together with its modality- focused pre-training helps with differentiating between the high number of visually similar classes.

GFM-Swin performs exceptionally well in the three major water categories while performing adequately in the rarer classes. This results in a high mean accuracy score, but for practical labeling of the target classes RemoteCLIP and SpectralGPT are both better candidates, since they achieve high IoU scores without over-focusing on the background categories.

Despite using all Sentinel-2 bands, SSL4EO models perform among the worst on the mIoU score, while keeping up in the mean accuracy metric. Investigation of the per-class results shows that the models perform well on water categories while performing significantly worse on the much rarer classes of interest. This could mean that the EO-specific adaptations that other models employ are important for detecting the rare classes. Another possible explanation is that the self-supervised methods combined with the mainly land-focused SSL4EO-S12 dataset do not provide good enough features for marine environments.

The best GFM s comfortably outperform both supervised baselines, highlighting the importance of foundation models for sparsely labeled datasets such as MADOS. Despite this, the UNet model achieves a very respectable score that places it in the middle of the pack.

## C. PASTIS-R

PASTIS is a comprehensive benchmark dataset designed for panoptic and semantic segmentation of agricultural parcels
<figure><img src="image_7.png" /><figcaption>Fig. A1: Example images and labels of **HLS Burn Scars** (Location: USA. Domain: Wildfire). The images are visualized with B4, B3 and B2 bands.</figcaption></figure>

<table><caption>TABLE A1: Results for the HLS Burn Scars comparison using stratified sampling with 10%, 50%, and 100% labels for best and worst models from Table V</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">mIoU ↑</th><th colspan="3">m-F1 ↑</th><th colspan="3">m-prec ↑</th><th colspan="3">m-rec ↑</th><th colspan="3">m-acc ↑</th></tr><tr><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th></tr></thead><tbody><tr><td>CROMA</td><td>76.44</td><td>81.52</td><td>81.95</td><td>85.42</td><td>89.10</td><td>89.40</td><td>85.51</td><td>88.31</td><td>89.13</td><td>85.32</td><td>89.94</td><td>89.68</td><td>95.09</td><td>96.41</td><td>96.46</td></tr><tr><td>DOFA</td><td>71.98</td><td>78.02</td><td>78.96</td><td>81.77</td><td>86.60</td><td>87.27</td><td>79.45</td><td>86.67</td><td>85.84</td><td>84.63</td><td>86.54</td><td>88.86</td><td>94.33</td><td>95.49</td><td>95.89</td></tr><tr><td>GFM-Swin</td><td>67.23</td><td>74.36</td><td>76.17</td><td>78.02</td><td>83.79</td><td>85.24</td><td>83.30</td><td>83.57</td><td>86.92</td><td>74.61</td><td>84.01</td><td>83.76</td><td>91.28</td><td>94.59</td><td>94.81</td></tr><tr><td>Prithvi</td><td>77.73</td><td>80.89</td><td>82.67</td><td>86.34</td><td>88.66</td><td>89.90</td><td>83.71</td><td>87.44</td><td>90.54</td><td>89.55</td><td>89.97</td><td>89.29</td><td>95.75</td><td>96.31</td><td>96.55</td></tr><tr><td>RemoteCLIP</td><td>69.40</td><td>74.29</td><td>75.55</td><td>79.59</td><td>83.74</td><td>84.77</td><td>78.36</td><td>84.09</td><td>86.54</td><td>80.98</td><td>83.41</td><td>83.21</td><td>93.43</td><td>94.48</td><td>94.63</td></tr><tr><td>SatlasNet</td><td>74.79</td><td>75.97</td><td>79.69</td><td>84.08</td><td>85.01</td><td>87.80</td><td>81.78</td><td>83.13</td><td>86.49</td><td>86.85</td><td>87.19</td><td>89.25</td><td>95.02</td><td>95.24</td><td>96.05</td></tr><tr><td>Scale-MAE</td><td>75.47</td><td>75.47</td><td>76.71</td><td>84.66</td><td>84.66</td><td>85.64</td><td>84.16</td><td>84.16</td><td>86.96</td><td>85.17</td><td>85.17</td><td>84.44</td><td>94.92</td><td>94.92</td><td>95.00</td></tr><tr><td>SpectralGPT</td><td><b>83.35</b></td><td>76.40</td><td>80.47</td><td><b>90.67</b></td><td>85.50</td><td>88.36</td><td><b>94.49</b></td><td>87.87</td><td>87.31</td><td>88.00</td><td>83.51</td><td>89.49</td><td>93.54</td><td>94.33</td><td>96.20</td></tr><tr><td>S12-MoCo</td><td>73.11</td><td>79.79</td><td>80.76</td><td>82.85</td><td>87.90</td><td>88.58</td><td>84.66</td><td>88.68</td><td>88.59</td><td>81.27</td><td>87.16</td><td>88.56</td><td>93.92</td><td>95.85</td><td>96.16</td></tr><tr><td>S12-DINO</td><td>75.93</td><td>80.12</td><td>81.44</td><td>84.99</td><td>88.13</td><td>89.04</td><td>83.58</td><td>88.63</td><td>88.06</td><td>86.57</td><td>87.66</td><td>90.08</td><td>95.16</td><td>95.96</td><td>96.41</td></tr><tr><td>S12-MAE</td><td>76.60</td><td>80.13</td><td>80.86</td><td>85.49</td><td>88.14</td><td>88.66</td><td>83.47</td><td><b>88.88</b></td><td>89.88</td><td>87.86</td><td>87.44</td><td>87.53</td><td>95.41</td><td>95.94</td><td>96.07</td></tr><tr><td>S12-Data2Vec</td><td>74.38</td><td>79.82</td><td>81.14</td><td>83.78</td><td>87.91</td><td>88.83</td><td>82.78</td><td>87.28</td><td>87.73</td><td>84.86</td><td>88.56</td><td>90.02</td><td>94.71</td><td>96.00</td><td>96.35</td></tr><tr><td>UNet Baseline</td><td><b>79.46</b></td><td><b>82.39</b></td><td><b>84.51</b></td><td><b>87.60</b></td><td><b>89.68</b></td><td><b>91.11</b></td><td><b>83.86</b></td><td>86.61</td><td><b>88.66</b></td><td><b>92.57</b></td><td><b>93.49</b></td><td><b>94.00</b></td><td><b>96.25</b></td><td><b>96.80</b></td><td><b>97.19</b></td></tr><tr><td>ViT Baseline</td><td>75.92</td><td>78.17</td><td>81.58</td><td>84.98</td><td>86.70</td><td>89.16</td><td>83.32</td><td>86.12</td><td>90.20</td><td>86.87</td><td>87.30</td><td>88.19</td><td>95.20</td><td>95.60</td><td>96.26</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Fig. A2: Example images and labels from **MADOS**. (Location: Global. Domain: Marine.) The images are visualized with B4, B3, B2 bands. Black labels represent unlabeled pixels.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. A3: Example time series data of **PASTIS-R** (Location: France. Domain: Agriculture.). The Sentinel-1 images are visualized by taking VV, VH, VV-VH bands, and the Sentinel-2 images are visualized by taking B4, B3, and B2 bands.</figcaption></figure>
<table><caption>TABLE A2: Results for MADOS dataset, best results in **bold**, second best **underlined**.</caption><thead><tr><th>Model</th><th>mIoU ↑</th><th>m-F1 ↑</th><th>m-prec ↑</th><th>m-rec ↑</th><th>m-acc ↑</th></tr></thead><tbody><tr><td>CROMA</td><td><strong>67.55</strong></td><td><strong>77.19</strong></td><td>91.38</td><td>90.85</td><td>91.04</td></tr><tr><td>DOFA</td><td>59.58</td><td>68.19</td><td><strong>92.12</strong></td><td><strong>94.26</strong></td><td><strong>92.81</strong></td></tr><tr><td>GFM-Swin</td><td>64.71</td><td>74.23</td><td><strong>94.97</strong></td><td><strong>94.77</strong></td><td><strong>95.30</strong></td></tr><tr><td>Prithvi</td><td>49.98</td><td>60.26</td><td>83.10</td><td>83.58</td><td>83.92</td></tr><tr><td>RemoteCLIP</td><td>60.00</td><td>70.48</td><td>91.63</td><td>90.59</td><td>91.64</td></tr><tr><td>SatlasNet</td><td>55.86</td><td>67.03</td><td>86.92</td><td>85.68</td><td>87.56</td></tr><tr><td>Scale-MAE</td><td>57.32</td><td>68.50</td><td>91.28</td><td>91.03</td><td>90.92</td></tr><tr><td>SpectralGPT</td><td>57.99</td><td>70.18</td><td>84.95</td><td>84.97</td><td>85.27</td></tr><tr><td>S12-Data2Vec</td><td>44.36</td><td>55.18</td><td>80.18</td><td>80.58</td><td>82.41</td></tr><tr><td>S12-DINO</td><td>49.38</td><td>60.65</td><td>83.25</td><td>82.93</td><td>83.66</td></tr><tr><td>S12-MAE</td><td>49.90</td><td>60.79</td><td>84.29</td><td>83.79</td><td>84.70</td></tr><tr><td>S12-MoCo</td><td>51.76</td><td>63.02</td><td>84.88</td><td>84.64</td><td>85.21</td></tr><tr><td>UNet Baseline</td><td>54.79</td><td>65.97</td><td>85.20</td><td>84.67</td><td>85.61</td></tr><tr><td>ViT Baseline</td><td>48.19</td><td>58.78</td><td>58.14</td><td>62.59</td><td>88.00</td></tr></tbody></table>

using satellite time series. It includes 2,433 image patches captured across the French metropolitan area, each with detailed panoptic annotations, providing both instance indices and semantic labels for every pixel. The original dataset is built using Sentinel-2 multi-spectral imagery, offering time series data with variable lengths for each patch. The Sentinel-2 available bands are B2, B3, B4, B5, B6, B7, B8, B8A, B11 and B12. To test the performance of the model foundations on multi-modal settings, we used the PASTIS-R version of the dataset. This extended dataset introduces aligned Sentinel-1 radar observations. This extension includes approximately 70 radar observations from both ascending and descending orbits for each of the 2,433 patches, complementing the optical data from Sentinel-2. In line with the multi-temporal setup of this paper, we have kept 6 captures evenly distributed over time for each satellite time series.

1) **Extensive Results:** In the next paragraph, we present the results obtained on the PASTIS-R dataset (Table A3), before considering the results obtained by reducing the size of the dataset, i.e. by reducing the number of samples seen during training (Table A4 and Table A5). Initially, we observe that two models stand out and achieve the best performance on the PASTIS-R dataset: CROMA on the optical modality and S12-DINO. Indeed, for all the metrics used in this benchmark, these two models produce the best results. The S12-DINO model is the strongest model when the temporal fusion strategy is linear, with a mIoU of 32.45. For the L-TAE strategy, the CROMA optical model performs best, with a mIoU of 36.18.

In general, L-TAE fusion produces better results than linear fusion, e.g. +6.46% mIoU points for CROMA, +7.38% mIoU points for s12-Data2Vec and +8.30% mIoU points for s12-MoCo.

Interestingly, we can see that in the case of our setup and the PASTIS-R dataset, multi-modality doesn't seem to bring any particular gain: the performances obtained by CROMA joint, i.e. on the optical and SAR modalities are worse than those obtained only on the optical modality (32.32% vs 38.51% mIoU). This underperformance seems to be partly explained by the poor results obtained by the various models solely on SAR images: 21.61% for CROMA + L-TAE and 14.20% for S12-MAE.

By decreasing the proportion of the dataset seen during

training (respectively Table A4 and Table A5 for linear multi-temporal fusion and L-TAE) we observe that performance varies little, e.g. 34.32% mIoU for s12-Data2Vec on the full dataset, 33.42% mIoU on 50% of the dataset and 33.09 on only 10% (for L-TAE strategy). Nevertheless, a performance drop of 2.78 mIoU points (24.74 mIoU on 10% and 27.53 mIoU on 100%) is observed for SpectralGPT.

## D. Sen1Floods11

Sen1Floods11[167] offers global flood mapping, featuring two bands (VV and VH) from Sentinel-1, all spectral bands from Sentinel-2, and manually labeled surface water, which includes both permanent and flood water. The dataset comprises 4,831 chips, each with dimensions of 512 x 512 pixels at a 10-meter resolution, encompassing 11 distinct flood events and covering a total area of 120,406 square kilometers. The hand-labeled data is split into training, validation, and testing data using a random 60-20-20 split. The cloudy pixels are labeled and not considered during the experiments. An example sample is shown in Figure A4.

1) **Extensive Results:** In the following, we present the comprehensive experimental results for the Sen1Floods11 flood mapping task. From Table A6, we observe that, as a relatively simple task with only two classes, the fully supervised UNet outperforms all other models. The benefits of RemoteCLIP are less pronounced in this limited-class setting, and Scale-MAE demonstrates suboptimal performance with the 10-meter resolution data.

Table A6 illustrates the impact of label reduction. Even when labels are reduced by 50% and 10%, the fully supervised UNet continues to deliver superior performance. Additionally, both CROMA and Prithvi consistently achieve results comparable to those of the fully supervised setting. Given the simplicity of this task, the benefit of using pre-trained models is not as evident when compared to a lightweight fully supervised approach.

## E. xView2

The xBD dataset [57] was used in the xView2 challenge and is thus often referred to as the xView2 dataset. It is a high-resolution, bi-temporal dataset focusing on detecting building
<table><caption>TABLE A3: Results for PASTIS-R dataset with 100% of the data, best results in **bold**, second best **underlined**.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="2">mIoU ↑</th><th colspan="2">m-F1 ↑</th><th colspan="2">m-prec ↑</th><th colspan="2">m-rec ↑</th><th colspan="2">m-acc ↑</th></tr><tr><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th><th>Linear</th><th>L-TAE</th></tr></thead><tbody><tr><td>CROMA</td><td>joint</td><td>27.75</td><td>32.32</td><td>39.91</td><td>45.27</td><td>36.79</td><td>44.09</td><td>49.28</td><td>51.33</td><td>71.93</td><td>74.47</td></tr><tr><td>CROMA</td><td>optical</td><td>32.06</td><td><u>38.51</u></td><td>44.92</td><td><u>52.53</u></td><td>41.97</td><td><u>48.76</u></td><td>52.95</td><td><strong>62.66</strong></td><td><u>74.58</u></td><td><u>77.76</u></td></tr><tr><td>CROMA</td><td>sar</td><td>20.68</td><td>21.61</td><td>30.12</td><td>31.93</td><td>28.82</td><td>32.22</td><td>34.29</td><td>39.43</td><td>68.72</td><td>66.91</td></tr><tr><td>DOFA</td><td>joint</td><td>24.49</td><td>30.02</td><td>35.88</td><td>42.37</td><td>33.28</td><td>39.45</td><td>43.64</td><td>50.54</td><td>70.29</td><td>74.09</td></tr><tr><td>GFM-Swin</td><td>optical</td><td>15.13</td><td>21.24</td><td>23.45</td><td>32.02</td><td>21.71</td><td>28.71</td><td>30.18</td><td>45.36</td><td>62.11</td><td>67.39</td></tr><tr><td>Prithvi</td><td>optical</td><td>27.40</td><td>33.93</td><td>39.56</td><td>46.95</td><td>36.69</td><td>43.95</td><td>47.22</td><td>56.46</td><td>71.60</td><td>75.75</td></tr><tr><td>RemoteCLIP</td><td>optical</td><td>13.31</td><td>18.23</td><td>20.83</td><td>27.57</td><td>21.34</td><td>26.57</td><td>23.17</td><td>32.46</td><td>58.39</td><td>65.80</td></tr><tr><td>SatlasNet</td><td>optical</td><td>17.51</td><td></td><td>27.14</td><td></td><td>23.97</td><td></td><td>38.03</td><td></td><td>63.04</td><td></td></tr><tr><td>Scale-MAE</td><td>optical</td><td>11.74</td><td>24.55</td><td>17.45</td><td>35.75</td><td>17.34</td><td>33.64</td><td>28.00</td><td>43.69</td><td>61.91</td><td>71.19</td></tr><tr><td>SpectralGPT</td><td>optical</td><td>27.52</td><td>35.44</td><td>40.05</td><td>49.19</td><td>38.22</td><td>46.60</td><td>46.59</td><td>56.67</td><td>72.84</td><td>76.11</td></tr><tr><td>S12-MoCo</td><td>optical</td><td>26.09</td><td>34.49</td><td>38.32</td><td>48.07</td><td>35.08</td><td>44.97</td><td>47.05</td><td>57.21</td><td>69.98</td><td>75.43</td></tr><tr><td>S12-DINO</td><td>optical</td><td><u>32.45</u></td><td>36.18</td><td><u>45.86</u></td><td>49.89</td><td><u>42.34</u></td><td>46.17</td><td><u>54.95</u></td><td>60.88</td><td>73.60</td><td>76.05</td></tr><tr><td>S12-MAE</td><td>optical</td><td>27.06</td><td>32.03</td><td>39.22</td><td>45.23</td><td>35.32</td><td>41.09</td><td>50.61</td><td>56.21</td><td>69.62</td><td>74.09</td></tr><tr><td>S12-MAE</td><td>sar</td><td>18.81</td><td>14.20</td><td>28.08</td><td>21.71</td><td>26.41</td><td>21.45</td><td>18.81</td><td>34.13</td><td>66.40</td><td>58.48</td></tr><tr><td>S12-Data2Vec</td><td>optical</td><td>26.94</td><td>34.32</td><td>39.22</td><td>47.61</td><td>36.34</td><td>44.89</td><td>47.01</td><td>59.39</td><td>70.56</td><td>75.28</td></tr><tr><td>UNet Baseline</td><td>optical</td><td></td><td>31.60</td><td></td><td>43.52</td><td></td><td>39.37</td><td></td><td>51.57</td><td></td><td>72.12</td></tr><tr><td>ViT Baseline</td><td>optical</td><td></td><td><strong>36.57</strong></td><td><strong>38.53</strong></td><td><strong>50.03</strong></td><td><strong>54.89</strong></td><td><strong>46.38</strong></td><td><strong>51.53</strong></td><td><strong>59.46</strong></td><td><u>62.61</u></td><td><strong>75.96</strong></td><td><strong>78.34</strong></td></tr></tbody></table>

<table><caption>TABLE A4: Results for PASTIS-R dataset with linear time strategy, best results in **bold**, second best **underlined**.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="3">mIoU ↑</th><th colspan="3">m-F1 ↑</th><th colspan="3">m-prec ↑</th><th colspan="3">m-rec ↑</th><th colspan="3">m-acc ↑</th></tr><tr><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th></tr></thead><tbody><tr><td>CROMA</td><td>joint</td><td>27.56</td><td>27.59</td><td>27.75</td><td>39.61</td><td>39.74</td><td>39.91</td><td>36.55</td><td>36.80</td><td>36.79</td><td>48.64</td><td>48.80</td><td>49.28</td><td>71.76</td><td>71.63</td><td>71.93</td></tr><tr><td>CROMA</td><td>optical</td><td><u>31.93</u></td><td><u>31.41</u></td><td>32.06</td><td><u>44.80</u></td><td><u>44.01</u></td><td>44.92</td><td>41.64</td><td>40.26</td><td>41.97</td><td>53.73</td><td><u>53.83</u></td><td>52.95</td><td><u>74.22</u></td><td>74.09</td><td>74.58</td></tr><tr><td>CROMA</td><td>sar</td><td>19.47</td><td>19.63</td><td>20.68</td><td>28.88</td><td>29.02</td><td>30.12</td><td>27.24</td><td>27.83</td><td>28.82</td><td>33.55</td><td>32.93</td><td>34.29</td><td>67.35</td><td>67.66</td><td>68.72</td></tr><tr><td>DOFA</td><td>joint</td><td>23.72</td><td>24.17</td><td>24.49</td><td>34.99</td><td>35.45</td><td>35.88</td><td>31.83</td><td>33.31</td><td>33.28</td><td>43.22</td><td>43.08</td><td>43.64</td><td>69.31</td><td>69.78</td><td>70.29</td></tr><tr><td>GFM-Swin</td><td>optical</td><td>15.58</td><td>15.79</td><td>15.13</td><td>24.13</td><td>24.72</td><td>23.45</td><td>23.63</td><td>23.58</td><td>21.71</td><td>28.78</td><td>28.46</td><td>30.18</td><td>60.69</td><td>60.36</td><td>62.11</td></tr><tr><td>Prithvi</td><td>optical</td><td>26.33</td><td>26.92</td><td>27.40</td><td>38.23</td><td>39.05</td><td>39.56</td><td>35.76</td><td>36.54</td><td>36.69</td><td>45.02</td><td>46.01</td><td>47.22</td><td>70.63</td><td>70.75</td><td>71.60</td></tr><tr><td>RemoteCLIP</td><td>optical</td><td>12.25</td><td>12.91</td><td>13.31</td><td>19.08</td><td>20.20</td><td>20.83</td><td>18.73</td><td>20.33</td><td>21.34</td><td>21.84</td><td>21.75</td><td>23.17</td><td>58.35</td><td>57.69</td><td>58.39</td></tr><tr><td>SatlasNet</td><td>optical</td><td>16.76</td><td>16.78</td><td>17.51</td><td>26.11</td><td>26.00</td><td>26.98</td><td>23.09</td><td>23.54</td><td>23.99</td><td>34.82</td><td>37.40</td><td>37.12</td><td>61.58</td><td>61.86</td><td>62.98</td></tr><tr><td>Scale-MAE</td><td>optical</td><td>10.40</td><td>10.45</td><td>11.74</td><td>15.57</td><td>15.87</td><td>17.45</td><td>15.19</td><td>15.56</td><td>17.34</td><td>28.12</td><td>29.97</td><td>28.00</td><td>59.73</td><td>58.61</td><td>61.91</td></tr><tr><td>SpectralGPT</td><td>optical</td><td>24.74</td><td>25.74</td><td>27.52</td><td>36.59</td><td>37.83</td><td>40.05</td><td>34.29</td><td>35.69</td><td>38.22</td><td>43.69</td><td>44.54</td><td>46.59</td><td>70.96</td><td>71.47</td><td>72.84</td></tr><tr><td>S12-MoCo</td><td>optical</td><td>24.29</td><td>24.79</td><td>26.09</td><td>35.97</td><td>36.67</td><td>38.32</td><td>33.06</td><td>33.09</td><td>35.08</td><td>44.53</td><td>46.53</td><td>47.05</td><td>68.97</td><td>69.19</td><td>69.98</td></tr><tr><td>S12-DINO</td><td>optical</td><td>30.84</td><td>30.82</td><td><u>32.45</u></td><td>43.93</td><td>43.84</td><td><u>45.86</u></td><td>40.82</td><td><u>40.44</u></td><td><u>42.34</u></td><td><u>53.97</u></td><td>52.95</td><td><u>54.95</u></td><td>72.43</td><td>72.56</td><td>73.60</td></tr><tr><td>S12-MAE</td><td>optical</td><td>25.31</td><td>26.40</td><td>27.06</td><td>36.68</td><td>38.40</td><td>39.22</td><td>33.48</td><td>35.08</td><td>35.32</td><td>45.94</td><td>47.44</td><td>50.61</td><td>68.48</td><td>68.27</td><td>69.62</td></tr><tr><td>S12-MAE</td><td>sar</td><td>17.29</td><td>17.54</td><td>18.81</td><td>25.86</td><td>26.04</td><td>28.08</td><td>24.51</td><td>24.75</td><td>26.41</td><td>29.58</td><td>30.20</td><td>18.81</td><td>65.23</td><td>66.12</td><td>66.40</td></tr><tr><td>S12-Data2Vec</td><td>optical</td><td>24.65</td><td>24.52</td><td>26.94</td><td>36.11</td><td>35.82</td><td>39.22</td><td>33.28</td><td>32.38</td><td>36.34</td><td>44.15</td><td>45.97</td><td>47.01</td><td>68.72</td><td>68.85</td><td>70.56</td></tr><tr><td>UNet Baseline</td><td>optical</td><td>29.52</td><td>30.24</td><td>31.60</td><td>41.19</td><td>41.87</td><td>43.52</td><td>39.70</td><td>40.40</td><td>39.37</td><td>52.27</td><td>51.53</td><td>51.57</td><td>74.57</td><td>75.11</td><td>72.12</td></tr><tr><td>ViT Baseline</td><td>optical</td><td></td><td><strong>36.38</strong></td><td><strong>36.13</strong></td><td><strong>36.57</strong></td><td><strong>50.07</strong></td><td><strong>49.74</strong></td><td><strong>50.03</strong></td><td><strong>46.16</strong></td><td><strong>46.21</strong></td><td><strong>46.38</strong></td><td><strong>59.32</strong></td><td><strong>59.43</strong></td><td><strong>59.46</strong></td><td><strong>75.37</strong></td><td><strong>75.36</strong></td><td><strong>75.96</strong></td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>Fig. A4: Example data of Sen1Floods11 (Location: Global. Domain: Flood.). The Sentinel1 images are visualized by taking the VV, VH, VV bands, and the Sentinel-2 images are visualized by taking the B4, B3, and B2 bands. In the semantic labels, red represents water, green indicates areas without water, and blue marks cloudy pixels that are ignored.</figcaption></figure>
<table><caption>TABLE A5: Results for PASTIS-R dataset with L-TAE time strategy, best results in **bold**, second best **underlined**.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Modality</th><th colspan="3">mIoU ↑</th><th colspan="3">m-F1 ↑</th><th colspan="3">m-prec ↑</th><th colspan="3">m-rec ↑</th><th colspan="3">m-acc ↑</th></tr><tr><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th></tr></thead><tbody><tr><td>CROMA</td><td>joint</td><td>32.80</td><td>32.33</td><td>32.32</td><td>45.73</td><td>45.12</td><td>45.27</td><td>42.59</td><td>42.21</td><td>44.09</td><td>54.42</td><td>53.12</td><td>51.33</td><td>75.37</td><td>74.89</td><td>74.47</td></tr><tr><td>CROMA</td><td>optical</td><td><u>37.24</u></td><td><u>37.76</u></td><td><u>38.51</u></td><td><u>50.94</u></td><td><u>51.52</u></td><td><u>52.53</u></td><td>46.76</td><td><u>48.49</u></td><td><u>48.76</u></td><td><b><u>62.19</u></b></td><td><b><u>61.78</u></b></td><td><b><u>62.66</u></b></td><td><b><u>77.17</u></b></td><td><u>77.47</u></td><td><u>77.76</u></td></tr><tr><td>CROMA</td><td>sar</td><td>21.62</td><td>21.31</td><td>21.61</td><td>31.42</td><td>31.21</td><td>31.93</td><td>30.78</td><td>29.97</td><td>32.22</td><td>36.80</td><td>35.75</td><td>39.43</td><td>68.31</td><td>67.01</td><td>66.91</td></tr><tr><td>DOFA</td><td>joint</td><td>27.68</td><td>28.60</td><td>30.02</td><td>39.32</td><td>40.43</td><td>42.37</td><td>36.16</td><td>37.52</td><td>39.45</td><td>49.52</td><td>50.16</td><td>50.54</td><td>73.13</td><td>73.69</td><td>74.09</td></tr><tr><td>GFM-Swin</td><td>optical</td><td>21.47</td><td>20.41</td><td>21.24</td><td>32.01</td><td>30.85</td><td>32.02</td><td>28.59</td><td>27.66</td><td>28.71</td><td>42.51</td><td>43.76</td><td>45.36</td><td>65.33</td><td>67.98</td><td>67.39</td></tr><tr><td>Prithvi</td><td>optical</td><td>33.56</td><td>33.13</td><td>33.93</td><td>46.96</td><td>46.01</td><td>46.95</td><td>43.55</td><td>43.23</td><td>43.95</td><td>55.05</td><td>54.19</td><td>56.46</td><td>74.58</td><td>75.51</td><td>75.75</td></tr><tr><td>RemoteCLIP</td><td>optical</td><td>17.19</td><td>17.46</td><td>18.23</td><td>26.52</td><td>26.66</td><td>27.57</td><td>25.39</td><td>26.15</td><td>26.57</td><td>30.01</td><td>32.88</td><td>32.46</td><td>63.14</td><td>64.23</td><td>65.80</td></tr><tr><td>SatlasNet</td><td>optical</td><td>16.76</td><td>16.78</td><td>17.51</td><td>26.11</td><td>26.00</td><td>26.98</td><td>23.09</td><td>23.54</td><td>23.99</td><td>34.82</td><td>37.40</td><td>37.12</td><td>61.58</td><td>61.86</td><td>62.98</td></tr><tr><td>Scale-MAE</td><td>optical</td><td>22.86</td><td>23.26</td><td>24.55</td><td>33.44</td><td>34.56</td><td>35.75</td><td>33.25</td><td>23.02</td><td>33.64</td><td>40.88</td><td>38.32</td><td>43.69</td><td>68.75</td><td>68.45</td><td>71.19</td></tr><tr><td>SpectralGPT</td><td>optical</td><td>34.53</td><td>34.61</td><td>35.44</td><td>48.16</td><td>48.25</td><td>49.19</td><td>45.59</td><td>45.96</td><td>46.60</td><td>54.33</td><td>54.46</td><td>56.67</td><td>75.95</td><td>75.64</td><td>76.11</td></tr><tr><td>S12-MoCo</td><td>optical</td><td>32.51</td><td>32.59</td><td>34.49</td><td>45.66</td><td>45.84</td><td>48.07</td><td>42.38</td><td>42.91</td><td>44.97</td><td>55.89</td><td>57.71</td><td>57.21</td><td>74.28</td><td>74.02</td><td>75.43</td></tr><tr><td>S12-DINO</td><td>optical</td><td>36.62</td><td>35.71</td><td>36.18</td><td>50.68</td><td>49.48</td><td>49.89</td><td>47.47</td><td>45.48</td><td>46.17</td><td><u>60.87</u></td><td><u>60.33</u></td><td><u>60.88</u></td><td>75.76</td><td>75.28</td><td>76.05</td></tr><tr><td>S12-MAE</td><td>optical</td><td>31.06</td><td>31.15</td><td>32.03</td><td>44.05</td><td>44.15</td><td>45.23</td><td>40.74</td><td>39.38</td><td>41.09</td><td>54.39</td><td>56.17</td><td>56.21</td><td>73.47</td><td>73.38</td><td>74.09</td></tr><tr><td>S12-MAE</td><td>sar</td><td>13.00</td><td>14.33</td><td>14.20</td><td>19.58</td><td>21.67</td><td>21.71</td><td>20.34</td><td>22.13</td><td>21.45</td><td>24.20</td><td>31.92</td><td>34.13</td><td>60.34</td><td>59.07</td><td>58.48</td></tr><tr><td>S12-Data2Vec</td><td>optical</td><td>33.09</td><td>33.42</td><td>34.32</td><td>45.98</td><td>46.63</td><td>47.61</td><td>43.46</td><td>43.60</td><td>44.89</td><td>56.51</td><td>56.14</td><td>59.39</td><td>74.58</td><td>74.31</td><td>75.28</td></tr><tr><td>UNet Baseline</td><td>optical</td><td>29.52</td><td>30.24</td><td>31.60</td><td>41.19</td><td>41.87</td><td>43.52</td><td>39.70</td><td>40.40</td><td>39.37</td><td>52.27</td><td>51.53</td><td>51.57</td><td>74.57</td><td>75.11</td><td>72.12</td></tr><tr><td>ViT Baseline</td><td>optical</td><td><b><u>38.44</u></b></td><td><b><u>38.77</u></b></td><td><b><u>38.53</u></b></td><td><b><u>52.49</u></b></td><td><b><u>52.59</u></b></td><td><b><u>54.89</u></b></td><td><b><u>49.51</u></b></td><td><b><u>49.25</u></b></td><td><b><u>51.53</u></b></td><td>60.35</td><td><u>61.27</u></td><td><u>62.61</u></td><td><u>76.93</u></td><td><b><u>77.71</u></b></td><td><b><u>78.34</u></b></td></tr></tbody></table>

<table><caption>TABLE A6: Sen1Floods11 results using different limited training labels with best results in **bold**, second best **underlined**.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">mIoU ↑</th><th colspan="3">m-F1 ↑</th><th colspan="3">water IoU ↑</th><th colspan="3">water F1 ↑</th></tr><tr><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th><th>10%</th><th>50%</th><th>100%</th></tr></thead><tbody><tr><td>CROMA</td><td><u>87.22</u></td><td><u>90.57</u></td><td><u>90.89</u></td><td><u>92.90</u></td><td><u>94.91</u></td><td><u>95.09</u></td><td><u>77.71</u></td><td><u>83.59</u></td><td><u>84.15</u></td><td><u>87.46</u></td><td><u>91.06</u></td><td><u>91.39</u></td></tr><tr><td>DOFA</td><td>82.84</td><td>88.39</td><td>89.37</td><td>90.13</td><td>93.62</td><td>94.20</td><td>70.70</td><td>79.84</td><td>81.54</td><td>82.83</td><td>88.79</td><td>89.83</td></tr><tr><td>GFM-Swin</td><td>62.57</td><td>71.61</td><td>72.60</td><td>73.11</td><td>81.70</td><td>82.51</td><td>34.21</td><td>50.68</td><td>52.36</td><td>50.98</td><td>67.26</td><td>68.73</td></tr><tr><td>Prithvi</td><td>86.28</td><td>89.69</td><td>90.37</td><td>92.32</td><td>94.40</td><td>94.80</td><td>76.21</td><td>82.11</td><td>83.26</td><td>86.50</td><td>90.18</td><td>90.86</td></tr><tr><td>RemoteCLIP</td><td>62.22</td><td>71.66</td><td>74.26</td><td>72.77</td><td>81.76</td><td>83.83</td><td>33.67</td><td>50.85</td><td>55.18</td><td>50.38</td><td>67.42</td><td>71.11</td></tr><tr><td>SatlasNet</td><td>83.92</td><td>89.45</td><td>90.30</td><td>90.82</td><td>94.25</td><td>87.51</td><td>72.30</td><td>81.65</td><td>83.12</td><td>83.92</td><td>89.90</td><td>90.79</td></tr><tr><td>Scale-MAE</td><td>64.74</td><td>72.54</td><td>74.13</td><td>75.51</td><td>82.43</td><td>83.75</td><td>38.81</td><td>52.09</td><td>55.04</td><td>55.92</td><td>68.50</td><td>71.00</td></tr><tr><td>SpectralGPT</td><td>83.12</td><td>87.52</td><td>89.07</td><td>90.31</td><td>93.09</td><td>94.03</td><td>71.11</td><td>78.29</td><td>81.02</td><td>83.12</td><td>87.82</td><td>89.52</td></tr><tr><td>S12-MoCo</td><td>79.58</td><td>89.22</td><td>89.26</td><td>87.92</td><td>94.12</td><td>94.14</td><td>65.24</td><td>81.29</td><td>81.37</td><td>78.96</td><td>89.68</td><td>89.73</td></tr><tr><td>S12-DINO</td><td>84.95</td><td>88.93</td><td>88.61</td><td>91.48</td><td>93.94</td><td>93.75</td><td>73.96</td><td>80.76</td><td>80.25</td><td>85.03</td><td>89.35</td><td>89.04</td></tr><tr><td>S12-MAE</td><td>84.81</td><td>88.43</td><td>87.79</td><td>91.39</td><td>93.64</td><td>93.25</td><td>73.78</td><td>79.93</td><td>78.79</td><td>84.91</td><td>88.85</td><td>88.13</td></tr><tr><td>S12-Data2Vec</td><td>81.91</td><td>86.58</td><td>88.15</td><td>89.48</td><td>92.51</td><td>93.47</td><td>68.80</td><td>76.71</td><td>79.44</td><td>81.52</td><td>86.82</td><td>88.54</td></tr><tr><td>UNet Baseline</td><td><b>88.55</b></td><td><b>90.91</b></td><td><b>91.42</b></td><td><b>93.72</b></td><td><b>95.11</b></td><td><b>95.40</b></td><td><b>80.19</b></td><td><b>84.19</b></td><td><b>85.05</b></td><td><b>89.01</b></td><td><b>91.42</b></td><td><b>91.92</b></td></tr><tr><td>ViT Baseline</td><td>81.85</td><td>86.08</td><td>87.66</td><td>89.40</td><td>92.19</td><td>93.71</td><td>68.28</td><td>75.81</td><td>78.56</td><td>81.15</td><td>86.24</td><td>87.99</td></tr></tbody></table>

damage, caused by various disasters, including floods, wildfires, earthquakes, hurricanes, and more. The images are sourced from Maxar satellites and offer RGB bands. Due to the difficulty of comparing damages caused by different events, the dataset authors use a four-level damage scale that is harmonized across disaster types. The damage is assessed per building, such that all pixels of a building have the same damage class. As a result, the dataset represents the task of change detection with one class representing 'no building' and four classes representing various levels of building damage caused by the respective disaster. All images have a size of 1024 × 1024. The dataset is highly imbalanced, with the majority of pixels not containing buildings. Furthermore, the distribution of damage classes varies between disaster types, making this a rather challenging dataset. Figure A5 shows example data.

1) Extensive Results: **Details on experimental setup:** To counteract the strong class imbalance, we follow a simple oversampling approach from the winning solution [175] of the xView2 competition: Images without buildings are seen once per training epoch, images with undamaged buildings are seen twice, and images with damaged buildings are seen thrice. Furthermore, when taking random crops in training, we consider multiple candidates and prefer those with more pixels of the underrepresented classes. As the decoder model, we use a UPerNet that concatenates the two embeddings computed for the pre- and post-disaster images by the encoder. For the UNet baseline, we use an input side length of 256 pixels. The ViT uses an input side length of 224 pixels. Due to optimization issues that occurred when using Dice loss in initial experiments, we deviate from the default training protocol when training
<figure><img src="image_11.png" /><figcaption>(a) Pre-disaster image</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Post-disaster image</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) Damage label</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(a) Image</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(b) Land cover label</figcaption></figure>

<caption>Fig. A6: Example data of Five Billion Pixels (Location: China. Domain: Urban Land Cover). The label colors indicate land cover categories.</caption>

models on xView2: Instead of evaluating the model on the validation set every five epochs, we validate after every single epoch. Furthermore, we increase the batch size from the default 8 to 32, to decrease the likelihood of batches that contain no or very few buildings, and thus lead the model into the undesired local minimum. We use the cross-entropy loss for all experiments on xView2. For Table XII, we use the second-worst model Prithvi instead of the worst model SpectralGPT, due to the large computational requirements of training the whole SpectralGPT encoder.

The full results in Table A7 for 100% of the data show that especially the minor and major damage classes are very hard to classify, which is also why we are oversampling them during training. No model achieves ≥30% IoU on minor damage, which is likely due to the dataset imbalance and the difficulty of distinguishing this class from no damage or major damage. Scale-MAE performs consistently strongly, similar to DOFA, GFM-Swin, and RemoteCLIP. All of them are pre-trained on high-resolution data, which makes them well-equipped to extract features from the high-resolution xView2 images. On the other hand, SpectralGPT, the S12 models and Prithvi perform less well, likely due to them being pre-trained on lower-resolution data. This reinforces the intuition that there should be a match between the GFM and the target dataset to achieve a good performance.

Compared to the UPerNets that are trained on top of the frozen GFM encoders, the two fully trained baselines perform surprisingly well. The UNet performs roughly as well as the GFM s trained on high-resolution images, which is surprising, given that it has fewer parameters than a UperNet decoder alone. The ViT model performs similarly well, while using ten times as many parameters as the UNet.

## F. Five Billion Pixels

The Five Billion Pixels dataset is a large-scale, high-resolution land cover classification dataset based on Gaofen-2 satellite imagery, originally covering over 50,000 square kilometers of China. It contains more than 5 billion labeled pixels with a 4-meter spatial resolution, providing detailed annotations across 24 categories, including artificial, agricultural, and natural land covers. An image and label paired example is shown in Fig. A6.

To include it in PANGAEA, we made key modifications to enhance its usability. We cropped the images into 520 × 520 patches, making them more manageable for model input. We then split it into train, validation, and test, making stratified sampling of 70%, 10%, and 20%. Before performing this split, we created an out-of-domain test set (which wasn't available in the original release), carefully selecting 22 patches from the northern regions of China to avoid spatial autocorrelation.

1) *Extensive Results:* The results from Table A8 reveal interesting key insights regarding GFM performance, especially in light of pre-training data characteristics. Models pre-trained on high-resolution images, such as RemoteCLIP, GFM-Swin, and Scale-MAE, consistently achieve the highest scores across most metrics. This superiority reflects the importance of high-resolution imagery in tasks involving a large number of classes, where greater image detail likely aids in capturing clear boundaries and distinguishing textures essential for accurate segmentation.

The effectiveness of RemoteCLIP -with the best mIOU (66.22%), m-f1 (76.41%), m-precision (74.55%), and m-accuracy (94.37%)- also underscores the value of integrating textual information, as it appears particularly beneficial for tasks requiring nuanced class differentiation (see MADOS as well). Models like RemoteCLIP that leverage textual embeddings perform exceptionally well, suggesting that text-based features provide enhanced semantic context. This capability likely helps the model to generalize more effectively across the large number of classes, where textual information augments visual input and reinforces finer distinctions that may not be easily identifiable in pixel-based imagery alone. This is true especially, when the distinctions among classes cannot be based on spectral information.

In contrast, GFM s pre-trained primarily on lower-resolution data perform poorly, as evidenced by models like SpectralGPT and Prithvi, which score significantly lower in nearly all metrics. SpectralGPT, with an mIOU of 32.08% and m-f1 of 40.47%, demonstrates the limitations of a low-resolution multi-spectral pre-training approach when the dataset has restricted spectral diversity, such as the FBPs dataset with only four bands.

UNet still demonstrates respectable results with an mIOU of 58.25% and an accuracy of 93.38%. This suggests that, although specialized models with high-resolution or textual pre-training perform best, the baseline UNet architecture can
<table><caption>TABLE A7: Bi-temporal change detection results on xView2 using 100% of the training set, best results in **bold**, second best underlined.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">mIoU ↑</th><th rowspan="2">m-F1 ↑</th><th rowspan="2">m-acc ↑</th><th colspan="4">Per-class IoU ↑</th></tr><tr><th>No build</th><th>No dam</th><th>Minor dam</th><th>Major dam</th><th>Destr.</th></tr></thead><tbody><tr><td>CROMA</td><td>53.27</td><td>65.81</td><td>97.21</td><td>97.77</td><td>60.64</td><td>19.97</td><td>36.13</td><td>51.85</td></tr><tr><td>DOFA</td><td><u>59.64</u></td><td><u>71.73</u></td><td><u>97.88</u></td><td>98.33</td><td><u>70.22</u></td><td>25.27</td><td>45.45</td><td><u>58.91</u></td></tr><tr><td>GFMSwin</td><td>59.15</td><td>71.45</td><td>97.73</td><td>98.19</td><td>68.11</td><td><u>25.92</u></td><td>45.45</td><td>58.06</td></tr><tr><td>Prithvi</td><td>49.35</td><td>61.92</td><td>96.92</td><td>97.36</td><td>56.66</td><td>16.54</td><td>35.13</td><td>41.05</td></tr><tr><td>RemoteCLIP</td><td>57.41</td><td>69.87</td><td>97.58</td><td>98.01</td><td>65.75</td><td>24.05</td><td>43.76</td><td>55.50</td></tr><tr><td>SatlasNet</td><td>52.23</td><td>64.66</td><td>97.14</td><td>97.76</td><td>59.96</td><td>19.25</td><td>32.43</td><td>51.78</td></tr><tr><td>Scale-MAE</td><td><b>60.72</b></td><td><b>72.70</b></td><td><b>97.99</b></td><td><b>98.45</b></td><td><b>71.43</b></td><td><b>26.18</b></td><td><b>47.76</b></td><td><b>59.78</b></td></tr><tr><td>SpectralGPT</td><td>48.40</td><td>60.89</td><td>96.72</td><td>97.36</td><td>54.20</td><td>17.06</td><td>28.64</td><td>44.74</td></tr><tr><td>S12-MoCo</td><td>51.59</td><td>64.29</td><td>97.09</td><td>97.57</td><td>58.71</td><td>18.97</td><td>37.56</td><td>45.14</td></tr><tr><td>S12-DINO</td><td>50.56</td><td>63.38</td><td>96.95</td><td>97.39</td><td>57.03</td><td>19.36</td><td>36.26</td><td>42.77</td></tr><tr><td>S12-MAE</td><td>50.44</td><td>63.05</td><td>96.99</td><td>97.44</td><td>57.95</td><td>17.44</td><td>36.15</td><td>43.24</td></tr><tr><td>S12-Data2Vec</td><td>51.36</td><td>64.14</td><td>97.03</td><td>97.49</td><td>57.97</td><td>19.29</td><td>38.32</td><td>43.73</td></tr><tr><td>UNet Baseline</td><td>58.68</td><td>70.80</td><td>97.82</td><td>98.34</td><td>68.97</td><td>23.62</td><td>44.49</td><td>57.96</td></tr><tr><td>ViT Baseline</td><td>57.43</td><td>70.00</td><td>97.53</td><td>97.91</td><td>65.26</td><td>24.36</td><td><u>46.74</u></td><td>52.87</td></tr></tbody></table>

reasonably capture fundamental spatial structures. However, its performance does not match models benefitting from additional pre-training on high-resolution imagery and textual data, which provide a distinct advantage for complex classification tasks. Overall, these results reinforce that models trained on high-resolution imagery and enhanced by textual information are particularly suited to tasks involving numerous classes and poor spectral information, whereas low-resolution pre-training limits segmentation effectiveness due to insufficient detail and limited spectral diversity.

## G. DynamicEarthNet

DynamicEarthNet dataset is known for its daily observations compared to most datasets with irregular and long revisit intervals. The sequences are acquired from January 2018 to December 2019 with imagery from Planet Labs, covering 75 areas of interest distributed over the globe. Four bands (blue, green, red, and near-infrared) are provided with a ground sample distance of 3 meters and a resolution of 1024 × 1024. Pixel-wise semantic segmentation labels are available for the first day of every month, which annotate 6 land use and land cover classes including impervious surface, agriculture, forest, other vegetation, wetlands, soil, water (snow & ice are annotated but not used due to its extreme scarcity). To reduce the computational requirements for our benchmark, we only sample the images from the first 6 dates in every month as the input image sequences. Example images are shown in Figure A7. Note that, DynamicEarthNet is a challenging dataset with a large distribution shift from train set to validation set and test set.

1) **Extensive Results:** Table A9 demonstrates the extensive results on the DynamicEarthNet Dataset using either linear or L-TAE temporal feature fusion. In general, fusing temporal features with a simple linear layer results in better performance than L-TAE on DynamicEarthNet. Given daily observations with extremely small changes, a simple linear fusion can be more straightforward to capture useful information. Another hypothesis is that linear fusion can better preserve the generalizable representations from GFMs while a heavier L-TAE time

strategy may cost more severe overfitting since its additional parameters require more data to be trained properly.

Tables A10 and A11 present the performance of foundation models given a limited portion of training data. When using the full dataset, the best-performing foundation models include CROMA, DOFA and SpectralGPT. These outstanding models are pre-trained on multi-modal data, which is likely the origin of the robustness against the large distribution shift from training to testing. However, Prithvi is an exception as it fails to cope with missing bands during testing time. UNet and ViT Baselines are also competitive on DynamicEarthNet. When using less data, the performance of fully-supervised UNet drops significantly while other models with pre-trained weights still mostly maintain the performance. We notice that using less training data might even lead to increasing performance, such as Prithvi, RemoteCLIP and ViT baseline. For these models, shorter training outweighs using more training data in DynamicEarthNet because of less overfitting.

## H. Crop Type Mapping-South Sudan

We include a crop-type semantic segmentation dataset in Africa, specifically South Sudan, to expand the benchmark dataset for underrepresented regions in agricultural areas. The Crop Type Mapping [164] dataset for South Sudan consists of Sentinel-1, Sentinel-2, and Planet satellite imagery in 2017, covering 837 fields and 4 types of crops. The temporal resolution of each scene is variable, with frames that are not temporally aligned. The area of interest is subdivided into 64 × 64 pixel grids. For our experiments, we use data from Sentinel-1 and Sentinel-2, which provide 10 spectral bands (B2-B8, B8A, B11, B12) and 2 polarimetric bands (VV, VH), respectively. An example sample is shown in Figure A8.

1) **Extensive Results:** Table A12 presents the extensive results of the Crop Type Mapping-SS Dataset, with both linear and L-TAE mapping strategies used. Notably, the S12 models, pre-trained on the full spectral bands of Sentinel-2 data, demonstrate top performance. This highlights the importance of leveraging the rich spectral information captured by Sentinel-2, which is particularly crucial for discerning subtle