# OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu

► To cite this version:

Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu. OmniSat: Self-Supervised Modality Fusion for Earth Observation. 2024. hal-04556598

HAL Id: hal-04556598

https://hal.science/hal-04556598

Preprint submitted on 23 Apr 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
<figure><img src="image_1.png" /></figure>

HAL
open science

# OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc, Nicolas Gonthier, Clément Mallet, Loic Landrieu

<figure><img src="image_2.png" /></figure>

## To cite this version:

Guillaume Astruc, Nicolas Gonthier, Clément Mallet, Loic Landrieu. OmniSat: Self-Supervised Modality Fusion for Earth Observation. ECCV, 2024, Milan (Italie), Italy. hal-04556598v2

HAL Id: hal-04556598

https://hal.science/hal-04556598v2

Submitted on 12 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc¹, ⁴, ³ Nicolas Gonthier¹, ² Clement Mallet¹ Loic Landrieu³

¹ Univ Gustave Eiffel, IGN, ENSG, LASTIG, France

³ LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France

⁴ CESBIO, Univ de Toulouse, CNES/CNRS/IRD/INRAE/UPS, France

² IGN, France

## Abstract

The field of Earth Observations (EO) offers a wealth of data from diverse sensors, presenting a great opportunity for advancing self-supervised multimodal learning. However, current multimodal EO datasets and models focus on a single data type, either mono-date images or time series, which limits their expressivity. We introduce OmniSat, a novel architecture that exploits the spatial alignment between multiple EO modalities to learn expressive multimodal representations without labels. To demonstrate the advantages of combining modalities of different natures, we augment two existing datasets with new modalities. As demonstrated on three downstream tasks—forestry, land cover classification, and crop mapping—OmniSat can learn rich representations in an unsupervised manner, leading to improved performance in the semi- and fully-supervised settings, even when only one modality is available for inference. The code and dataset are available at github.com/gastruc/OmniSat.

## 1 Introduction

Self-supervised multimodal learning is of significant interest within the computer vision [1, 2, 3] and Earth Observation (EO) [4, 5] communities. EO is particularly well-suited for developing and evaluating such approaches, thanks to the large amount of open-access data captured by sensing technologies with complementary capabilities [6, 7]. Moreover, combining different sources of EO observations is crucial for several high-impact applications, including environmental [8, 9, 10] and climate monitoring [11, 12], as well as improving food security [13]. Learning with few or no labels is essential for developing regions with limited data annotation capabilities [14, 15, 16].

Despite this potential, most multimodal EO datasets and models focus on a single data type, either mono-date images or time series. This limitation prevents them from simultaneously leveraging the spatial resolution of aerial images [17, 18], the temporal and spectral resolutions of optical satellite time series [19], and the resilience of radar to weather effects [20, 21]. Additionally, existing approaches are often specialized for a given set of sensors, resulting in poor generalization and limited applicability to downstream tasks.

To address these challenges, we propose OmniSat, a novel architecture designed for the self-supervised fusion of diverse EO data. Existing multimodal approaches consider multiple unrelated observations from different modalities, and map each one to a pivot modality [2, 3] or a shared latent space [22, 23]. In contrast, OmniSat combines multiple views of the same area from different modalities into a single representation. The resulting multimodal feature merges the specific information captured by each modality into a single vector [24, 25, 26].

In computer vision, obtaining finely aligned multimodal observations generally requires specialized sensors [27, 28, 29] or the computation of complex mappings between each modality [30, 31]. On

arXiv:2404.08351v1 [cs.CV] 12 Apr 2024
# OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc<sup>1,4,3</sup>, Nicolas Gonthier<sup>1,2</sup>,
Clement Mallet<sup>1</sup>, and Loic Landrieu<sup>1,3</sup>

<sup>1</sup> Univ Gustave Eiffel, IGN, ENSG, LASTIG, France

<sup>2</sup> IGN, France

<sup>3</sup> LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France

<sup>4</sup> CNES, France

**Abstract.** The diversity and complementarity of sensors available for Earth Observations (EO) calls for developing bespoke self-supervised multimodal learning approaches. However, current multimodal EO datasets and models typically focus on a single data type, either mono-date images or time series, which limits their impact. To address this issue, we introduce OmniSat, a novel architecture able to merge diverse EO modalities into expressive features without labels by exploiting their alignment. To demonstrate the advantages of our approach, we create two new multimodal datasets by augmenting existing ones with new modalities. As demonstrated for three downstream tasks—forestry, land cover classification, and crop mapping—OmniSat can learn rich representations without supervision, leading to state-of-the-art performances in semi- and fully supervised settings. Furthermore, our multimodal pretraining scheme improves performance even when only one modality is available for inference. The code and dataset are available at https://github.com/gastruc/OmniSat.

**Keywords:** Earth observation · Multi-modality · Self-supervised learning

## 1 Introduction

Self-supervised multimodal learning has recently gathered significant interest within computer vision [38, 84, 104]. Earth Observation (EO) is also particularly well-suited for developing and evaluating such approaches [29, 52], thanks to the large amount of open-access data captured by sensing technologies with complementary capabilities [36, 81]. Combining different sources of EO observations is crucial for several high-impact applications, including environmental [21, 82, 85] and climate monitoring [58, 99], as well as improving food security [68]. Moreover, learning with few or no labels is essential for developing regions with limited data annotation capabilities [5, 56, 64].

Despite this potential, most multimodal EO datasets and models focus on a single data type, either mono-date images or time series. This limitation prevents them from simultaneously leveraging the spatial resolution of aerial images [59, 66], the temporal and spectral resolutions of optical satellite time series [26], and the resilience of radar to weather effects [4, 67]. Additionally, existing approaches are often limited for a given set of sensors, limiting their applicability.

To address these challenges, we introduce OmniSat, a novel architecture designed for the self-supervised fusion of diverse EO data. Existing multimodal approaches often map
<table><caption>Table 1: **Publicly Available Multimodal EO Datasets.** We provide in parenthesis the spatial resolutions of the single-date images and labels, and the temporal resolutions of time series. S1/S2 denotes Sentinel-1 and 2. \* : modalities added in this work.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">Modalities</th><th rowspan="2">Labels</th></tr><tr><th>images (single date)</th><th>time series</th></tr></thead><tbody><tr><td>SpaceNet6 [38]</td><td>SAR+optical (0.5m-2m)</td><td>✗</td><td>building footprint (1m)</td></tr><tr><td>TreeSatAI [35]</td><td>aerial (0.2m) &amp; S1/S2 (10m)</td><td>✗</td><td>forestry (60m)</td></tr><tr><td>BigEarthNet [39]</td><td>S1/S2 (10m)</td><td>✗</td><td>land cover (100m)</td></tr><tr><td>DFC20 [40]</td><td>S1/S2 (10m)</td><td>✗</td><td>land cover (500m)</td></tr><tr><td>MDAS [41]</td><td>S1/S2 + hyperspectral (2.2-10m)</td><td>✗</td><td>land cover (0.25m)</td></tr><tr><td>PASTIS-R [36, 37]</td><td>✗</td><td>S1/S2 (30-140 / year)</td><td>agriculture (10m)</td></tr><tr><td>SSL4EO-S12 [42]</td><td>✗</td><td>S1/S2 (4 / year)</td><td>✗</td></tr><tr><td>DFC21-DSE [43]</td><td>✗</td><td>S1/S2 + LS8 (3-9/year)</td><td>human activity (500m)</td></tr><tr><td>MapInWild [44]</td><td>✗</td><td>S1/S2 (4 / years)</td><td>protected areas (10m)</td></tr><tr><td>SEN12MS-CR-TS [45]</td><td>✗</td><td>S1/S2 (30 / years)</td><td>cloud cover (10m)</td></tr><tr><td>MultiSenge [46]</td><td>✗</td><td>S1/S2 (30-140 / years)</td><td>land cover (10m)</td></tr><tr><td>WildfireSpreadTS [47]</td><td>✗</td><td>VIIRS + Weather (1 / day)</td><td>fire events (375m)</td></tr><tr><td>FLAIR [48]</td><td>aerial (0.2m)</td><td>S2 (20-114 / year)</td><td>land cover (0.2m)</td></tr><tr><td>Satlas [49]</td><td>NAIP (1m)</td><td>S2 (8-12 / year)</td><td>various</td></tr><tr><td><strong>PASTIS-HD</strong></td><td>* SPOT 6-7 (1.5m)</td><td>S1/S2 (30-140 / year)</td><td>agriculture (10m)</td></tr><tr><td><strong>TreeSatAI-TS</strong></td><td>aerial (0.2m)</td><td>* S1/S2 (10-70 / year)</td><td>forestry (60m)</td></tr></tbody></table>

the other hand, EO data can be naturally aligned spatially with georeferencing. To leverage this property, we adapt multimodal contrastive learning [32, 33] and cross-modal masked auto-encoding techniques [34] to learn rich multimodal EO representations with a generalist fusion scheme.

To address the scarcity of EO datasets with a diverse range of heterogeneous modalities (see Table 1), we enrich the TreeSatAI [35] and PASTIS-R [36, 37] datasets with new aligned modalities. This allows us to evaluate OmniSat's ability to handle an arbitrary number of inputs with varying natures and resolutions. Our contributions can be summarized as follows:

* We introduce OmniSat, a new model that learns to combine varied sources of EO observations in a self-supervised manner, resulting in richer joint representations that capture the unique characteristics of each modality.

* We augment two EO benchmarks to create the first datasets with three modalities of different natures (very high resolution images, optical time series, and SAR time series).

* We demonstrate that utilizing diverse modalities with our flexible model leads to better representations, establishing new states-of-the-art for tree species, crop type, and land cover classification. Furthermore, our self-supervised training with multiple modalities improves performance even when only one modality is available during inference.

## 2 Related Work

This section provides an overview of the fields of self-supervised and multimodal learning, emphasizing the specificities of their usage for Earth observation. Lastly, we highlight the scarcity of multimodal EO datasets with diverse data types.

**Self-Supervised Learning.** This technique consists in learning expressive data representations without labels by using a pretext task. This approach has been particularly successful for natural language [50] and image [51] analysis. Initially focused on discriminative tasks [52, 53, 54], recent self-supervised approaches for images can be categorized as contrastive or generative.

Contrastive methods minimize the distance between representations of paired samples, often the same image under different transformations, and maximize the distance with other samples
multiple unrelated observations from different modalities to one pivot modality [38,84] or a shared latent space [39, 86]. In contrast, OmniSat merges multiple views of the same area from different modalities into a single representation combining the specific information of each modality [14, 41, 75].

In computer vision, obtaining finely aligned multimodal observations generally requires specialized sensors [55, 61, 69] or the computation of complex mappings between modalities [23, 77]. On the other hand, EO data can be naturally aligned with georeferencing. To leverage this property, we adapt multimodal contrastive learning [50, 74] and cross-modal masked auto-encoding techniques [43] to learn rich multimodal EO representations with a generalist fusion scheme and without annotations.

To address the scarcity of EO datasets with a diverse range of heterogeneous modalities (see Tab. 1), we enrich the TreeSatAI [2] and PASTIS-R [33, 34] datasets with new aligned modalities. This allows us to evaluate OmniSat's ability to handle an arbitrary number of inputs with varying natures and resolutions. Our contributions can be summarized as follows:

- We introduce OmniSat, a new model that learns to combine varied sources of EO observations in a self-supervised manner, resulting in richer joint representations that capture the unique characteristics of each modality.

- We augment two EO benchmarks to create the first datasets with three modalities of different natures (very high resolution images, optical and SAR time series).

- We demonstrate that OmniSat can leverage diverse modalities to learn rich representations, establishing new states-of-the-art for tree species, crop type, and land cover classification. Furthermore, our cross modal self-supervised training scheme improves performance even when only one modality is available during inference.

## 2 Related Work

This section provides an overview of self-supervised and multimodal learning, emphasizing the specificities of their usage for Earth observation. Lastly, we highlight the scarcity of multimodal EO datasets with diverse data types.

**Self-Supervised Learning.** This technique consists in learning expressive data representations without labels by using a pretext task. This approach has been particularly successful for natural language [53] and image [72] analysis. Initially focused on discriminative tasks [37, 70, 102], recent self-supervised approaches for images can be categorized as contrastive or generative.

*Contrastive methods* minimize the distance between representations of paired samples, often the same image under different transformations, and maximize the distance with other samples [15, 17, 46]. More efficient methods only consider positive samples and avoid mode collapse by introducing various asymmetries [18, 42] or normalization [16]. Such approaches have been successfully adapted to EO, for which samples are paired according to their location [91] or time of acquisition [7, 65].

*Generative methods* reason at the level of individual token—a small portion of the input, typically a patch for images [25]. The objective is to reconstruct the masked tokens of an input image in pixel [10, 45, 97] or feature space [6]. This principle has been successfully adapted to EO analysis [20, 30, 101], and was further extended to handle multiple spatial scales [76], multimodality [29,52], or hyperspectral observations [51,62].
<figure><img src="image_1.png" /><figcaption>**Figure 1:** Datasets. We represent three tiles from our considered multilabel classification datasets: FLAIR (a), TreeSatAI-TS (b) and PASTIS-HD (c). TreeSatAI-TS is a new dataset built by replacing the single-date Sentinel-1 and 2 images of TreeSatAI [35] by year-long time series. PASTIS-HD (c) is a version of PASTIS-R [37] enriched with VHR satellite images. NIR stands for Near-InfraRed.

*: modalities added in this work.</figcaption></figure>

[55, 56, 57]. More efficient methods only consider positive samples and avoid mode collapse by
introducing various asymmetries [58, 59] or normalization [60]. Such approaches have been suc-
cessfully adapted to EO, for which samples are paired according to their location [61] or time of
acquisition [62, 63].

*Generative methods* reason at the level of individual token—a small portion of the input, typically a patch for images [64]. The objective is to reconstruct the masked tokens of an input image in pixel [65, 66, 67] or feature space [68]. This principle has been successfully adapted to EO analysis [69, 70, 71], and was further extended to handle multiple spatial scales [72], multimodality [4, 5], or hyperspectral observations [73, 74].

Several hybrid approaches combine the discriminative power of contrastive methods and the scalability of generative objectives for natural images [51, 75] and EO data [4]. Our proposed OmniSat also implements both mechanisms. A key difference is that the precise alignment between different sources of EO data allows us to contrastively match small patches of different modalities rather than entire images or time series.

**Self-Supervised Multimodal Learning.** Multimodal computer vision has received a lot of interest [76], notably due to the success of cross-modal pre-training [32]. Recent models align the embeddings of heterogeneous modalities such as video and sound [33], depth and images [77], text and image [78, 79], or multiple combinations of these modalities [2, 3, 22, 23].

Multimodal learning also has a long history in EO [80, 81, 82] due to the large variety and comple-
mentarity of sensors [6, 7]. However, recent transformer-based architectures [83] for EO are often
limited to one type of modality, be it a single image [70, 72] or time-series [37, 84]. For example,
CROMA [4] and PRESTO [85] are specifically designed for paired optical and radar observations,
but cannot handle very high resolution (VHR) data. USat [5] considers images with different res-
olutions, but only takes a single date within a time series. UT&T [48] can natively take single and
multi-date observations of different modalities, but cannot be easily pre-trained in a self-supervised
manner since it relies on convolutions and an ad-hoc late fusion scheme.
<table><caption>**Table 1: Publicly Available Multimodal EO Datasets.** We provide in parenthesis the spatial resolutions of the single-date images and labels, and the temporal resolutions of time series. S1/S2 denotes Sentinel-1 and 2. * : **modalities added in this work.**</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">Modalities</th><th rowspan="2">Labels</th></tr><tr><th>images (single date)</th><th>time series</th></tr></thead><tbody><tr><td>SpaceNet6 [83]</td><td>SAR+optical (0.5m-2m)</td><td>✗</td><td>building footprint (&lt;1m)</td></tr><tr><td>TreeSatAI [2]</td><td>aerial + S1/S2 (0.2-10m)</td><td>✗</td><td>forestry (60m)</td></tr><tr><td>BigEarthNet [88]</td><td>S1/S2 (10m)</td><td>✗</td><td>land cover (100m)</td></tr><tr><td>DFC20 [78]</td><td>S1/S2 (10m)</td><td>✗</td><td>land cover (500m)</td></tr><tr><td>MDAS [49]</td><td>S1/S2 + hyperspectral (2.2-10m)</td><td>✗</td><td>land cover (0.25m)</td></tr><tr><td>DOFA [98]</td><td>NAIP + Gaofen + S1/S2 + EnMAP (1-30m)</td><td>✗</td><td>✗</td></tr><tr><td>PASTIS-R [33, 34]</td><td>✗</td><td>S1/S2 (30-140 / year)</td><td>agriculture (10m)</td></tr><tr><td>SSL4EO-S12 [94]</td><td>✗</td><td>S1/S2 (4 / year)</td><td>✗</td></tr><tr><td>DFC21-DSE [63]</td><td>✗</td><td>S1/S2 + LS8 (3-9/year)</td><td>human activity (500m)</td></tr><tr><td>MapInWild [28]</td><td>✗</td><td>S1/S2 (4 / years)</td><td>protected areas (10m)</td></tr><tr><td>SEN12MS-CR-TS [27]</td><td>✗</td><td>S1/S2 (30 / years)</td><td>cloud cover (10m)</td></tr><tr><td>MultiSenge [95]</td><td>✗</td><td>S1/S2 (30-140 / years)</td><td>land cover (10m)</td></tr><tr><td>FLAIR [31]</td><td>aerial (0.2m)</td><td>S2 (20-114 / year)</td><td>land cover (0.2m)</td></tr><tr><td>Satlas [12]</td><td>NAIP (0.5 -2m)</td><td>S2 (8-12 / year)</td><td>various</td></tr><tr><td>PASTIS-HD</td><td>* SPOT 6-7 (1.5m)</td><td>S1/S2 (30-140 / year)</td><td>agriculture (10m)</td></tr><tr><td>TreeSatAI-TS</td><td>aerial (0.2m)</td><td>* S1/S2 (10-70 / year)</td><td>forestry (60m)</td></tr></tbody></table>

Several hybrid approaches combine the discriminative power of contrastive methods and the scalability of generative objectives for natural images [72, 103] and EO data [29]. Our proposed OmniSat model also implements both mechanisms. A key feature is that we leverage the precise alignment between different sources of EO data to contrastively match small patches of different modalities rather than entire images or time series.

**Self-Supervised Multimodal Learning.** Multimodal computer vision has received a lot of interest [13], notably due to the success of cross-modal pre-training [74]. Recent models align the embeddings of heterogeneous modalities such as video and sound [50], depth and images [44], text and image [3, 9], or multiple combinations of these modalities [38, 39, 84, 86].

Multimodal learning also has a long history in EO [60, 73, 100] due to the large variety and complementarity of sensors [36, 81]. However, recent transformer-based architectures [92] for EO are often limited to one type of modality, be it a single image [20, 76] or time-series [34, 89]. For example, CROMA [29] and PRESTO [90] are specifically designed for paired optical and radar observations, but cannot handle Very High Resolution (VHR) data. USat [52] considers images with different resolutions, but only takes a single date within a time series. UT&T [31] can natively take single and multi-date observations of different modalities, but cannot be easily pre-trained in a self-supervised manner since it relies on convolutions and an ad-hoc late fusion scheme.
<figure><img src="image_2.png" /><figcaption>**Figure 2: OmniSat Architecture.** We illustrate OmniSat for *M* = 3 modalities, and a tile split into *P* = 4 patches. The *M* × *P* input tokens $x_{\mathbf{P}}^{M}$ are encoded by *M* modality-specific encoders $\mathcal{E}^{M}$, yielding the token representations $f_{\mathbf{P}}^{M}$. The module *C* combines them into multimodal patch representations $f_{\mathbf{P}}^{*}$. The token embeddings $f_{\mathbf{P}}^{M}$ are supervised by a contrastive cross-modal objective. We also use a reconstruction objective: the masked multimodal representations $f_{\mathbf{P}}^{*}$ are decoded by modality-specific networks $\mathcal{D}^{M}$ to reconstruct their corresponding inputs in $x_{\mathbf{P}}^{M}$.</figcaption></figure>

**Multimodal EO Datasets.** As reported in Table 1, many multimodal EO datasets use Sentinel-1 [86] and 2 [19] data for applications ranging from land cover to forestry analysis and fire detection. We also note that most multimodal datasets only contain data of one type: mono-date image or time series. Several datasets (BigEarthNet [39], DFC20 [40], MDAS [41]) select a single date from time series. However, single Sentinel-1 and 2 acquisitions can be significantly affected by rain and cloud cover, respectively. Furthermore, capturing the temporal dynamics is crucial to characterize the phenology of vegetation [87],

FLAIR [48] is the first multimodal EO dataset to propose both very high spatial resolution (≤ 2m) and high temporal resolution (> 4 images/year). Satlas [49] combines sentinel-2 time series and for 5% to tiles (continental US), very high definition NAIP images. The functional map of the World [88] integrates observations from various sensors, but most areas are only observed with one sensor. Two other datasets contain time series and single images from multiple sources, but were not available at the time of submission: IARPA-SMART [89] and DOFA [90].

To showcase how OmniSat can consume an arbitrary number of modalities with different spatial, spectral, and temporal resolutions, we selected two commonly used EO benchmarks, TreeSatAI [35] and PASTIS-R [37], whose focus on crop type mapping and forestry differs from the land cover analysis of FLAIR. We added new modalities to these datasets to reach three distinct data types: VHR aerial images, optical time series, and SAR time series. See Figure 1 for an illustration, and Section 4.1 for more details on how we extended these datasets.

3 Method

We consider a multimodal dataset *x*, defined as a collection of tiles divided into a set **P** of *P* small
spatial patches. We denote by **M** the set of *M* available modalities. The patches are defined con-
sistently across modalities: $x_p^M = \{x_p^m\}_{m \in \mathcal{M}}$ corresponds to *M* distinct views of the same patch
*p* with different sensors or modalities. Each modality *m* has its unique input space $\Omega^m$ such that
$x_p^m \in \Omega^m$. We define an input token as a pair $(m, p)$ for a given modality *m* and a patch *p*, for a
total of $M \times P$ tokens.

Our goal is to learn multimodal representations $f_p^*$ that capture the content of each spatial patch *p* captured by all modalities *M* in a self-supervised fashion. To achieve this, we employ a cross-modal
<figure><img src="image_3.png" /><figcaption>**Fig. 1: Datasets.** We represent three tiles from the considered multilabel classification datasets: FLAIR (a), TreeSatAI-TS (b) and PASTIS-HD (c). TreeSatAI-TS is a new dataset built by replacing the single-date Sentinel-1 and 2 images of TreeSatAI [2] by year-long time series. PASTIS-HD (c) adds VHR satellite images to PASTIS-R [34]. ⋆: modalities added in this work.</figcaption></figure>

**Multimodal EO Datasets.** As reported in Table 1, many multimodal EO datasets use Sentinel-1 [11] and 2 [26] data for applications ranging from land cover to forestry analysis and fire detection. We also note that most multimodal datasets only contain data of one type: mono-date image or time series. Several datasets (BigEarthNet [88], DFC20 [78], MDAS [49]) select a single date from time series. However, single Sentinel-1 and 2 acquisitions can be significantly affected by rain and cloud cover, respectively. Furthermore, capturing the temporal dynamics is crucial to characterize the phenology of vegetation [93],

FLAIR [31] is the first multimodal EO dataset to propose both very high spatial resolution (≤ 2m) and high temporal resolution (> 4 images/year). Satlas [12] combines sentinel-2 time series and for 5% to tiles (continental US), very high definition NAIP images. The functional map of the World [19] integrates observations from various sensors, but most areas are only observed with one sensor. Two other datasets contain time series and single images from multiple sources, but were not available at the time of writing this article: IARPA-SMART [40] and DOFA [98].

To showcase how OmniSat can consume an arbitrary number of modalities with different spatial, spectral, and temporal resolutions, we selected two commonly used EO benchmarks, TreeSatAI [2] and PASTIS-R [34], whose focus on crop type mapping and forestry differs from the land cover analysis of FLAIR. We added new modalities to these datasets to reach three distinct data types: VHR aerial images, optical time series, and SAR time series. See Fig. 1 for an illustration, and Sec. 4.1 for more details on how we extended these datasets.

3 Method

We consider a tile $x$ observed through a set $M$ of $M$ distinct sensors or modalities.
The goal of the OmniSat model is to learn in a self-supervised fashion to combine
<figure><img src="image_3.png" /><figcaption>Figure 3: **Contrastive Loss.** We represent the token matching matrix for two tiles $T_1$ and $T_2$ viewed across 3 modalities $m_1, m_2$, and $m_3$. $T_1$ is composed of the patches $p_1$ and $p_2$, while $T_2$ comprises $q_1$ and $q_2$. In contrast to classic approaches which ignore the diagonal and assign each sample with a single positive match, our loss defines operates at the patch level, considers multiple positives per token, and excludes tokens in a block-diagonal fashion.</figcaption></figure>

<figure><img src="image_4.png" /></figure>

contrastive objective (Section 3.1) and a multimodal masked encoding task (Section 3.2). We then give further details on the implementation of each module in Section 3.3. The overall architecture is represented in Figure 2.

## 3.1 Contrastive Objective

We associate each modality $m$ with a dedicated patch encoder $\mathcal{E}^m : \Omega^m \to \mathbb{R}^d$ and denote by $f_p^m = \mathcal{E}^m(x_p^m)$ the $d$-dimensional embedding of the input token $x_p^m$. We would like $f_p^m$ to capture robust and expressive features of $x_p^m$. To do so, we build a matching objective: patches should have consistent embeddings across modalities. Indeed, while each modality captures distinct characteristics of $p$, all encodings $f_p^m$ share the same latent variable: the semantic content of patch $p$.

In practice, we want $f_p^m$ close to $f_q^n$ for $n \neq m$, but far from $f_q^n$ for other patches $q \neq p$. We adapt the classic InfoNCE loss [91] to our setting with two main differences, illustrated in Figure 3: (i) each token $(m, p)$ has $M-1$ positive matches: the tokens corresponding to the same patch $p$ but viewed in another modality $n \neq m$; and (ii) as EO observations are generally spatially regular, nearby patches may be visually indistinguishable. Therefore, we exclude from the negative matches of $(m, p)$ all tokens in modality $m$ that are too close to $p$. To this end, we remove the set $T(m, p)$ of tokens with modality $m$ and whose patches are in the same tile as $p$. Our loss function $\mathcal{L}_{\text{contrast}}$ is defined as such:

$$ \mathcal{L}_{\text{contrast}} = \frac{1}{MP} \sum_{(m,p) \in \mathbf{M} \times \mathbf{P}} \log \left( \frac{\sum_{n \neq m} \exp(\langle f_p^m, f_p^n \rangle / \gamma)}{\sum_{(n,q) \in \mathbf{M} \times \mathbf{P} \setminus T(m,p)} \exp(\langle f_p^m, f_q^n \rangle / \gamma)} \right), \quad (1) $$

with $\gamma$ a temperature parameter, and $\langle \cdot, \cdot \rangle$ the scalar product in $\mathbb{R}^d$. This function, specifically designed for geospatial data, allows us to contrast individual patches across modalities, which is not typically feasible for natural images. However, as the contrastive objective aligns multimodal representations, the patch encoders $\mathcal{E}^M$ may be encouraged to overlook the distinct attributes of their respective modality. Instead, they may focus only on features shared by all modalities, i.e. their common denominator. To ensure that encoders also capture modality-specific information, we incorporate a reconstruction objective, detailed in Section 3.2.

## 3.2 Multimodal Reconstruction Objective

This section introduces the modality combiner network $\mathcal{C}$ and its reconstruction objective. We mask a fraction of tokens $\mathbf{K} \subset \mathbf{M} \times \mathbf{P}$ and replace their embeddings with a learned vector $f^{\text{mask}} \in \mathbb{R}^d$. Note that the masking can differ across modalities, and some patches may be entirely masked. All tokens are then processed by the modality combining network $\mathcal{C}$, which outputs $\mathbf{P}$ multimodal embeddings $f_{\mathbf{P}}^*$:

$$ f_{\mathbf{P}}^* = \mathcal{C} \left( \{f_p^m\}_{(m,p) \notin \mathbf{K}} \cup \{f^{\text{mask}}\}_{(m,p) \in \mathbf{K}} \right). \quad (2) $$

To encourage the patch embeddings $f_{\mathbf{P}}^*$ to capture information from all modalities, we build a multimodal reconstruction objective. We equip each modality $m$ with a dedicated decoder
<figure><img src="image_4.png" /><figcaption>**Fig. 2: OmniSat Architecture.** We illustrate OmniSat for $M = 3$ modalities, and a tile split into $P = 4$ patches. The $M \times P$ input tokens $x_P^M$ are encoded by $M$ modality-specific encoders $\mathcal{E}^M$, yielding the token representations $f_P^M$. The module $C$ combines them into multimodal patch representations $f_P^*$. The token embeddings $f_P^M$ are supervised by a contrastive cross-modal objective. We also use a reconstruction objective: the masked multimodal representations $f_P^*$ are decoded by modality-specific networks $\mathcal{D}^M$ to reconstruct their corresponding inputs in $x_P^M$.</figcaption></figure>

all modalities $M$ into a multimodal representation $f^*$. We first provide details about OmniSat's architecture in Sec. 3.1. We then explain our training scheme, which consist of a cross-modal contrastive objective (Sec. 3.2) and a multimodal masked encoding task (Sec. 3.3). Finally, we present the implementation details in Sec. 3.4. The overall method is represented in Fig. 2.

## 3.1 Architecture

This section presents the tokenization process, the structure of the encoder and decoder for each modality, and the architecture of the modality combiner network.

**Multimodal Tokenization.** All available modalities are spatially aligned through georeferencing. This allows us to divide the tile $x$ into a set $\mathbf{P}$ of $P$ non-overlapping patches consistently across all modalities: $x_p^M = \{x_p^m\}_{m \in \mathcal{M}}$ corresponds to $M$ distinct views of the same patch $p$ with different modalities. Each modality $m$ takes its values in a space $\Omega^m$ such that $x_p^m \in \Omega^m$. We index tokens with pairs $(m, p)$, defined for each modality $m$ and patch $p$, for a total of $M \times P$ tokens.

Time series from Sentinel satellites may experience registration errors spanning several meters, complicating their precise alignment with high-resolution imagery. However, using temporal sequences of satellite data mitigates these errors as aggregation over time tends to balance out misalignments.
<figure><img src="image_5.png" /><figcaption>Figure 4: Module's Architectures. (a) represents the encoder and decoder of image patches, and (b) illustrates their architectures for temporal patches with a length of $L = 4$ time stamps. The modality combining module $\mathcal{C}$ is depicted in (c) with $P = 2$ and $M = 3$. Elements colored in orange are learned networks or parameters.</figcaption></figure>

$\mathcal{D}^m : \mathbb{R}^d \to \Omega^m$ and write the reconstruction loss as:

$$ \mathcal{L}_{\text{reconstruction}} = \frac{1}{|\mathbf{K}|} \sum_{(m,p) \in \mathbf{K}} \frac{1}{\dim(\Omega^m)} \| \mathcal{D}^m(f_p^*) - x_p^m \|^2, \quad (3) $$

where $\dim(\Omega^m)$ is the dimension of the input space $\Omega^m$. The total loss is the sum of $\mathcal{L}_{\text{reconstruction}}$ and $\mathcal{L}_{\text{contrast}}$.

## 3.3 Implementation

This section presents the tokenization process, the structure of the encoder and decoder for each modality, and the architecture of the modality combiner network. In particular, we highlight several design choices that leverage the specific properties of EO data [92].

**Multimodal Tokenization.** All available modalities are spatially aligned through georeferencing. We split each tile along a regular spatial grid to produce a set of non-overlapping patches $\mathbf{P}$ consistent across all modalities, thus ensuring that $x_p^M$ correspond to the same area for all modalities.

For TreeSat and FLAIR, we use a $10 \times 10$ m grid, meaning that the VHR input tokens are small image patches of size $50 \times 50$ with 0.2m per pixel. The patches of Sentinel observations with a resolution of 10m are single-pixel temporal sequences of spectral measurements. For PASTIS-HD, we use a $40 \times 40$ m grid, meaning that the VHR patches are of size $40 \times 40$ with 1.0m per pixel. The patches of Sentinel observations [19] are $4 \times 4$ image time series which we spatially flatten before encoding.

Time series from Sentinel satellites may experience registration errors spanning several meters, complicating their precise alignment with high-resolution imagery. However, using temporal sequences of satellite data mitigates these errors as aggregation over time tends to balance out misalignments.

**Encoder-Decoder for Images.** Image tiles are split into small square patches: $\Omega^{\text{img}} = \mathbb{R}^{C \times I \times I}$ with $I$ the size of the patches and $C$ the number of channels. As shown in Figure 4a, we encode these inputs with a sequence of convolutions and max-pool layers until the spatial dimension is fully collapsed. Decoding involves a symmetric sequence of convolutions and un-pooling layers. Contrary to existing masked auto-encoders, we pass the pooling indices from the encoder's max-pooling to the decoder's un-pooling in the manner of SegNet [93]. This dispenses the encoder from learning the intra-token spatial configuration and allows it to focus on the radiometric information, which may be more relevant depending on the application.

**Encoder-Decoder for Time Series.** Each temporal patch is represented by $L$ sequential observations with $C$ channels: $\Omega^{\text{TS}} = \mathbb{R}^{C \times L}$, each associated with a time stamp. We encode the temporal patches using a Lightweight Temporal Attention Encoder (LTAE) model [94], an efficient network
<figure><img src="image_5.png" /><figcaption>**Fig. 3: OmniSat Architecture.** OmniSat is composed of dedicated patch encoders for image (a) and time series b, here represented for a length of *L* = 4 time stamps. The modality combining module *C* is depicted in (c) with *P* = 2 and *M* = 3. Elements colored in orange are learned networks or parameters.</figcaption></figure>

**Encoder-Decoder for Images.** Image tiles are split into small square patches: $\Omega^{img} = \mathbb{R}^{C \times W \times W}$ with *W* the size of the patches in pixels and *C* the number of channels. As shown in Fig. 3a, we encode these inputs with a sequence of convolutions and max-pool layers until the spatial dimension is fully collapsed. Decoding involves a symmetric sequence of convolutions and un-pooling layers. Contrary to existing masked auto-encoders, we pass the pooling indices from the encoder's max-pooling to the decoder's un-pooling in the manner of SegNet [8]. This dispenses the encoder from learning the intra-patch spatial configuration. This allows the image encoder to focus on the radiometric information, which may be more relevant depending on the application.

**Encoder-Decoder for Time Series.** Each temporal patch is represented by *L* sequential observations with *C* channels: $\Omega^{TS} = \mathbb{R}^{C \times L}$, each associated with a time stamp. We encode the temporal patches using a Lightweight Temporal Attention Encoder (LTAE) model [32], an efficient network for geospatial time series processing. We decode vector representations into time series by repeating the vector *L* times across the temporal dimension, adding a temporal encoding for each time step, and using an MLP to map the results to size *C*. See Fig. 3b for an illustration.

Optical time series are notoriously affected by clouds [87]. This may affect the validity of the reconstruction task: the decoder cannot know which observations are cloudy, making the reconstruction objective unpredictable. To circumvent this issue, we use the temporal attention maps of the encoder's LTAE to select dates to reconstruct: cloudless observations are more informative and should have a higher attention score [80]. We only consider in the reconstruction loss $\mathcal{L}_{reconstr}$ the top 25% dates in terms of the LTAE's attention maps.

**Modality Combining Network.** The modality combining network *C*, represented in Fig. 3c, takes the *M* x *P* token embeddings $f_{\mathbf{P}}^{\mathbf{M}}$, some of whom can potentially be masked. We equip each token with a Euclidean relative positional encoding [96] calculated based on their patch's position {$r(p, q) | (p, q) \in \mathbf{P}^2$}, allowing each token to
for geospatial time series processing. We decode each multimodal embedding $f_p^*$ into temporal patches by repeating it $L$ times across the temporal dimension, adding a temporal encoding for each time step, and using an MLP to map the results to size $C$. See Figure 4b for an illustration.

Optical time series are notoriously affected by clouds [95]. This may affect the validity of the reconstruction task: the decoder cannot know which observations are cloudy, making the reconstruction objective unpredictable. To circumvent this issue, we use the temporal attention maps of the encoder's LTAE to select dates to reconstruct: cloudless observations are more informative and should have a higher attention score [96]. We only consider in the reconstruction loss $L_{reconstruction}$ the top 25% dates in terms of the LTAE's attention maps.

**Modality Combining Network.** The modality combining network $\mathcal{C}$, represented in Figure 4c, takes the $M \times P$ token embeddings $f_P^M$, some of whom can potentially be masked. We equip each token with a Euclidean relative positional encoding [97] calculated based on their patch's position $\{r(p, q) | (p, q) \in \mathbb{P}^2\}$, and only defined for patches within the same tile. This way, each token can selectively consider its spatial surroundings. As most EO data are captured from above (satellite or aerial), their distribution is invariant by horizontal translation, making this choice of encoding preferable to an absolute position encoding.

The modality combining module $\mathcal{C}$ starts with a series of $B$ residual self-attention blocks connecting all tokens across modality and within the same tile. We then perform cross-attention between the resulting token embeddings $g_P^M \in \mathbb{R}^{d \times M \times P}$ and $P$ copies $f_P^{\text{comb}}$ of a modality combining token $f^{\text{comb}} \in \mathbb{R}^d$ learned as a free parameter. Each copy of $f_p^{\text{comb}}$ is spatially located at the patch $p$ for the relative positional encoding $r$. The mechanism of $\mathcal{C}$ writes:

$$g_P^M = \text{self-attention}(f_P^M; r), \qquad (4)$$

$$f_P^* = \text{cross-attention}(f_P^{\text{comb}}, g_P^M; r). \qquad (5)$$

**Hyperparameters.** To show the versatility of OmniSat, we use the same configuration throughout all experiments. The embedding size is $d = 256$, resulting in image encoders and decoders with 3.6M and 1.8M parameters, 403K and 96K for optical time series, and 402K and 95K for radar time series. The modality combiner module is composed of $B = 6$ residual self-attention blocks and a single cross-attention block, for a total of 3.6M parameters. We train our model with the ADAM optimizer [98], with a learning rate of $10^{-4}$ for pretraining and $2 \times 10^{-5}$ for fine-tuning, and a ReduceLROnPlateau scheduler [99] with a patience of 10 epochs and a decay rate of 0.1. When re-implementing competing methods, we use the hyperparameters of their open-source repository.

# 4 Experiments

In this section, we evaluate OmniSat's performance across three multimodal datasets, including two new datasets introduced in this work and presented in Section 4.1. We outline our experimental protocol and our adaptation of competing methods in Section 4.2. We then present in Section 4.3 our quantitative results and analysis, and qualitative results in Figure 5. Lastly, we conduct an ablation study to in Section 4.4.

## 4.1 Datasets

We evaluate OmniSat on three multimodal datasets: FLAIR [48], and the augmented TreeSatAI-TS [35] and PASTIS-HD [36, 37]. See Figure 1 for an illustration of these two datasets.

**TreeSatAI-TS:** TreeSatAI [35] is a multimodal dataset for tree species identification, containing 50, 381 tiles of 60 × 60m with multi-label annotations for 20 classes and all taken in Germany. Each tile is associated with a very high resolution RGB and near-infrared (NIR) image (0.2m pixel resolution), a single Sentinel-2 multi-spectral image (10m per pixel resolution, 10 bands), and a single Sentinel-1 radar image (10m per pixel resolution, 3 bands: two polarization channels and their ratio).

Motivated by the fact that fine-grained vegetation discrimination relies heavily on temporal dynamics information [87], we introduce TreeSatAI-TS<fnref n="1" />. This extended version uses open-source data to

<footnote n="1">¹The dataset is available at https://huggingface.co/datasets/IGNF/TreeSatAI-Time-Series.</footnote>
selectively consider its spatial surroundings. As most EO data are captured from above (satellite or aerial), their distribution is invariant by horizontal translation, making this choice of encoding preferable to an absolute position encoding.

The modality combining module C starts with a series of B residual self-attention blocks connecting all tokens across modality. We then perform cross-attention between the resulting token embeddings $g_P^M \in \mathbb{R}^{d \times M \times P}$ and P copies $f_P^{comb}$ of a modality combining token $f^{comb} \in \mathbb{R}^d$ learned as a free parameter. Each copy of $f_P^{comb}$ is spatially located at the patch p for the relative positional encoding r. The module C outputs P multimodal encodings $f_P^*$ combining all available modalities for each patch:

$$g_P^M = \text{self-attention}(f_P^M; r) \qquad (1)$$

$$f_P^* = \text{cross-attention}(f_P^{\text{comb}}, g_P^M; r). \qquad (2)$$

## 3.2 Contrastive Objective

We denote by $f_p^m$ the d-dimensional encodings of the input patch $x_p^m$ given by their dedicated encoders. We propose to supervise the embeddings $f_p^m$ with a contrastive objective encouraging spatial consistency across modalities. Indeed, while each modality captures distinct characteristics of p, all encodings $f_p^m$ share the same latent variable: the semantic content of the patch.

In practice, we want $f_p^m$ to be closer to $f_p^n$ for $n \neq m$ than to $f_q^n$ for other patches $q \neq p$. We define B as the set of patches within the current batch of observations. We adapt the classic InfoNCE loss [71] to our setting with two main differences, illustrated in Fig. 4. (i) each token $(m, p)$ has $M-1$ positive matches: the tokens corresponding to the same patch p but viewed in another modality $n \neq m$; and (ii) as EO observations are generally spatially regular, nearby patches may be visually indistinguishable. Therefore, we exclude from the negative matches of $(m, p)$ all tokens in modality m and which are too close to p. To this end, we remove the set $T(m, p)$ of tokens with modality m and whose patches are in the same tile as p. Our loss function $\mathcal{L}_{\text{contrast}}$ is defined as such:

$$\mathcal{L}_{\text{contrast}} = \frac{1}{M|\mathbf{B}|} \sum_{(m,p) \in \mathbf{M} \times \mathbf{B}} \log \left( \frac{\sum_{n \neq m} \exp(\langle f_p^m, f_p^n \rangle / \gamma)}{\sum_{(n,q) \in \mathbf{M} \times \mathbf{B} \setminus T(m,p)} \exp(\langle f_p^m, f_q^n \rangle / \gamma)} \right), \quad (3)$$

with $\gamma$ a temperature parameter, and $\langle \cdot, \cdot \rangle$ the scalar product in $\mathbb{R}^d$. This function, specifically designed for geospatial data, allows us to contrast individual patches across modalities, which is not typically feasible for natural images. However, as the contrastive objective aligns multimodal representations, the patch encoders may be encouraged to overlook the distinct attributes of their respective modality. Instead, they may focus only on features shared by all modalities, i.e. their common denominator. To ensure that encoders also capture modality-specific information, we incorporate a reconstruction objective, detailed in Sec. 3.3.

## 3.3 Multimodal Reconstruction Objective

During training, we mask a fraction of tokens $K \subset M \times P$ and replace their embeddings with a learned vector $f^{\text{mask}} \in \mathbb{R}^d$. Note that the masking can differ across modalities,
<figure><img src="image_6.png" /><figcaption>Figure 5: **Qualitative Results.** We report predictions of OmniSat and two competing models on tiles from our datasets. OmniSat can detect classes with recognizable temporal dynamics such as agricultural lands or mixed forest areas with both coniferous 🅿 and deciduous trees 👷. Other methods, and in particular ScaleMAE, struggle to detect these classes.</figcaption></figure>

add Sentinel-1 and Sentinel-2 time series to each tile, spanning the closest available year to the VHR observation for Sentinel-2. Note that due to the weather patterns and position of the area of interest with respect to Sentinel-2's orbit, the optical time series is particularly irregular and occluded, with up to 50% of acquisitions being non-exploitable. Despite this challenge, we included the raw observations without pre-processing, whereas TreeSatAI's single-date images have been manually selected.

**PASTIS-HD:** The PASTIS dataset [36], is designed for semantic and panoptic segmentation of agricultural parcels using Sentinel-2 time series and covers 18 crop types across 2433 image time series with dimensions of 1280 × 1280m. Each series contains between 38 and 61 observations with 10 spectral bands. PASTIS-R [37] adds the corresponding Sentinel-1 radar time series. We only used the ascendent time series of Sentinel-1 for our training and evaluation, for a total of 169,587 radar images with three bands.

To enhance the spatial resolution and utility of PASTIS, we introduce PASTIS-HD<fnref n="2" />, which integrates contemporaneous VHR satellite images (SPOT 6-7 [100]). We apply orthorectification and pansharpening, resample the resulting images to a 1m resolution, and finally convert them to 8 bits.

We follow the protocol of Irvin et al. [5] to use the dense annotations for a multi-label classification task: each patch is associated with the labels of all its pixels. This conversion allows us to evaluate all methods in the same setting and configuration as TreeSatAI.

**FLAIR.** The FLAIR dataset [48] combines VHR aerial images with time series data. It comprises 77,762 aerial tiles (512 × 512 pixels, 0.2m resolution) with five channels (RGB, near-infrared, and a normalized digital surface model) taken in France, alongside corresponding Sentinel-2 time series (10m resolution, 10 spectral bands, 20 to 114 observations per year). We apply the same processing as PASTIS to use the dense annotation for a multi-label classification task.

## 4.2 Experimental Setting

This section details our experimental protocol and our adaption of competing algorithms.

<footnote n="2">²This companion dataset can be found at https://zenodo.org/records/10908628.</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 4: Contrastive Loss.** We represent the token matching matrix for two tiles Tile1 and Tile2 viewed across 3 modalities $m_1$, $m_2$, and $m_3$. Tile1 is composed of the patches $p_1$ and $p_2$, while Tile2 comprises $q_1$ and $q_2$. In contrast to classic approaches which ignore the diagonal and assign each sample with a single positive match, our loss defines operates at the patch level, considers multiple positives per token, and excludes tokens in a block-diagonal fashion.</figcaption></figure>

<figure><img src="image_7.png" /></figure>

and some patches may be entirely masked. All tokens are then processed by the modality combining network $\mathcal{C}$, which outputs $P$ multimodal embeddings $f_{\mathbf{P}}^*$:

$$f_{\mathbf{P}}^* = \mathcal{C}(\{f_p^m\}_{(m,p) \notin \mathbf{K}} \cup \{f_p^{\text{mask}}\}_{(m,p) \in \mathbf{K}}). \quad (4)$$

To encourage the patch embeddings $f_{\mathbf{P}}^*$ to capture information from all modalities, we build a multimodal reconstruction objective. We denote by $\mathcal{D}^m : \mathbb{R}^d \to \Omega^m$ the dedicated decoder of each modality $m$ and write the reconstruction loss as:

$$\mathcal{L}_{\text{restr}} = \frac{1}{|\mathbf{K}|} \sum_{(m,p) \in \mathbf{K}} \frac{1}{\text{dim}(\Omega^m)} \|\mathcal{D}^m(f_p^*) - x_p^m\|^2, \quad (5)$$

with $\text{dim}(\Omega^m)$ the dimension of $\Omega^m$. The total loss is the sum of $\mathcal{L}_{\text{restr}}$ and $\mathcal{L}_{\text{contrast}}$.

## 3.4 Implementation Details

We detail here the specific parameters chosen in all our experiments.

**Tokenization.** We split each tile along a regular spatial grid to produce a set of non-overlapping patches $P$ consistent across all modalities. For TreeSat and FLAIR, we use a 10 × 10m grid, meaning that the VHR input tokens are small image patches of size 50 × 50 with 0.2m per pixel. The patches of Sentinel observations with a resolution of 10m are single-pixel temporal sequences of spectral measurements. For PASTIS-HD, we use a 40 × 40m grid, meaning that the VHR patches are of size 40 × 40 with 1.0m per pixel. The patches of Sentinel observations [26] are 4 × 4 image time series which we spatially flatten before encoding.

**Hyperparameters.** To show the versatility of OmniSat, we use the same configuration throughout all experiments. The embedding size is $d = 256$, resulting in image encoders and decoders with 3.6M and 1.8M parameters, 403K and 96K for optical time series, and 402K and 95K for radar time series. The modality combiner module is composed of $B = 6$ residual self-attention blocks and a single cross-attention block, for a total of 3.6M parameters. We train our model on 3 A6000 GPUs with a batch size of 128
<table><caption>Table 2: **Performance on TreeSatAI-TS.** We report the weighted F1 for multi-label tree species classification on TreeSatAI (TSAI) and our extended TreeSatAI-TS (TSAI-TS) dataset when fine-tuning with 10% and 100% of training labels. The first line of the table is the modality used for evaluation. We distinguish methods that are **best for one modality** within a dataset, **best in a dataset** across all modalities, and the **best overall** performance. *: late feature fusion with a ResNet. †: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-training</th><th colspan="2">All</th><th colspan="2">Sentinel-1</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR Image</th></tr><tr><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td colspan="10" style="text-align: center;">Evaluated on TreeSatAI: single date for Sentinel-1 and Sentinel-2</td></tr><tr><td>MLP[35]</td><td>ImageNet</td><td>42.6*</td><td><b>71.5*</b></td><td>3.4</td><td>10.1</td><td>22.1</td><td>52.0</td><td>-</td><td>-</td></tr><tr><td>ResNet [35]</td><td>ImageNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><b>58.8</b></td><td><b>70.1</b></td></tr><tr><td>LightGBM [35]</td><td>ImageNet</td><td>-</td><td>54.3*</td><td>-</td><td>11.9</td><td>-</td><td>48.2</td><td>-</td><td>44.0</td></tr><tr><td>† PSE [101]</td><td>None</td><td>47.2*</td><td>68.1*</td><td><b>11.5</b></td><td>14.6</td><td><b>48.5</b></td><td><b>58.3</b></td><td>-</td><td>-</td></tr><tr><td>† ViT [64]</td><td>None</td><td>42.7</td><td>57.1</td><td>8.7</td><td>17.5</td><td>39.8</td><td>57.3</td><td>36.7</td><td>51.7</td></tr><tr><td>PRESTO [85]</td><td>PRESTO</td><td>-</td><td>-</td><td>-</td><td><b>19.8</b></td><td>-</td><td>46.3</td><td>-</td><td>-</td></tr><tr><td>MOSAIKS [102, 103]</td><td>TSAI</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>56.0</td><td>-</td><td>-</td></tr><tr><td>† CROMA [4]</td><td>TSAI</td><td>49.6</td><td>61.0</td><td>10.1</td><td>12.7</td><td>47.8</td><td>55.7</td><td>-</td><td>-</td></tr><tr><td>† SatMAE [70]</td><td>TSAI</td><td>46.1</td><td>61.5</td><td>-</td><td>-</td><td>40.3</td><td>49.7</td><td>44.1</td><td>61.4</td></tr><tr><td>† ScaleMAE [72]</td><td>TSAI</td><td>47.6</td><td>62.5</td><td>-</td><td>-</td><td>46.7</td><td>55.2</td><td>46.9</td><td>63.6</td></tr><tr><td><b>OmniSAT (ours)</b></td><td>TSAI</td><td>56.2</td><td>70.4</td><td>5.3</td><td>6.4</td><td><b>48.4</b></td><td>57.1</td><td>52.8</td><td>68.9</td></tr><tr><td colspan="10" style="text-align: center;">Evaluated on TreeSatAI-TS: Sentinel-1 and Sentinel-2 time series spanning one year</td></tr><tr><td>† UT&T [48]</td><td>ImageNet</td><td>43.8</td><td>56.7</td><td>42.3</td><td>55.2</td><td>41.5</td><td>57.0</td><td>44.3</td><td>55.9</td></tr><tr><td>† Scale-MAE [72]</td><td>TSAI-TS</td><td>44.1</td><td>60.4</td><td>-</td><td>-</td><td>11.0</td><td>31.5</td><td>46.9</td><td>63.6</td></tr><tr><td>† PSE+LTAE [101]</td><td>None</td><td>59.4*</td><td>71.2*</td><td>42.6</td><td>52.4</td><td>44.0</td><td>57.2</td><td>-</td><td>-</td></tr><tr><td><b>OmniSAT (ours)</b></td><td>None</td><td>52.2</td><td>73.3</td><td>31.6</td><td>55.9</td><td>33.9</td><td>49.7</td><td>51.4</td><td><b>71.0</b></td></tr><tr><td><b>OmniSAT (ours)</b></td><td>TSAI-TS</td><td><b>61.1</b></td><td><b>74.2</b></td><td><b>48.2</b></td><td><b>56.7</b></td><td><b>51.4</b></td><td><b>62.9</b></td><td><b>58.3</b></td><td>70.5</td></tr></tbody></table>

**Evaluation Protocol.** All experiments follow a similar setting:

* **Pre-training (optional).** Methods that support self-supervised pre-training (OmniSat, Sat-MAE [70], ScaleMAE [72], CROMA [4]) are pre-trained for up to 250 epochs on the entire training set without access to labels.

* **Fine-Tuning.** We propose two settings for fine-tuning:

* **Fully Supervised Fine-Tuning.** We train the resulting models using all the labels in the training set.

* **Semi-Supervised Fine-Tuning.** We use a portion of 10% or 20% of the training set, stratified by the distribution of classes, to fine-tune the models. For models without pre-training, this corresponds to supervision in the low-data regime.

* **Monomodal and Multimodal Evaluation.** We evaluate all methods using each available modality independently and combining all supported modalities.

**Adapting Competing Approaches.** We report the performance of several methods taken from the literature on our considered datasets: LightGBM [35], PRESTO [85], and MOSAIKS [103]. However, few existing methods can operate on single- and multi-date data at the same time. To ensure a fair evaluation of competing approaches, we modify various state-of-the-art models to handle a broader combination of modalities. We performed multiple tests for each approach and kept the configurations leading to their highest performance.

* **Multimodality.** We train methods that are not natively multimodal (PSE [101], ViT [64], SatMAE, ScaleMAE) using a late-fusion scheme [104] by concatenating the embeddings learned in each modality, as suggested by Ahlswede et al. [35]. For UT&T [48], initially designed for VHR images and Sentinel-2 time series, we add a branch for Sentinel-1 integration, which is identical to the Sentinel-2 branch except for the first layer.

* **Handling Temporal Data.** To evaluate image models (SatMAE, ScaleMAE, CROMA) on time series, we convert image sequences to single images by concatenating for each pixel
multimodal tiles per GPU and set the contrastive temperature $\gamma$ to 0.1. We train our model with the ADAM optimizer [54], with a learning rate of $10^{-4}$ for pretraining and $2 \times 10^{-5}$ for fine-tuning, and a ReduceLROnPlateau scheduler [1] with a patience of 10 epochs and a decay rate of 0.1. When re-implementing competing methods, we use the hyperparameters of their open-source repository.

# 4 Experiments

We evaluate OmniSat's performance across three multimodal datasets, including two new datasets introduced in this work and presented in Sec. 4.1. We outline our experimental protocol and our adaptation of competing methods in Sec. 4.2. We then present our quantitative results and analysis in Sec. 4.3 and conduct an ablation study to in Sec. 4.4.

## 4.1 Datasets

We consider three multimodal datasets: FLAIR [31], and the augmented TreeSatAI-TS [2] and PASTIS-HD [33, 34]. See Fig. 1 for an illustration of these two last datasets.

**TreeSatAI-TS:** TreeSatAI [2] is a multimodal dataset for tree species identification, containing 50,381 tiles of 60 × 60 m with multi-label annotations for 20 classes and all taken in Germany. Each tile is associated with a very high resolution RGB and near-infrared (NIR) image (0.2 m pixel resolution), a single Sentinel-2 multi-spectral image (10m per pixel resolution, 10 bands), and a single Sentinel-1 radar image (10m per pixel resolution, 3 bands: two polarization channels and their ratio).

Motivated by the fact that fine-grained vegetation discrimination relies heavily on temporal dynamics information [93], we introduce TreeSatAI-TS<fnref n="1" />. This extended version uses open-source data to add Sentinel-1 and Sentinel-2 time series to each tile, spanning the closest available year to the VHR observation for Sentinel-2. Note that due to the weather patterns and position of the area of interest with respect to Sentinel-2's orbit, the optical time series is particularly irregular and occluded, with up to 50% of acquisitions being non-exploitable. Despite this challenge, we included the raw observations without pre-processing, whereas TreeSatAI's single-date images have been manually selected.

**PASTIS-HD:** The PASTIS dataset [33], is designed for semantic and panoptic segmentation of agricultural parcels using Sentinel-2 time series and covers 18 crop types across 2433 image time series with dimensions of 1280 × 1280m. Each series contains between 38 and 61 observations with 10 spectral bands. PASTIS-R [34] adds the corresponding Sentinel-1 radar time series. We only used the ascending time series of Sentinel-1 for our training and evaluation, for a total of 169,587 radar images with three bands.

To enhance the spatial resolution and utility of PASTIS, we introduce PASTIS-HD², which integrates contemporary VHR satellite images (SPOT 6-7 [24]). We apply orthorectification and pansharpening, resample the resulting images to a 1m resolution, and finally convert them to 8 bits. We follow the protocol of Irvin et al. [52] to use the dense annotations for a multi-label classification task: each patch is associated with the

<footnote n="1">¹The dataset is available at https://huggingface.co/datasets/IGNF/TreeSatAI-Time-Series.
²The dataset is available at https://huggingface.co/datasets/IGNF/PASTIS-HD.</footnote>
<table><caption>Table 3: **Performance on PASTIS-HD.** We report the macro-averaged F1-score for crop-type multi-class classification on the PASTIS-HD dataset. We distinguish methods that are **best for one modality**, **best in a dataset** across all modalities. *: late feature fusion with a ResNet. †: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-trained</th><th colspan="2">All</th><th colspan="2">Sentinel-1</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR image</th></tr><tr><th>20%</th><th>100%</th><th>20%</th><th>100%</th><th>20%</th><th>100%</th><th>20%</th><th>100%</th></tr></thead><tbody><tr><td>† ResNet50 [106]</td><td>ImageNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>57.6</strong></td><td><strong>59.3</strong></td></tr><tr><td>† Scale-MAE [72]</td><td>PASTIS-HD</td><td>42.0</td><td>42.2</td><td>-</td><td>-</td><td>41.2</td><td>46.1</td><td>48.8</td><td>51.9</td></tr><tr><td>† UTAE [36, 37]</td><td>No</td><td>36.8*</td><td>46.9*</td><td>20.1</td><td>40.7</td><td>32.7</td><td>37.6</td><td>-</td><td>-</td></tr><tr><td>† UT&T [48]</td><td>ImageNet</td><td>54.2</td><td>53.5</td><td>58.8</td><td>62.8</td><td>54.9</td><td>61.3</td><td>51.1</td><td>49.8</td></tr><tr><td>† CROMA [4]</td><td>PASTIS-HD</td><td>57.5</td><td>60.1</td><td>55.3</td><td>56.1</td><td>53.0</td><td>56.7</td><td>-</td><td>-</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>No</td><td>42.0</td><td>59.1</td><td>58.2</td><td>60.2</td><td>51.7</td><td>60.1</td><td>47.3</td><td>52.8</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>PASTIS-HD</td><td><u>62.6</u></td><td><u>69.9</u></td><td><u>60.8</u></td><td><u>69.0</u></td><td><u>61.8</u></td><td><u>70.8</u></td><td>54.6</td><td><u>59.3</u></td></tr></tbody></table>

<table><caption>Table 4: **Performance on FLAIR.** We report the macro-averaged F1-score for land cover multi-class classification on the FLAIR dataset. We distinguish methods that are **best for one modality** and **best in a dataset**. †: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-trained</th><th colspan="2">All</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR Image</th></tr><tr><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td>† UT&T [48]</td><td>ImageNet</td><td>44.2</td><td>48.8</td><td>57.4</td><td>62.0</td><td>58.9</td><td>65.5</td></tr><tr><td>† ScaleMAE[72]</td><td>FLAIR</td><td><strong>63.1</strong></td><td>70.0</td><td>52.5</td><td>61.0</td><td>61.2</td><td>67.3</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>No</td><td>62.5</td><td>70.0</td><td>56.1</td><td><strong>65.4</strong></td><td>64.7</td><td>71.5</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>FLAIR</td><td>60.6</td><td><strong>73.4</strong></td><td><strong>56.8</strong></td><td><strong>65.4</strong></td><td><strong>65.2</strong></td><td><strong>71.6</strong></td></tr></tbody></table>

and channel channel-wise the median observation for the four seasons: spring, summer, fall, and winter [105].

* **Handling VHR Data.** To evaluate methods designed for low-resolution images (PSE, LTAE [94]) in a multimodal setting that includes VHR images, we concatenate the representations obtained with a ResNet network to their final embedding.

* **Scaling Models.** Our considered datasets are smaller than the ones used to train large ViT-based models, making them prone to overfitting. We address this issue by selecting a ViT-Small [64] backbone instead of a ViT-Large for SatMAE, ScaleMAE and CROMA.

* **Multi-Class Prediction.** To evaluate ViT-based models on classification experiments, we insert a linear layer that maps the embedding of the class token (CLS) to a vector of class scores. For the UT&T model, we compute a spatial average of the last feature map, followed by a similar linear projection.

## 4.3 Numerical Experiments and Analysis

In this section, we report our model's performance and efficiency compared to other approaches across the considered datasets and propose our analysis.

**TreeSatAI-TS.** Table 2 presents the performance of different models on TreeSatAI and TreeSatAI-TS. We report several key observations:

* **Benefit of Time Series.** For the original TreeSatAI dataset with single-date Sentinel-1/2 observations, none of the pre-training schemes significantly improve performance beyond simple baselines such as ResNet, PSE, or MLP, even in a semi-supervised setting. In particular, single-date S1 observations yield low performance for all methods (below 20 F1-score), emphasizing the need to use the entire time series.

OmniSat exhibits significantly improved results on TreeSatAI-TS, with or without pre-training. In contrast, Scale-MAE struggles to extract meaningful dynamic features from the highly occluded time series, and CROMA would not converge. These models perform
labels of all of its pixels. This conversion allows us to evaluate all methods in the same
setting and configuration as TreeSatAI.

**FLAIR.** The FLAIR dataset [31] combines VHR aerial images with time series data. It comprises 77,762 aerial tiles (512 × 512 pixels, 0.2 m resolution) with five channels (RGB, near-infrared, and a normalized digital surface model) taken in France, alongside corresponding Sentinel-2 time series (10 m resolution, 10 spectral bands, 20 to 114 observations per year). We apply the same processing as PASTIS to use the dense annotation for a multi-label classification task.

## 4.2 Experimental Setting

This section details our experimental protocol and our adaption of competing algorithms.
**Evaluation Protocol.** All experiments follow a similar setting:

- **Pre-training (optional).** Methods that support self-supervised pre-training (OmniSat, SatMAE [20], ScaleMAE [76], CROMA [29]) are pre-trained for up to 250 epochs on the entire training set without access to labels.

- **Fine-Tuning.** We propose two settings for fine-tuning:

* **Fully Supervised Fine-Tuning.** We train the resulting models using all the labels in the training set.

* **Semi-Supervised Fine-Tuning.** We use a portion of 10% or 20% of the training set, stratified by the distribution of classes, to fine-tune the models. For models without pre-training, this corresponds to supervision in the low-data regime.

- **Unimodal and Multimodal Evaluation.** We evaluate all methods using each available modality independently and combining all supported modalities.

**Adapting Competing Approaches.** We report the performance of several methods taken from the literature on our considered datasets: LightGBM [2], PRESTO [90], and MOSAIKS [79]. However, few existing methods can operate on single- and multi-date data at the same time. To ensure a fair evaluation of competing approaches, we modify various state-of-the-art models to handle a broader combination of modalities. We provide details on these changes in the appendix.

## 4.3 Numerical Experiments and Analysis

In this section, we report our model's performance and efficiency compared to other approaches across the considered datasets and propose our analysis.

**TreeSatAI-TS.** Tab. 2 presents the performance of different models on TreeSatAI and TreeSatAI-TS. We report several key observations:

- **Benefit of Time Series.** For the original TreeSatAI dataset with single-date Sentinel-1/2 observations, none of the pre-training schemes significantly improve performance beyond simple baselines such as ResNet, PSE, or MLP, even in a semi-supervised setting. In particular, single-date S1 observations yield low performance for all methods (below 20 F1-score), emphasizing the need to use the entire time series. OmniSat exhibits significantly improved results on TreeSatAI-TS, with or without pretraining. Image models struggles to extract meaningful features temporally aggregated temporal observations, while OmniSat learn rich dynamic features.
<figure><img src="image_7.png" /><figcaption>Figure 6: **Efficiency.** We report the best performance of different models between TreeSatAI and TreeSatAI-TS, with pre-training and fine-tuning using 100% of labels. The area of the markers is proportional to the training time, broken down in pre-training and fine-tuning when applicable.</figcaption></figure>

better with the manually chosen single-date images of TreeSatAI than with temporally ag-
gregated temporal observations, whereas OmniSat can leverage dynamic features.

*   **Benefits of Multimodality.** When using all modalities, OmniSat outperforms all competing methods by a margin of 3% F1-score. The multimodal performance of OmniSat and CROMA, which learn to combine data sources, is strictly superior to the F1-score of their best modality by 3.7% and 5.3% points, respectively. Conversely, the performance of methods that rely on late-fusion (SatMAE, ScaleMAE, ViT) is comparable to their best modality. This demonstrates the value of learning to combine information from different sources end-to-end.
*   **Benefits of Cross-Modal Pre-Training.** With access to all modalities, our self-supervised pre-training improves by 0.9% point the F1-score of the model fine-tuned on 100% of labels, compared to not pre-training, and 8.9% when using only 10% of labels. This shows that our pre-training leads to more expressive multimodal features. Interestingly, when performing inference with Sentinel-2 time series alone, the performance increase linked to the pre-training becomes 13.2% with 100% labels and 17.5% with 10%. This illustrates that our pre-training scheme also improves the features learned by each encoder despite only relying on spatial matching.

**Experiments on PASTIS-HD.** The analysis of the performance of various models on PASTIS-HD is reported in Table 3, and is consistent with the ones of TreeSatAI-TS. First, by learning to combine all modalities despite their different resolutions, OmniSat achieves state-of-the-art results on this benchmark. Second, our cross-modal pertaining significantly improves OmniSat’s performance in the multimodal (+10.8 pF1-score with 100% of training label) and all single-modality settings (8.8 points for Sentinel-1, 10.7 for Sentinel-2, and 6.5 for the VHR images).

**Experiments on FLAIR.** We report in Table 4 the results on the bimodal FLAIR dataset for multilabel classification. OmniSat outperforms the much larger ScaleMAE [72] and UT&T [48] models with 100% of labels and both modalities by 3.4%. Our pre-training scheme had a smaller impact than for the TreeSatAI-TS experiment, which may be attributed to the fact that only two modalities are available, which decreases the supervisory power of our cross-modal contrastive objective and our multimodal reconstruction loss. This highlights a limitation of OmniSat: the model needs
<table><caption>**Table 2: Performance on TreeSatAI-TS.** We report the weighted F1 for multi-label tree species classification on TreeSatAI (TSAI) and our extended TreeSatAI-TS (TSAI-TS) dataset when fine-tuning with 10% and 100% of training labels. The first line of the table is the modality used for evaluation. We distinguish methods that are **best for one modality** within a dataset, **best in a dataset** across all modalities, and the **best overall** performance. *: late feature fusion with a ResNet pretrained on ImageNet. ⊕: Foundation model trained on extensive external data.
†: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-training</th><th colspan="2">All</th><th colspan="2">Sentinel-1</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR Image</th></tr><tr><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td colspan="10" style="text-align: center;">Evaluated on TreeSatAI: single date for Sentinel-1 and Sentinel-2</td></tr><tr><td>† PSE [35]</td><td>None</td><td>47.2*</td><td>68.1*</td><td>11.5</td><td>14.6</td><td><strong>48.5</strong></td><td><strong>58.3</strong></td><td>-</td><td>-</td></tr><tr><td>† ViT [25]</td><td>None</td><td>42.7</td><td>57.1</td><td>8.7</td><td>17.5</td><td>39.8</td><td>57.3</td><td>36.7</td><td>51.7</td></tr><tr><td>MLP [2]</td><td>None</td><td>42.6*</td><td>71.5*</td><td>3.4</td><td>10.1</td><td>22.1</td><td>52.0</td><td>-</td><td>-</td></tr><tr><td>ResNet [2]</td><td>ImageNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>58.8</td><td><strong>70.1</strong></td></tr><tr><td>LightGBM [2]</td><td>ImageNet</td><td>-</td><td>54.3*</td><td>-</td><td>11.9</td><td>-</td><td>48.2</td><td>-</td><td>44.0</td></tr><tr><td>PRESTO [90]</td><td>⊕</td><td>-</td><td>-</td><td>-</td><td><strong>19.8</strong></td><td>-</td><td>46.3</td><td>-</td><td>-</td></tr><tr><td>† DOFA [98]</td><td>⊕</td><td><strong>59.5</strong></td><td><strong>71.6</strong></td><td><strong>11.6</strong></td><td>19.3</td><td>48.2</td><td>57.0</td><td>51.6</td><td>67.5</td></tr><tr><td>MOSAIKS [22, 79] TSAI</td><td></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>56.0</td><td>-</td><td>-</td></tr><tr><td>† CROMA [29]</td><td>TSAI</td><td>49.6</td><td>61.0</td><td>10.1</td><td>12.7</td><td>47.8</td><td>55.7</td><td>-</td><td>-</td></tr><tr><td>† SatMAE [20]</td><td>TSAI</td><td>46.1</td><td>61.5</td><td>-</td><td>-</td><td>40.3</td><td>49.7</td><td>44.1</td><td>61.4</td></tr><tr><td>† ScaleMAE [76]</td><td>TSAI</td><td>47.6</td><td>62.5</td><td>-</td><td>-</td><td>46.7</td><td>55.2</td><td>46.9</td><td>63.6</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>TSAI</td><td>56.2</td><td>70.4</td><td>5.3</td><td>6.4</td><td>48.4</td><td>57.1</td><td>52.8</td><td>68.9</td></tr><tr><td colspan="10" style="text-align: center;">Evaluated on TreeSatAI-TS: Sentinel-1 and Sentinel-2 time series spanning one year</td></tr><tr><td>† PSE+LTAE [35]</td><td>None</td><td>59.4*</td><td>71.2*</td><td>42.6</td><td>52.4</td><td>44.0</td><td>57.2</td><td>-</td><td>-</td></tr><tr><td>† UT&T [31]</td><td>ImageNet</td><td>43.8</td><td>56.7</td><td>42.3</td><td>55.2</td><td>41.5</td><td>57.0</td><td>44.3</td><td>55.9</td></tr><tr><td>† DOFA [98]</td><td>⊕</td><td>41.8</td><td>71.3</td><td>0.0</td><td>0.0</td><td>25.0</td><td>39.4</td><td>51.6</td><td>67.5</td></tr><tr><td>† Scale-MAE [76]</td><td>TSAI-TS</td><td>44.1</td><td>60.4</td><td>-</td><td>-</td><td>11.0</td><td>31.5</td><td>46.9</td><td>63.6</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>None</td><td>52.2</td><td>73.3</td><td>31.6</td><td>55.9</td><td>33.9</td><td>49.7</td><td>51.4</td><td><strong>71.0</strong></td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>TSAI-TS</td><td><strong>61.1</strong></td><td><strong>74.2</strong></td><td><strong>48.2</strong></td><td><strong>56.7</strong></td><td><strong>51.4</strong></td><td><strong>62.9</strong></td><td><strong>58.3</strong></td><td>70.5</td></tr></tbody></table>

The foundation model DOFA [98], with 111 million parameters and a large closed-source training set, outperforms all models when evaluated on single-date modalities. However, OmniSat reaches higher performances on TreeSatAI-TS with only 10 million parameters, which we attribute to its ability to leverage temporal modalities.

- **Benefits of Multimodality.** When using all modalities, OmniSat outperforms all competing methods by a margin of 3% F1-score. The multimodal performance of OmniSat and CROMA, which learn to combine data sources, is strictly superior to the F1-score of their best modality by 3.7% and 5.3% points, respectively. Conversely, the performance of methods that rely on late-fusion (SatMAE, ScaleMAE, ViT) is comparable to their best modality. This demonstrates the value of learning to combine information from different sources end-to-end.
<table><caption>Table 5: **Ablation Study.** We present the impact of several design choices on the TreeSatAI-TS dataset, measured in terms of macro-averaged F1-score.</caption><thead><tr><th>Experiment</th><th>10%</th><th>100%</th><th>Experiment</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td><strong>OmniSat</strong></td><td><strong>61.1</strong></td><td><strong>74.2</strong></td><td>b) no contrastive loss</td><td>55.6</td><td>73.4</td></tr><tr><td>a) no index bypass</td><td>57.5</td><td>73.5</td><td>b) naive contrastive loss</td><td>57.8</td><td>72.2</td></tr><tr><td>a) no date filtering</td><td>58.2</td><td>71.6</td><td>b) no reconstruction loss</td><td>59.0</td><td>72.2</td></tr></tbody></table>

to be pretrained on a modality-rich dataset to achieve its best performance. We also note the poor performance of UT&T, which we attribute to its semantic segmentation-driven design.

**Efficiency Evaluation.** We plot in Figure 6 the best performance between TreeSatAI and TreeSatAI-TS for different models according to their size and training time. OmniSat is more compact, faster to train, and performs better than all evaluated models. The highly-specialized combination of PSE, LTAE, and ResNet is a strong contender, outperforming significantly more complex models with generic encoding-decoding schemes.

## 4.4 Ablation Study

In this section, we report the results of several experiments evaluating the impact and validity of our main design choices, see Table 5.

**a) Encoder/Decoder Architecture.** We propose several improvements to the standard image encoder-decoder scheme used in computer vision to accommodate the specificities of EO data. In particular, passing the max-pool indices from the image patch encoder to its decoder allows the learned representation to focus on characterizing the spectral signature instead of fine-grained spatial information, and leads to a performance increase of 0.7% in the full supervision setting.

As clouds frequently obstruct optical time series, we use an unsupervised date-filtering scheme to reconstruct only meaningful acquisitions. This approach leads to a significant improvement of 3.6%, showcasing the benefit of developing modality-aware approaches for EO.

**b) Role of Loss Functions.** When training without contrastive loss, we observe a small decrease in performance of 0.8% in the fully supervised regime and a more pronounced drop of 5.5% in the semi-supervised regime. This demonstrates how harmonizing the encoding across encoders facilitates their subsequent fusion. Interestingly, when implementing a naive contrastive loss that considers negative examples within the same tile and modality, the decrease is greater than simply removing this loss (2% in full supervision). This strategy may introduce indistinguishable negative examples and perturb the learning process.

We also remove the reconstruction loss, meaning that only the encoders are learned contrastively during pre-training. This results in a drop of 2% F1-score point, illustrating the importance of pre-training the transformer *C* alongside its encoders.

**Limitations.** All datasets used in our experiments are based in Europe, primarily due to the availability of open-access annotations. This regional focus prevents us from evaluating our model's performance in tropical and developing countries, which present unique challenges in terms of label provision, heterogeneity, and complex classes.

A limitation of our pre-training scheme is its dependence on a sufficient number of aligned modalities, as illustrated by its moderate impact on the bimodal FLAIR dataset.

# 5 Conclusion

We introduced OmniSat, a new architecture for the self-supervised modality fusion of Earth Observation (EO) data from various sources. To facilitate its evaluation, we augmented two existing benchmarks to form the first open-access datasets with three distinct modalities of different na-tures and resolutions. We experimentally showed that leveraging diverse modalities with a flexible
<table><caption>**Table 3: Performance on PASTIS-HD.** We report the macro-averaged F1-score for crop-type multi-class classification on the PASTIS-HD dataset. We distinguish methods that are **best for one modality, best in a dataset** across all modalities. *: late feature fusion with a ResNet. †: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-trained</th><th colspan="2">All</th><th colspan="2">Sentinel-1</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR image</th></tr><tr><th>20%</th><th>100%</th><th>20%</th><th>100%</th><th>20%</th><th>100%</th><th>20%</th><th>100%</th></tr></thead><tbody><tr><td>† UTAE [33, 34]</td><td>None</td><td>36.8*</td><td>46.9*</td><td>20.1</td><td>40.7</td><td>32.7</td><td>37.6</td><td>-</td><td>-</td></tr><tr><td>† ResNet50 [47]</td><td>ImageNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td><strong>57.6</strong></td><td><strong>59.3</strong></td></tr><tr><td>† UT&T [31]</td><td>ImageNet</td><td>54.2</td><td>53.5</td><td>58.8</td><td>62.8</td><td>54.9</td><td>61.3</td><td>51.1</td><td>49.8</td></tr><tr><td>† DOFA [98]</td><td>●</td><td>53.7</td><td>55.7</td><td>36.7</td><td>41.5</td><td>50.8</td><td>53.4</td><td>47.9</td><td>54.8</td></tr><tr><td>† Scale-MAE [76]</td><td>PASTIS-HD</td><td>42.0</td><td>42.2</td><td>-</td><td>-</td><td>41.2</td><td>46.1</td><td>48.8</td><td>51.9</td></tr><tr><td>† CROMA [29]</td><td>PASTIS-HD</td><td>57.5</td><td>60.1</td><td>55.3</td><td>56.1</td><td>53.0</td><td>56.7</td><td>-</td><td>-</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>No</td><td>42.0</td><td>59.1</td><td>58.2</td><td>60.2</td><td>51.7</td><td>60.1</td><td>47.3</td><td>52.8</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>PASTIS-HD</td><td><strong>62.6</strong></td><td><strong>69.9</strong></td><td><strong>60.8</strong></td><td><strong>69.0</strong></td><td><strong>61.8</strong></td><td><strong>70.8</strong></td><td>54.6</td><td><strong>59.3</strong></td></tr></tbody></table>

<table><caption>**Table 4: Performance on FLAIR.** We report the macro-averaged F1-score for land cover multi-class classification on the FLAIR dataset. We distinguish methods that are **best for one modality** and **best in a dataset**. †: model evaluated on this dataset for the first time.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">pre-trained</th><th colspan="2">All</th><th colspan="2">Sentinel-2</th><th colspan="2">VHR Image</th></tr><tr><th>10%</th><th>100%</th><th>10%</th><th>100%</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td>† UT&T [31]</td><td>ImageNet</td><td>44.2</td><td>48.8</td><td><strong>57.4</strong></td><td>62.0</td><td>58.9</td><td>65.5</td></tr><tr><td>† DOFA [98]</td><td>●</td><td><strong>70.6</strong></td><td><strong>74.9</strong></td><td>57.0</td><td>61.0</td><td><strong>66.8</strong></td><td><strong>72.1</strong></td></tr><tr><td>† ScaleMAE [76]</td><td>FLAIR</td><td>63.1</td><td>70.0</td><td>52.5</td><td>61.0</td><td>61.2</td><td>67.3</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>No</td><td>62.5</td><td>70.0</td><td>56.1</td><td><strong>65.4</strong></td><td>64.7</td><td>71.5</td></tr><tr><td><strong>OmniSAT (ours)</strong></td><td>FLAIR</td><td>60.6</td><td>73.4</td><td>56.8</td><td><strong>65.4</strong></td><td>65.2</td><td>71.6</td></tr></tbody></table>

- **Benefits of Cross-Modal Pre-Training.** With access to all modalities, our self-supervised pre-training improves by 0.9% point the F1-score of the model fine-tuned on 100% of labels, compared to not pre-training, and 8.9% when using only 10% of labels. This shows that our pre-training leads to more expressive multimodal features. Interestingly, when performing inference with Sentinel-2 time series alone, the performance increase linked to the pre-training becomes 13.2% with 100% labels and 17.5% with 10%. This illustrates that our self-supervised pre-training scheme improves the features learned by each encoder despite not relying on annotated data.

**Experiments on PASTIS-HD.** The analysis of the performance of various models on PASTIS-HD is reported in Tab. 3, and is consistent with the ones of TreeSatAI-TS. First, by learning to combine all modalities despite their different resolutions, OmniSat achieves state-of-the-art results on this benchmark. Second, our cross-modal pretraining significantly improves OmniSat's performance in the multimodal (+10.8 pF1-score with
model improves the model’s performance in both fully and semi-supervised settings. Moreover, our training scheme can exploit the spatial alignment of multiple modalities to improve our model’s monomodal performance. Finally, we proposed several improvements to leverage the unique structure of EO data in the architecture of our model, such as automatic date filtering for reconstructing time series and index bypass in image patch decoders. We hope that our promising results and new datasets will encourage the computer vision community to consider EO data as a playing field for evaluating and developing novel self-supervised multimodal algorithms.

**Acknowledgements.** This work was supported by ANR project READY3D ANR-19-CE23-0007, and was granted access to the HPC resources of IDRIS under the allocation AD011014719 made by GENCI. We thank Anatol Garioud and Sébastien Giordano for their help on the creation of TreeSatAI-TS and PASTIS-HD datasets. The SPOT images are opendata thanks to the Dataterra Dinamis initiative in the case of the "Couverture France DINAMIS" program. We thank Jordi Inglada for inspiring discussions and valuable feedback.

## References

[1] Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey. *arXiv preprint arXiv:2304.01008*, 2023. 1

[2] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In *CVPR*, 2023. 1, 3

[3] Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. *TMLR*, 2023. 1, 3

[4] Anthony Fuller, Koreen Millard, and James R Green. CROMA: Remote sensing representations with contrastive radar-optical masked autoencoders. In *NeurIPS*, 2023. 1, 3, 9, 10

[5] Jeremy Irvin, Lucas Tao, Joanne Zhou, Yuntao Ma, Langston Nashold, Benjamin Liu, and Andrew Y Ng. USat: A unified self-supervised encoder for multi-sensor satellite imagery. *arXiv preprint arXiv:2312.02199*, 2023. 1, 3, 8

[6] Pedram Ghamisi, Behnood Rasti, Naoto Yokoya, Qunming Wang, Bernhard Hofle, Lorenzo Bruzzone, Francesca Bovolo, Mingmin Chi, Katharina Anders, Richard Gloaguen, et al. Multisource and multitemporal data fusion in remote sensing: A comprehensive review of the state of the art. *IEEE Geoscience and Remote Sensing Magazine*, 2019. 1, 3

[7] Michael Schmitt and Xiao Xiang Zhu. Data fusion and remote sensing: An ever-growing relationship. *IEEE Geoscience and Remote Sensing Magazine*, 2016. 1, 3

[8] Pol Coppin, Eric Lambin, Inge Jonckheere, and Bart Muys. Digital change detection methods in natural ecosystem monitoring: A review. *Analysis of multi-temporal remote sensing images*, 2002. 1

[9] Cristina Secades, Brian O'Connor, Claire Brown, Matt Walpole, et al. Earth observation for biodiversity monitoring: A review of current approaches and future opportunities for tracking progress towards the aichi biodiversity targets. *CBD technical series*, 2014. 1

[10] Andrew K Skidmore, Nicholas C Coops, Elnaz Neinavaz, Abebe Ali, Michael E Schaepman, Marc Paganini, W Daniel Kissling, Petteri Vihervaara, Roshanak Darvishzadeh, Hannes Feilhauer, et al. Priority list of biodiversity metrics to observe from space. *Nature Ecology & Evolution*, 2021. 1

[11] Jun Yang, Peng Gong, Rong Fu, Minghua Zhang, Jingming Chen, Shunlin Liang, Bing Xu, Jiancheng Shi, and Robert Dickinson. The role of satellite remote sensing in climate change studies. *Nature climate change*, 2013. 1

[12] Alexandre Lacoste, Evan David Sherwin, Hannah Kerner, Hamed Alemohammad, Björn Lütjens, Jeremy Irvin, David Dao, Alex Chang, Mehmet Gunturkun, Alexandre Drouin, et al.
<figure><img src="image_8.png" /><figcaption>**Fig. 5: Efficiency.** We report the best performance of different models between TreeSatAI and TreeSatAI-TS, with pre-training and fine-tuning using 100% of labels. The area of the markers is proportional to the training time, broken down in pre-training and fine-tuning when applicable.</figcaption></figure>

100% of training label) and all single-modality settings (8.8 points for Sentinel-1, 10.7 for Sentinel-2, and 6.5 for the VHR images).

**Experiments on FLAIR.** We report in Tab. 4 the results on the bimodal FLAIR dataset for multilabel classification. OmniSat outperforms the much larger ScaleMAE [76] and UT&T [31] models with 100% of labels and both modalities by 3.4%. Our pre-training scheme had a smaller impact than for the TreeSatAI-TS experiment. We attribute this to the fact that only two modalities are available, which decreases the supervisory power of our cross-modal contrastive objective and our multimodal reconstruction loss. This highlights a limitation of OmniSat: the model needs to be pretrained on a modality-rich dataset to achieve its best performance.

**Efficiency Evaluation.** We plot in Fig. 5 the best performance between TreeSatAI and TreeSatAI-TS for different models according to their size and training time. OmniSat is more compact, faster to train, and performs better than all evaluated models, including the DOFA foundation model. The highly-specialized combination of PSE, LTAE, and ResNet is a strong contender, outperforming significantly larger models with generic encoding-decoding schemes.

## 4.4 Ablation Study

In this section, we report the results of several experiments evaluating the impact and validity of our main design choices, see Tab. 5.

**a) Encoder/Decoder Architecture.** We propose several improvements to the standard image encoder-decoder scheme used in computer vision to accommodate the specificities of EO data. In particular, passing the max-pool indices from the image patch encoder to its decoder allows the learned representation to focus on characterizing the spectral signature instead of fine-grained spatial information, and leads to a performance increase of 0.7% in the full supervision setting.
Toward foundation models for Earth monitoring: Proposal for a climate change benchmark. *arXiv preprint arXiv:2112.00570*, 2021. 1

[13] Catherine Nakalembe. Urgent and critical need for sub-saharan african countries to invest in Earth observation-based agricultural early warning and monitoring systems. *Environmental Research Letters*, 2020. 1

[14] Monika Kuffer, Dana R Thomson, Gianluca Boo, Ron Mahabir, Taïs Grippa, Sabine Vanhuysse, Ryan Engstrom, Robert Ndugwa, Jack Makau, Edith Darin, et al. The role of Earth observation in an integrated deprived area mapping “system” for low-to-middle income countries. *Remote sensing*, 2020. 1

[15] Katherine Anderson, Barbara Ryan, William Sonntag, Argyro Kavvada, and Lawrence Friedl. Earth observation in service of the 2030 agenda for sustainable development. *Geo-spatial Information Science*, 2017. 1

[16] Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. *arXiv preprint arXiv:2304.06798*, 2023. 1

[17] DeRen Li, QingXi Tong, RongXing Li, JianYa Gong, and LiangPei Zhang. Current issues in high-resolution Earth observation technology. *Science China Earth sciences*, 2012. 1

[18] Salvatore Manfreda, Matthew F McCabe, Pauline E Miller, Richard Lucas, Victor Pa-juelo Madrigal, Giorgos Mallinis, Eyal Ben Dor, David Helman, Lyndon Estes, Giuseppe Ciraolo, et al. On the use of unmanned aerial systems for environmental monitoring. *Remote sensing*, page 641, 2018. 1

[19] Matthias Drusch, Umberto Del Bello, Stefane Carlier, Olivier Colin, Valérie Fernandez, Fer-ran Gascon, Bianca Hoersch, Claudia Isola, Paolo Laberinti, Philippe Martinort, Aimé Mey-gret, François Spoto, Omar Sy, Franco Marchese, and Pier Bargellini. Sentinel-2: ESA's optical high-resolution mission for GMES operational services. *Remote Sensing of Environ-ment*, 2012. 1, 4, 6

[20] Alberto Moreira, Pau Prats-Iraola, Marwan Younis, Gerhard Krieger, Irena Hajnsek, and Kon-stantinos P Papathanassiou. A tutorial on synthetic aperture radar. *IEEE Geoscience and Remote Sensing Magazine*, 2013. 1

[21] Donato Amitrano, Gerardo Di Martino, Raffaella Guida, Pasquale Iervolino, Antonio Iodice, Maria Nicolina Papa, Daniele Riccio, and Giuseppe Ruello. Earth environmental monitor-ing using multi-temporal synthetic aperture radar: A critical review of selected applications. *Remote Sensing*, 2021. 1

[22] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In *CVPR*, 2022. 1, 3

[23] Siddharth Srivastava and Gaurav Sharma. OmniVec: Learning robust representations with cross modal sharing. In *WACV*, 2024. 1, 3

[24] Adrià Recasens, Jason Lin, João Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, et al. Zorro: The masked multi-modal transformer. *arXiv preprint arXiv:2301.09595*, 2023. 1

[25] Connor Greenwell, Jon Crall, Matthew Purri, Kristin Dana, Nathan Jacobs, Armin Hadzic, Scott Workman, and Matt Leotta. WATCH: Wide-area terrestrial change hypercube. In *WACV*, 2024. 1

[26] Paola Benedetti, Dino Ienco, Raffaele Gaetano, Kenji Ose, Ruggero G Pensa, and Stephane Dupuy. M³-fusion: A deep learning architecture for multiscale multimodal multitemporal satellite data fusion. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 2018. 1
<table><caption>**Table 5: Ablation Study.** We present the impact of several design choices on the TreeSatAI-TS dataset, measured in terms of macro-averaged F1-score.</caption><thead><tr><th>Experiment</th><th>10%</th><th>100%</th><th>Experiment</th><th>10%</th><th>100%</th></tr></thead><tbody><tr><td><strong>OmniSat</strong></td><td><strong>61.1</strong></td><td><strong>74.2</strong></td><td>b) no contrastive loss</td><td>55.6</td><td>73.4</td></tr><tr><td>a) no index bypass</td><td>57.5</td><td>73.5</td><td>b) naive contrastive loss</td><td>57.8</td><td>72.2</td></tr><tr><td>a) no date filtering</td><td>58.2</td><td>71.6</td><td>b) no reconstruction loss</td><td>59.0</td><td>72.2</td></tr></tbody></table>

As clouds frequently obstruct optical time series, we use an unsupervised date-filtering scheme to reconstruct only meaningful acquisitions. This approach leads to a significant improvement of 3.6%, showcasing the benefit of developing modality-aware approaches for EO.

**b) Role of Loss Functions.** When training without contrastive loss, we observe a decrease in performance of 0.8% in the fully supervised regime, and a more pronounced drop of 5.5% in the semi-supervised regime. This demonstrates how learning consistent encoding across encoders facilitates their subsequent fusion. Interestingly, when implementing a naive contrastive loss that considers all negative examples from the batch, the decrease is greater than simply removing this loss (2% in full supervision). This strategy may introduce indistinguishable negative examples and perturb the learning process.

We also remove the reconstruction loss, meaning that only the encoders are learned contrastively during pre-training. This results in a drop of 2% F1-score point, illustrating the importance of pre-training the transformer *C* alongside its encoders.

**Limitations.** All datasets used in our experiments are based in Europe, primarily due to the availability of open-access annotations. This regional focus prevents us from evaluating our model's performance in tropical and developing countries, which present unique challenges in terms of label provision, heterogeneity, and complex classes.

A limitation of our pre-training scheme is its dependence on a sufficient number of aligned modalities, as illustrated by its moderate impact on the bimodal FLAIR dataset.

# 5 Conclusion

We introduced OmniSat, a new architecture for the self-supervised modality fusion of Earth Observation (EO) data from various sources. To facilitate its evaluation, we augmented two existing datasets with new modalities of different natures and resolutions. We experimentally showed that leveraging diverse modalities with a flexible model improves the model's performance in both fully and semi-supervised settings. Moreover, our training scheme can exploit the spatial alignment of multiple modalities to improve our model's unimodal performance. Finally, we proposed several improvements to leverage the unique structure of EO data in the architecture of our model, such as automatic date filtering for reconstructing time series. We hope that our promising results and new datasets will encourage the computer vision community to consider EO data as a playing field for evaluating and developing novel self-supervised multimodal algorithms.
[27] Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. *IEEE TPAMI*, 2022. 1

[28] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from RGBD images. In *ECCV*, 2012. 1

[29] Georg Krispel, Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Fuseseg: LiDAR point cloud segmentation fusing multi-modal data. In *WACV*, 2020. 1

[30] Damien Robert, Bruno Vallet, and Loic Landrieu. Learning multi-view aggregation in the wild for large-scale 3D semantic segmentation. In *CVPR*, 2022. 1

[31] Angela Dai and Matthias Nießner. 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation. In *ECCV*, 2018. 1

[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *ICML*, 2021. 2, 3

[33] Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, and Christoph Feichtenhofer. MAViL: Masked audio-video learners. In *NeurIPS*, 2023. 2, 3

[34] Jakob Hackstein, Gencer Sumbul, Kai Norman Clasen, and Begüm Demir. Exploring masked autoencoders for sensor-agnostic image retrieval in remote sensing. *arXiv preprint arXiv:2401.07782*, 2024. 2

[35] Steve Ahlswede, Christian Schulz, Christiano Gava, Patrick Helber, Benjamin Bischke, Michael Förster, Florencia Arias, Jörn Hees, Begüm Demir, and Birgit Kleinschmit. TreeSatAI Benchmark Archive: A multi-sensor, multi-label dataset for tree species classification in remote sensing. *Earth System Science Data Discussions*, 2022. 2, 3, 4, 7, 9

[36] Vivien Sainte Fare Garnot and Loic Landrieu. Panoptic segmentation of satellite image time series with convolutional temporal attention networks. In *ICCV*, 2021. 2, 7, 8, 10

[37] Vivien Sainte Fare Garnot, Loic Landrieu, and Nesrine Chehata. Multi-modal temporal attention models for crop mapping from satellite time series. *ISPRS Journal of Photogrammetry and Remote Sensing*, 2022. 2, 3, 4, 7, 8, 10

[38] Jacob Shermeyer, Daniel Hogan, Jason Brown, Adam Van Etten, Nicholas Weir, Fabio Pacifici, Ronny Hansch, Alexei Bastidas, Scott Soenen, Todd Bacastow, et al. Spacenet 6: Multi-sensor all weather mapping dataset. In *CVPR Workshop EarthVision*, 2020. 2

[39] Gencer Sumbul, Arne De Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benedides, Mario Caetano, Begüm Demir, and Volker Markl. BigEarthNet-MM: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval. *IEEE Geoscience and Remote Sensing Magazine*, 2021. 2, 4

[40] Caleb Robinson, Kolya Malkin, Nebojsa Jojic, Huijun Chen, Rongjun Qin, Changlin Xiao, Michael Schmitt, Pedram Ghamisi, Ronny Hänsch, and Naoto Yokoya. Global land-cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS data fusion contest. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 2021. 2, 4

[41] Jingliang Hu, Rong Liu, Danfeng Hong, Andrés Camero, Jing Yao, Mathias Schneider, Franz Kurz, Karl Segl, and Xiao Xiang Zhu. MDAS: A new multimodal benchmark dataset for remote sensing. *Earth System Science Data Discussions*, 2022. 2, 4

[42] Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu. SSL4EO-S12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in Earth observation. *IEEE Geoscience and Remote Sensing Magazine*, 2023. 2
## Acknowledgements

This work was supported by ANR project READY3D ANR-19-CE23-0007, and was granted access to the HPC resources of IDRIS under the allocations AD011014719 and AD011014286R1 made by GENCI. We thank Anatol Garioud and Sébastien Giordano for their help on the creation of TreeSatAI-TS and PASTIS-HD datasets. The SPOT images are opendata thanks to the Dataterra Dinamis initiative in the case of the "Couverture France DINAMIS" program. We thank Jordi Inglada for inspiring discussions and valuable feedback.

## References

1. PyTorch: ReduceLROnPlateau. org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#torch.optim.lr_scheduler.ReduceLROnPlateau, accessed: 2024-02-29 9

2. Ahlswede, S., Schulz, C., Gava, C., Helber, P., Bischke, B., Förster, M., Arias, F., Hees, J., Demir, B., Kleinschmit, B.: TreeSatAI Benchmark Archive: A multi-sensor, multi-label dataset for tree species classification in remote sensing. Earth System Science Data Discussions (2022) 2, 3, 4, 9, 10, 11, 22

3. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., Simonyan, K.: Flamingo: A visual language model for few-shot learning. In: NeurIPS (2022) 3

4. Amitrano, D., Di Martino, G., Guida, R., Iervolino, P., Iodice, A., Papa, M.N., Riccio, D., Ruello, G.: Earth environmental monitoring using multi-temporal synthetic aperture radar: A critical review of selected applications. Remote Sensing (2021) 1

5. Anderson, K., Ryan, B., Sonntag, W., Kavvada, A., Friedl, L.: Earth observation in service of the 2030 agenda for sustainable development. Geo-spatial Information Science (2017) 1

6. Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: CVPR (2023) 2

7. Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., Ermon, S.: Geography-aware self-supervised learning. In: ICCV (2021) 2

8. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE TPAMI (2017) 6

9. Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., Auli, M.: Data2vec: A general framework for self-supervised learning in speech, vision and language. In: ICML (2022) 3

10. Bao, H., Dong, L., Piao, S., Wei, F.: BEiT: BERT pre-training of image transformers. In: ICLR (2021) 2

11. Bao, X., Zhang, R., Lv, J., Wu, R., Zhang, H., Chen, J., Zhang, B., Ouyang, X., Liu, G.: Vegetation descriptors from Sentinel-1 SAR data for crop growth monitoring. ISPRS Journal of Photogrammetry and Remote Sensing (2023) 4

12. Bastani, F., Wolters, P., Gupta, R., Ferdinando, J., Kembhavi, A.: SatlasPretrain: A large-scale dataset for remote sensing image understanding. In: ICCV (2023) 3, 4

13. Bayoudh, K., Knani, R., Hamdaoui, F., Mtibaa, A.: A survey on deep multimodal learning for computer vision: Advances, trends, applications, and datasets. The Visual Computer (2022) 3
[43] Yanbiao Ma, Yuxin Li, Kexin Feng, Yu Xia, Qi Huang, Hongyan Zhang, Colin Prieur, Giorgio Licciardi, Hana Malha, Jocelyn Chanussot, et al. The outcome of the 2021 IEEE GRSS data fusion contest-Track DSE: Detection of settlements without electricity. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 2021. 2

[44] Burak Ekim, Timo T. Stomberg, Ribana Roscher, and Michael Schmitt. MapInWild: A remote sensing dataset to address the question of what makes nature wild. *IEEE Geoscience and Remote Sensing Magazine*, 2023. 2

[45] Patrick Ebel, Yajin Xu, Michael Schmitt, and Xiao Xiang Zhu. SEN12MS-CR-TS: A remote-sensing data set for multimodal multitemporal cloud removal. *IEEE TGRS*, 2022. 2

[46] Romain Wenger, Anne Puissant, Jonathan Weber, Lhassane Idoumghar, and Germain Forestier. MultiSenGE: A multimodal and multitemporal benchmark dataset for land use/land cover remote sensing applications. *ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 2022. 2

[47] Sebastian Gerard, Yu Zhao, and Josephine Sullivan. WildfireSpreadTS: A dataset of multi-modal time series for wildfire spread prediction. In *NeurIPS*, 2024. 2

[48] Anatol Garioud, Nicolas Gonthier, Loic Landrieu, Apolline De Wit, Marion Valette, Marc Poupée, Sébastien Giordano, and Boris Wattrelos. FLAIR: A country-scale land cover semantic segmentation dataset from multi-source optical imagery. In *NeurIPS Dataset and Benchmark*, 2023. 2, 3, 4, 7, 8, 9, 10, 11, 22, 23

[49] Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. SatlasPretrain: A large-scale dataset for remote sensing image understanding. In *ICCV*, 2023. 2, 4

[50] Jacob Devlin, Ming-Wei Chang, Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*, 2019. 2

[51] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khali-dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. *TLMR*, 2023. 2, 3

[52] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In *ICLR*, 2018. 2

[53] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In *ECCV*, 2016. 2

[54] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In *ECCV*, 2016. 2

[55] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In *ICML*, 2020. 3

[56] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In *CVPR*, 2020. 3

[57] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In *NeurIPS*, 2020. 3

[58] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh-laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In *NeurIPS*, 2020. 3

[59] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In *CVPR*, 2021. 3
14. Benedetti, P., Ienco, D., Gaetano, R., Ose, K., Pensa, R.G., Dupuy, S.: M³-fusion: A deep learning architecture for multiscale multimodal multitemporal satellite data fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2018) 2

15. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020) 2

16. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021) 2

17. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML (2020) 2

18. Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR (2021) 2

19. Christie, G., Fendley, N., Wilson, J., Mukherjee, R.: Functional map of the world. In: CVPR (2018) 4

20. Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., Ermon, S.: SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery. In: NeurIPS (2022) 2, 3, 10, 11

21. Coppin, P., Lambin, E., Jonckheere, I., Muys, B.: Digital change detection methods in natural ecosystem monitoring: A review. Analysis of multi-temporal remote sensing images (2002) 1

22. Corley, I., Robinson, C., Dodhia, R., Ferres, J.M.L., Najafirad, P.: Revisiting pre-trained remote sensing model benchmarks: Resizing and normalization matters. arXiv preprint arXiv:2305.13456 (2023) 11

23. Dai, A., Nießner, M.: 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation. In: ECCV (2018) 2

24. DataTerra Dinamis: Diffusion OpenData Dinamis, https://dinamis.data-terra.org/opendata/, accessed: 2023-12-15 9

25. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2020) 2, 11, 22

26. Drusch, M., Del Bello, U., Carlier, S., Colin, O., Fernandez, V., Gascon, F., Hoersch, B., Isola, C., Laberinti, P., Martimort, P., Meygret, A., Spoto, F., Sy, O., Marchese, F., Bargellini, P.: Sentinel-2: ESA's optical high-resolution mission for GMES operational services. Remote Sensing of Environment (2012) 1, 4, 8

27. Ebel, P., Xu, Y., Schmitt, M., Zhu, X.X.: SEN12MS-CR-TS: A remote-sensing data set for multimodal multitemporal cloud removal. IEEE TGRS (2022) 3

28. Ekim, B., Stomberg, T.T., Roscher, R., Schmitt, M.: MapInWild: A remote sensing dataset to address the question of what makes nature wild. IEEE Geoscience and Remote Sensing Magazine (2023) 3

29. Fuller, A., Millard, K., Green, J.R.: CROMA: Remote sensing representations with contrastive radar-optical masked autoencoders. In: NeurIPS (2023) 1, 2, 3, 10, 11, 12

30. Gao, Y., Sun, X., Liu, C.: A general self-supervised framework for remote sensing image classification. Remote Sensing (2022) 2

31. Garioud, A., Gonthier, N., Landrieu, L., De Wit, A., Valette, M., Poupée, M., Giordano, S., Wattrelos, B.: FLAIR: A country-scale land cover semantic segmentation dataset from multi-source optical imagery. In: NeurIPS Dataset and Benchmark (2023) 3, 4, 9, 10, 11, 12, 13, 22, 25, 26

32. Garnot, V.S.F., Landrieu, L.: Lightweight temporal self-attention for classifying satellite images time series. In: Advanced Analytics and Learning on Temporal Data: ECML PKDD Workshop (2020) 6, 22

33. Garnot, V.S.F., Landrieu, L.: Panoptic segmentation of satellite image time series with convolutional temporal attention networks. In: ICCV (2021) 2, 3, 9, 12
[60] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In *ICCV*, 2021. 3

[61] Wei-Hsin Tseng, Hoàng-ÂN Lê, Alexandre Boulch, Sébastien Lefèvre, and Dirk Tiede. CROCO: Cross-modal contrastive learning for localization of Earth observation data. *ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 2022. 3

[62] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-aware self-supervised learning. In *ICCV*, 2021. 3

[63] Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In *ICCV*, 2021. 3

[64] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In *ICLR*, 2020. 3, 9, 10

[65] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In *CVPR*, 2022. 3

[66] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In *ICLR*, 2021. 3

[67] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMim: A simple framework for masked image modeling. In *CVPR*, 2022. 3

[68] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In *CVPR*, 2023. 3

[69] Yuan Gao, Xiaojuan Sun, and Chao Liu. A general self-supervised framework for remote sensing image classification. *Remote Sensing*, 2022. 3

[70] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery. In *NeurIPS*, 2022. 3, 9

[71] Yuan Yuan, Lei Lin, Qingshan Liu, Renlong Hang, and Zeng-Guang Zhou. SITS-Former: A pre-trained spatio-spectral-temporal representation model for sentinel-2 time series classification. *International Journal of Applied Earth Observation and Geoinformation*, 2022. 3

[72] Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning. In *ICCV*, 2023. 3, 8, 9, 10, 11, 22, 23

[73] Damian Ibanez, Ruben Fernandez-Beltran, Filiberto Pla, and Naoto Yokoya. Masked auto-encoding spectral-spatial transformer for hyperspectral image classification. *IEEE TGRS*, 2022. 3

[74] Yufei Liu, Xiaorun Li, Ziqiang Hua, Chaoqun Xia, and Liaoying Zhao. A band selection method with masked convolutional autoencoder for hyperspectral image. *IEEE Geoscience and Remote Sensing Letters*, 2022. 3

[75] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In *ICLR*, 2022. 3
34. Garnot, V.S.F., Landrieu, L., Chehata, N.: Multi-modal temporal attention models for crop mapping from satellite time series. ISPRS Journal of Photogrammetry and Remote Sensing (2022) 2, 3, 4, 9, 12

35. Garnot, V.S.F., Landrieu, L., Giordano, S., Chehata, N.: Satellite image time series classification with pixel-set encoders and temporal self-attention. In: CVPR (2020) 11, 22

36. Ghamisi, P., Rasti, B., Yokoya, N., Wang, Q., Hofle, B., Bruzzone, L., Bovolo, F., Chi, M., Anders, K., Gloaguen, R., et al.: Multisource and multitemporal data fusion in remote sensing: A comprehensive review of the state of the art. IEEE Geoscience and Remote Sensing Magazine (2019) 1, 3

37. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018) 2

38. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: CVPR (2023) 1, 2, 3

39. Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., Misra, I.: Omnivore: A single model for many visual modalities. In: CVPR (2022) 2, 3

40. Goldberg, H.R., Ratto, C.R., Banerjee, A., Kelbaugh, M.T., Giglio, M., Vermote, E.F.: Automated global-scale detection and characterization of anthropogenic activity using multi-source satellite-based remote sensing imagery. In: Geospatial Informatics XIII. SPIE (2023) 4

41. Greenwell, C., Crall, J., Purri, M., Dana, K., Jacobs, N., Hadzic, A., Workman, S., Leotta, M.: WATCH: Wide-area terrestrial change hypercube. In: WACV (2024) 2

42. Grill, J.B., Strub, F., Altché, F., Tallec, C., Richmond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. In: NeurIPS (2020) 2

43. Hackstein, J., Sumbul, G., Clasen, K.N., Demir, B.: Exploring masked autoencoders for sensor-agnostic image retrieval in remote sensing. arXiv preprint arXiv:2401.07782 (2024) 2

44. Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture. In: ACCV (2017) 3

45. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR (2022) 2

46. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR (2020) 2

47. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016) 12

48. Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., Zhang, B.: More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE TGRS (2020) 22

49. Hu, J., Liu, R., Hong, D., Camero, A., Yao, J., Schneider, M., Kurz, F., Segl, K., Zhu, X.X.: MDAS: A new multimodal benchmark dataset for remote sensing. Earth System Science Data Discussions (2022) 3, 4

50. Huang, P.Y., Sharma, V., Xu, H., Ryali, C., Fan, H., Li, Y., Li, S.W., Ghosh, G., Malik, J., Feichtenhofer, C.: MAViL: Masked audio-video learners. In: NeurIPS (2023) 2, 3

51. Ibanez, D., Fernandez-Beltran, R., Pla, F., Yokoya, N.: Masked auto-encoding spectral-spatial transformer for hyperspectral image classification. IEEE TGRS (2022) 2

52. Irvin, J., Tao, L., Zhou, J., Ma, Y., Nashold, L., Liu, B., Ng, A.Y.: USat: A unified self-supervised encoder for multi-sensor satellite imagery. arXiv preprint arXiv:2312.02199 (2023) 1, 2, 3, 9

53. Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019) 2
[76] Khaled Bayoudh, Raja Knani, Faycal Hamdaoui, and Abdellatif Mtibaa. A survey on deep multimodal learning for computer vision: Advances, trends, applications, and datasets. *The Visual Computer*, 2022. 3

[77] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture. In *ACCV*, 2017. 3

[78] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In *ICML*, 2022. 3

[79] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: A visual language model for few-shot learning. In *NeurIPS*, 2022. 3

[80] Michael Ying Yang, Loic Landrieu, Devis Tuia, and Charles Toth. Muti-modal learning in photogrammetry and remote sensing. *ISPRS Journal of Photogrammetry and Remote Sensing*, 2021. 3

[81] Jiaxin Li, Danfeng Hong, Lianru Gao, Jing Yao, Ke Zheng, Bing Zhang, and Jocelyn Chanussot. Deep learning in multimodal remote sensing data fusion: A comprehensive review. *International Journal of Applied Earth Observation and Geoinformation*, 112:102926, 2022. 3

[82] Cle Pohl and John L Van Genderen. Multisensor image fusion in remote sensing: Concepts, methods and applications. *International Journal of Remote Sensing*, 1998. 3

[83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In *NeurIPS*, 2017. 3

[84] Michail Tarasiou, Erik Chavez, and Stefanos Zafeiriou. ViTs for SITS: Vision transformers for satellite image time series. In *CVPR*, 2023. 3

[85] Gabriel Tseng, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner. Lightweight, pre-trained transformers for remote sensing timeseries. *arXiv preprint arXiv:2304.14065*, 2023. 3, 9

[86] Xin Bao, Rui Zhang, Jichao Lv, Renzhe Wu, Hongsheng Zhang, Jie Chen, Bo Zhang, Xiaoying Ouyang, and Guoxiang Liu. Vegetation descriptors from Sentinel-1 SAR data for crop growth monitoring. *ISPRS Journal of Photogrammetry and Remote Sensing*, 2023. 4

[87] Anton Vrieling, Michele Meroni, Roshanak Darvishzadeh, Andrew K Skidmore, Tiejun Wang, Raul Zurita-Milla, Kees Oosterbeek, Brian O'Connor, and Marc Paganini. Vegetation phenology from Sentinel-2 and field cameras for a Dutch barrier island. *Remote sensing of environment*, 2018. 4, 7

[88] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In *CVPR*, 2018. 4

[89] Hirsh R Goldberg, Christopher R Ratto, Amit Banerjee, Michael T Kelbaugh, Mark Giglio, and Eric F Vermote. Automated global-scale detection and characterization of anthropogenic activity using multi-source satellite-based remote sensing imagery. In *Geospatial Informatics XIII*. SPIE, 2023. 4

[90] Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu. Neural plasticity-inspired foundation model for observing the Earth crossing modalities. *arXiv preprint arXiv:2403.15356*, 2024. 4
54. Kingma, D.P., Ba, J.: ADAM: A method for stochastic optimization. ICLR (2015) 9

55. Krispel, G., Opitz, M., Waltner, G., Possegger, H., Bischof, H.: Fuseseg: LiDAR point cloud segmentation fusing multi-modal data. In: WACV (2020) 2

56. Kuffer, M., Thomson, D.R., Boo, G., Mahabir, R., Grippa, T., Vanhuysse, S., Engstrom, R., Ndugwa, R., Makau, J., Darin, E., et al.: The role of Earth observation in an integrated deprived area mapping “system” for low-to-middle income countries. Remote sensing (2020) 1

57. Kussul, N., Lavreniuk, M., Skakun, S., Shelestov, A.: Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters (2017) 22

58. Lacoste, A., Sherwin, E.D., Kerner, H., Alemohammad, H., Lütjens, B., Irvin, J., Dao, D., Chang, A., Gunturkun, M., Drouin, A., et al.: Toward foundation models for Earth monitoring: Proposal for a climate change benchmark. arXiv preprint arXiv:2112.00570 (2021) 1

59. Li, D., Tong, Q., Li, R., Gong, J., Zhang, L.: Current issues in high-resolution Earth observation technology. Science China Earth sciences (2012) 1

60. Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., Chanussot, J.: Deep learning in multimodal remote sensing data fusion: A comprehensive review. International Journal of Applied Earth Observation and Geoinformation 112, 102926 (2022) 3

61. Liao, Y., Xie, J., Geiger, A.: KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. IEEE TPAMI (2022) 2

62. Liu, Y., Li, X., Hua, Z., Xia, C., Zhao, L.: A band selection method with masked convolutional autoencoder for hyperspectral image. IEEE Geoscience and Remote Sensing Letters (2022) 2

63. Ma, Y., Li, Y., Feng, K., Xia, Y., Huang, Q., Zhang, H., Prieur, C., Licciardi, G., Malha, H., Chanussot, J., et al.: The outcome of the 2021 IEEE GRSS data fusion contest-Track DSE: Detection of settlements without electricity. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2021) 3

64. Mai, G., Huang, W., Sun, J., Song, S., Mishra, D., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al.: On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798 (2023) 1

65. Manas, O., Lacoste, A., Giró-i Nieto, X., Vazquez, D., Rodriguez, P.: Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In: ICCV (2021) 2

66. Manfreda, S., McCabe, M.F., Miller, P.E., Lucas, R., Pajuelo Madrigal, V., Mallinis, G., Ben Dor, E., Helman, D., Estes, L., Ciraolo, G., et al.: On the use of unmanned aerial systems for environmental monitoring. Remote sensing p. 641 (2018) 1

67. Moreira, A., Prats-Iraola, P., Younis, M., Krieger, G., Hajnsek, I., Papathanassiou, K.P.: A tutorial on synthetic aperture radar. IEEE Geoscience and Remote Sensing Magazine (2013) 1

68. Nakalembe, C.: Urgent and critical need for sub-saharan african countries to invest in Earth observation-based agricultural early warning and monitoring systems. Environmental Research Letters (2020) 1

69. Nathan Silberman, Derek Hoiem, P.K., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: ECCV (2012) 2

70. Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016) 2

71. Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018) 7

72. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. TLMR (2023) 2, 3
[91] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 5

[92] Esther Rolf, Konstantin Klemmer, Caleb Robinson, and Hannah Kerner. Mission critical–Satellite data is a distinct modality in machine learning. arXiv preprint arXiv:2402.01444, 2024. 6

[93] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE TPAMI, 2017. 6

[94] Vivien Sainte Fare Garnot and Loic Landrieu. Lightweight temporal self-attention for classifying satellite images time series. In Advanced Analytics and Learning on Temporal Data: ECML PKDD Workshop, 2020. 6, 10

[95] Martin Sudmanns, Dirk Tiede, Hannah Augustin, and Stefan Lang. Assessing global Sentinel-2 coverage dynamics and data availability for operational Earth observation (EO) applications using the EO-Compass. International Journal of Digital Earth, 2019. 7

[96] Marc Rußwurm and Marco Körner. Self-attention for raw optical satellite time series classification. ISPRS Journal of Photogrammetry and Remote Sensing, 2020. 7

[97] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021. 7

[98] Diederik P Kingma and Jimmy Ba. ADAM: A method for stochastic optimization. ICLR, 2015. 7

[99] PyTorch: ReduceLROnPlateau. org/docs/stable/generated/torch.optim(lr_scheduler.ReduceLROnPlateau.html#torch.optim(lr_scheduler.ReduceLROnPlateau). Accessed: 2024-02-29. 7

[100] DataTerra Dinamis. Diffusion OpenData Dinamis. URL https://dinamis.data-terra.org/opendata/. Accessed: 2023-12-15. 8

[101] Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, and Nesrine Chehata. Satellite image time series classification with pixel-set encoders and temporal self-attention. In CVPR, 2020. 9

[102] Isaac Corley, Caleb Robinson, Rahul Dodhia, Juan M Lavista Ferres, and Peyman Najafi-rad. Revisiting pre-trained remote sensing model benchmarks: Resizing and normalization matters. arXiv preprint arXiv:2305.13456, 2023. 9

[103] Esther Rolf, Jonathan Proctor, Tamma Carleton, Ian Bolliger, Vaishaal Shankar, Miyabi Ishihara, Benjamin Recht, and Solomon Hsiang. A generalizable and accessible approach to machine learning with global satellite imagery. Nature communications, 2021. 9

[104] Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, and Bing Zhang. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE TGRS, 2020. 9

[105] Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 2017. 10

[106] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 10
73. Pohl, C., Van Genderen, J.L.: Multisensor image fusion in remote sensing: Concepts, methods and applications. International journal of remote sensing (1998) 3

74. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021) 2, 3

75. Recasens, A., Lin, J., Carreira, J., Jaegle, D., Wang, L., Alayrac, J.b., Luc, P., Miech, A., Smaira, L., Hemsley, R., et al.: Zorro: The masked multimodal transformer. arXiv preprint arXiv:2301.09595 (2023) 2

76. Reed, C.J., Gupta, R., Li, S., Brockman, S., Funk, C., Clipp, B., Keutzer, K., Candido, S., Uyttendaele, M., Darrell, T.: Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning. In: ICCV (2023) 2, 3, 10, 11, 12, 13, 25, 26

77. Robert, D., Vallet, B., Landrieu, L.: Learning multi-view aggregation in the wild for large-scale 3D semantic segmentation. In: CVPR (2022) 2

78. Robinson, C., Malkin, K., Jojic, N., Chen, H., Qin, R., Xiao, C., Schmitt, M., Ghamisi, P., Hänsch, R., Yokoya, N.: Global land-cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS data fusion contest. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2021) 3, 4

79. Rolf, E., Proctor, J., Carleton, T., Bolliger, I., Shankar, V., Ishihara, M., Recht, B., Hsiang, S.: A generalizable and accessible approach to machine learning with global satellite imagery. Nature communications (2021) 10, 11

80. Rußwurm, M., Körner, M.: Self-attention for raw optical satellite time series classification. ISPRS Journal of Photogrammetry and Remote Sensing (2020) 6

81. Schmitt, M., Zhu, X.X.: Data fusion and remote sensing: An ever-growing relationship. IEEE Geoscience and Remote Sensing Magazine (2016) 1, 3

82. Secades, C., O'Connor, B., Brown, C., Walpole, M., et al.: Earth observation for biodiversity monitoring: A review of current approaches and future opportunities for tracking progress towards the aichi biodiversity targets. CBD technical series (2014) 1

83. Shermeyer, J., Hogan, D., Brown, J., Van Etten, A., Weir, N., Pacifici, F., Hansch, R., Bastidas, A., Soenen, S., Bacastow, T., et al.: Spacenet 6: Multi-sensor all weather mapping dataset. In: CVPR Workshop EarthVision (2020) 3

84. Shukor, M., Dancette, C., Rame, A., Cord, M.: UnIVAL: Unified model for image, video, audio and language tasks. TMLR (2023) 1, 2, 3

85. Skidmore, A.K., Coops, N.C., Neinavaz, E., Ali, A., Schaepman, M.E., Paganini, M., Kissling, W.D., Vihervaara, P., Darvishzadeh, R., Feilhauer, H., et al.: Priority list of biodiversity metrics to observe from space. Nature Ecology & Evolution (2021) 1

86. Srivastava, S., Sharma, G.: OmniVec: Learning robust representations with cross modal sharing. In: WACV (2024) 2, 3

87. Sudmanns, M., Tiede, D., Augustin, H., Lang, S.: Assessing global Sentinel-2 coverage dynamics and data availability for operational Earth observation (EO) applications using the EO-Compass. International Journal of Digital Earth (2019) 6

88. Sumbul, G., De Wall, A., Kreuziger, T., Marcelino, F., Costa, H., Benevides, P., Caetano, M., Demir, B., Markl, V.: BigEarthNet-MM: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval. IEEE Geoscience and Remote Sensing Magazine (2021) 3, 4

89. Tarasiou, M., Chavez, E., Zafeiriou, S.: ViTs for SITS: Vision transformers for satellite image time series. In: CVPR (2023) 3

90. Tseng, G., Zvonkov, I., Purohit, M., Rolnick, D., Kerner, H.: Lightweight, pre-trained transformers for remote sensing timeseries. arXiv preprint arXiv:2304.14065 (2023) 3, 10, 11
# Appendix

In this appendix, we present an extended ablation study in Section 1, detailed classwise performance in Section 2, and analysis of failure cases on challenging examples in Figure 2.

## 1 Supplementary Ablations

We propose two supplementary ablation cases for a more in-depth analysis and comprehensive assessment of OmniSat:

**Relative vs. Absolute Positional Encoding.** We evaluate the impact of replacing the relative positional encoding of tokens, based on the patch position, with an absolute position encoding, based on the position of the patches in their tile—similar to what is classically done for image processing.

With an absolute positional encoding, OmniSat reaches an F1-score of 58.4 and 73.0 when fine-tuned with 10% and 100% of the training set of TreeSatAI-TS, respectively. This is 2.7 and 1.2% below a model trained with relative positional encodings. We conclude that relative positional encodings are better suited for analyzing EO images. While the upper patches of natural images are bound to correspond to the sky, and the lower patches contain ground, no such analogy can be made for EO data, whose distribution is equivariant through small horizontal translation.

**Impact of Pretraining on Monomodal Performance.** We aim to determine how our multimodal pretraining scheme improves the monomodal performance (e.g., +13.2% for Sentinel-2 in full supervision). We consider two mechanisms that may lead to more discriminative features: (i) multimodality allows us to train the modality combiner network $C$ with more data, or (ii) our cross-modal and token-wise alignment-based losses provide a strong supervisory signal. We propose an experiment to verify which mechanism is the leading reason of our scheme's strong performance.

We pre-train OmniSat on TreeSatAI-TS in mono- and multimodal settings with a constant amount of tokens. More precisely, we pre-train OmniSat using all input tokens from the S2 modality only, and using all 3 modalities but only 33% of patches. This means that each experiment considering the same number $P$ of input tokens. We then train a single linear layer to map these representations to class scores (linear probing) using 10 and 100% of the annotated S2 data. Finally, we evaluate the quality of these linear mappings on the test set using only the S2 modality.

The model trained with a multimodal pretext task reaches a F1-score of 44.7 for 10% and 46.3 for 100% of the training data. The model trained only with S2 performs significantly worse: 26.9 for 10% and 29.8 for 100% of data. This result suggests that the key to the efficacy of our pretraining scheme is the supervisory signal of per-patch contrastive and reconstruction objectives, rather than just increasing the number of tokens viewed by the transformer backbone.

## 2 Supplementary Results

We report the performance of different approaches for each class for the two datasets graphically in Figure 1 and as a table in Table 1.

**Failure Case.** We report in Figure 2 hard examples from our three datasets and compare the prediction of OmniSat and other models. For the TreeSatAI-TS example, the Sentinel-2 optical time-series is highly occluded: over 80% of acquisitions are covered by clouds. Furthermore, the forest tile contains a large variety of tree species organized in densely connected canopy, making its classification particularly hard. Indeed, the texture of the images in closed forests does not bring additional discriminative information.

The example from FLAIR is a scrap yard, which is almost entirely covered by broken vehicles. Since FLAIR's annotations focus on the ground rather than transient or stationary objects, identifying the actual land cover in such scenarios is very challenging.

The image taken from PASTIS contains a mix of several different crop types, including the class mixed cereal which can already correspond to a parcel with various cereal types. This leads to a hard classification problem for all methods.
91. Tseng, W.H., Lê, H.Â., Boulch, A., Lefèvre, S., Tiede, D.: CROCO: Cross-modal contrastive learning for localization of Earth observation data. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2022) 2

92. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 3

93. Vrieling, A., Meroni, M., Darvishzadeh, R., Skidmore, A.K., Wang, T., Zurita-Milla, R., Oosterbeek, K., O'Connor, B., Paganini, M.: Vegetation phenology from Sentinel-2 and field cameras for a Dutch barrier island. Remote sensing of environment (2018) 4, 9

94. Wang, Y., Braham, N.A.A., Xiong, Z., Liu, C., Albrecht, C.M., Zhu, X.X.: SSL4EO-S12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in Earth observation. IEEE Geoscience and Remote Sensing Magazine (2023) 3

95. Wenger, R., Puissant, A., Weber, J., Idoumghar, L., Forestier, G.: MultiSenGE: A multimodal and multitemporal benchmark dataset for land use/land cover remote sensing applications. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2022) 3

96. Wu, K., Peng, H., Chen, M., Fu, J., Chao, H.: Rethinking and improving relative position encoding for vision transformer. In: ICCV (2021) 6

97. Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., Hu, H.: SimMim: A simple framework for masked image modeling. In: CVPR (2022) 2

98. Xiong, Z., Wang, Y., Zhang, F., Stewart, A.J., Hanna, J., Borth, D., Papoutsis, I., Saux, B.L., Camps-Valls, G., Zhu, X.X.: Neural plasticity-inspired foundation model for observing the Earth crossing modalities. arXiv preprint arXiv:2403.15356 (2024) 3, 4, 11, 12, 22

99. Yang, J., Gong, P., Fu, R., Zhang, M., Chen, J., Liang, S., Xu, B., Shi, J., Dickinson, R.: The role of satellite remote sensing in climate change studies. Nature climate change (2013) 1

100. Yang, M.Y., Landrieu, L., Tuia, D., Toth, C.: Multimodal learning in photogrammetry and remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing (2021) 3

101. Yuan, Y., Lin, L., Liu, Q., Hang, R., Zhou, Z.G.: SITS-Former: A pre-trained spatio-spectral-temporal representation model for sentinel-2 time series classification. International Journal of Applied Earth Observation and Geoinformation (2022) 2

102. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016) 2

103. Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., Kong, T.: Image BERT pre-training with online tokenizer. In: ICLR (2022) 3

104. Zong, Y., Mac Aodha, O., Hospedales, T.: Self-supervised multimodal learning: A survey. arXiv preprint arXiv:2304.01008 (2023) 1

# Appendix

In this appendix, we present an extended ablation study in Section A-1, details our competing methods in Section A-2, provide the classwise performance in Section A-3, and provide qualitative illustrations in Figure A-2.

## A-1 Supplementary Ablations

We propose supplementary ablations to evaluate the impact of several design choices.
<figure><img src="image_8.png" /><figcaption>Figure 1: **Class-Wise Performance.** We plot the performance of different models for each class, sorted by decreasing frequency. OmniSat improves the performance across the board, and for rare classes in particular.</figcaption></figure>
<table><caption>**Table A-1: Supplementary Ablation.** Performance (weighted F1) on TreeSatAI-TS of alternate VHR encoders (a-c) and masking schemes (d-e).</caption><thead><tr><th>Experiment</th><th>All</th><th>VHR</th><th>S2</th><th>S1</th></tr></thead><tbody><tr><td>Default</td><td><strong>74.2</strong></td><td>70.5</td><td>62.9</td><td><strong>56.7</strong></td></tr><tr><td>a linear w. random init</td><td>66.8</td><td>57.3</td><td>58.9</td><td>54.8</td></tr><tr><td>b ViT</td><td>70.5</td><td><strong>70.8</strong></td><td>64.0</td><td>52.6</td></tr><tr><td>c linear from ScaleMAE</td><td>68.9</td><td>51.2</td><td><strong>66.7</strong></td><td>52.2</td></tr><tr><td>d Spatial masking</td><td>73.2</td><td>70.1</td><td>63.2</td><td>54.6</td></tr><tr><td>e Modality masking</td><td>72.4</td><td>70.2</td><td>61.2</td><td>55.4</td></tr></tbody></table>

**Alternate VHR Encoder.** To train OmniSat on both VHR (20cm) and Sentinel (10m) images, we must embed patches of 50×50 pixels. We consider here alternative encoders to CNNs: a linear layer (Tab. A-1.a) and a ViT with 10×10 patches (Tab. A-1.b). The results suggest that 50 × 50 patches are too large to use linear projection. While ViTs reach slightly higher unimodal performance, CNNs allow us to bypass maxpool indices to the decoders (Sec. 3.1) leading to higher multimodal performances.

**Using Pretrained VHR models.** Rescaling the 50 × 50 patches to the 224 × 224 resolution of ScaleMAE or SatMAE proved impractical in terms of memory. Instead, we use the pretrained patch encoder of ScaleMAE by rescaling our 50×50 patches to 16×16, removing the infrared channel, and adding a projection layer to our token size *D* = 256 (Tab. A-1.c). Interestingly, this leads to a cross-modal distillation which improves the results for S2. The VHR and multimodal performance remain below OmniSat, which can be attributed to the lack of a NIR channel.

**Masking Strategies.** We report the results for spatially consistent masking (patches are masked for all modalities simultaneously, Tab. A-1.d) and modality masking (the patches of a random modality are all masked, Tab. A-1.e). Our random masking strategy performs better.

**Relative vs. Absolute Positional Encoding.** We evaluate the impact of replacing the relative positional encoding of tokens, based on the patch position, with an absolute positional encoding, based on the position of the patches in their tile—similar to what is classically done for image processing.

With an absolute positional encoding, OmniSat reaches an F1-score of 58.4 and 73.0 when fine-tuned with 10% and 100% of the training set of TreeSatAI-TS, respectively. This is 2.7 and 1.2% below a model trained with relative positional encodings. We conclude that relative positional encodings are better suited for analyzing EO images. While the upper patches of natural images are bound to correspond to the sky, and the lower patches contain ground, no such analogy can be made for EO data, whose distribution is equivariant through small horizontal translation.

**Impact of Pretraining on Monomodal Performance.** We aim to determine how our multimodal pretraining scheme improves the monomodal performance (e.g., +13.2% for Sentinel-2 in full supervision). We consider two mechanisms that may lead to more
## TreeSatAI-TS

<table><thead><tr><th>Method</th><th>Macro F1</th><th>Abies</th><th>Acer</th><th>Alnus</th><th>Betula</th><th>Cleared</th><th>Fagus</th><th>Fraxinus</th><th>Larix</th><th>Picea</th><th>Pinus</th><th>Populus</th><th>Prunus</th><th>Pseudotsuga</th><th>Quercus</th><th>Tilia</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>2.5</td><td>6.5</td><td>6.5</td><td>7.1</td><td>11.2</td><td>22.6</td><td>5.7</td><td>9.7</td><td>21.8</td><td>23.0</td><td>1.8</td><td>0.8</td><td>8.9</td><td>22.8</td><td>0.5</td></tr><tr><td>UT&amp;T</td><td>48.8</td><td>52.8</td><td>43.4</td><td>49.8</td><td>36.8</td><td>53.7</td><td>60.7</td><td>45.9</td><td>43.7</td><td>63.6</td><td>72.1</td><td>44.8</td><td>32.4</td><td>45.4</td><td>62.5</td><td>25.0</td></tr><tr><td>Scale-MAE</td><td>47.3</td><td>53.2</td><td>32.4</td><td>42.7</td><td>38.5</td><td>64.4</td><td>63.8</td><td>38.3</td><td>51.7</td><td>71.5</td><td>78.2</td><td>25.4</td><td>11.8</td><td>55.1</td><td>64.0</td><td>18.2</td></tr><tr><td>OmniSat</td><td><strong>73.4</strong></td><td><strong>74.1</strong></td><td><strong>58.9</strong></td><td><strong>62.5</strong></td><td><strong>56.5</strong></td><td><strong>74.9</strong></td><td><strong>72.9</strong></td><td><strong>60.0</strong></td><td><strong>69.5</strong></td><td><strong>77.6</strong></td><td><strong>85.4</strong></td><td><strong>64.6</strong></td><td><strong>46.2</strong></td><td><strong>68.5</strong></td><td><strong>76.1</strong></td><td><strong>43.4</strong></td></tr></tbody></table>

## FLAIR

<table><thead><tr><th>Method</th><th>Macro F1</th><th>building</th><th>perv. surface</th><th>imperv. surface</th><th>bare soil</th><th>water</th><th>coniferous</th><th>deciduous</th><th>brushwood</th><th>vineyard</th><th>herbaceous</th><th>agri. land</th><th>plowed land</th><th>other</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>27.6</td><td>45.0</td><td>37.7</td><td>8.0</td><td>12.5</td><td>36.0</td><td>84.2</td><td>49.6</td><td>6.5</td><td>74.1</td><td>39.4</td><td>7.1</td><td>11.0</td></tr><tr><td>UT&amp;T</td><td>57.3</td><td>52.4</td><td>63.3</td><td>60.4</td><td>32.7</td><td>46.9</td><td>64.4</td><td>91.2</td><td>66.3</td><td>57.1</td><td>84.7</td><td><strong>71.7</strong></td><td>27.1</td><td>26.2</td></tr><tr><td>Scale-MAE</td><td>70.0</td><td>90.1</td><td>72.0</td><td>87.1</td><td>47.1</td><td><strong>81.3</strong></td><td>65.1</td><td><strong>95.2</strong></td><td>72.3</td><td>53.7</td><td><strong>88.8</strong></td><td>70.2</td><td><strong>39.7</strong></td><td>45.7</td></tr><tr><td>OmniSat</td><td><strong>75.8</strong></td><td><strong>91.8</strong></td><td><strong>73.2</strong></td><td><strong>88.2</strong></td><td><strong>54.4</strong></td><td><strong>81.3</strong></td><td><strong>70.6</strong></td><td><strong>95.2</strong></td><td><strong>73.4</strong></td><td><strong>72.1</strong></td><td>87.5</td><td>71.1</td><td>36.0</td><td><strong>48.5</strong></td></tr></tbody></table>

## PASTIS-HD

<table><caption>Table 1: Class-Wise Performance. We report the F1-score for each class for TreeSatAI-TS, FLAIR, and PASTIS-HD for multilabel classification. We also report the unweighted class-averaged F1-score (Macro-F1). We can observe that OmniSat outperforms UT&T [48] and Scale-MAE [72] on nearly all classes for both datasets. In particular, we observe the most significant gains for classes with discriminative temporal dynamics, such as broadleaf tree species and the vineyards class.</caption><thead><tr><th>Method</th><th>Macro F1</th><th>Meadow</th><th>Soft winter wheat</th><th>Corn</th><th>Winter barley</th><th>Winter rapeseed</th><th>Spring barley</th><th>Sunflower</th><th>Grapevine</th><th>Beet</th><th>Winter triticale</th><th>Winter durum wheat</th><th>Fruits, vegetables, flowers</th><th>Potatoes</th><th>Leguminous-toddler</th><th>Soybeans</th><th>Orchard</th><th>Mixed cereal</th><th>Sorghum</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>94.4</td><td>63.7</td><td>61.9</td><td>46.4</td><td>29.0</td><td>19.4</td><td>17.3</td><td>20.0</td><td>12.3</td><td>19.8</td><td>15.1</td><td>22.6</td><td>12.7</td><td>42.7</td><td>14.9</td><td>17.7</td><td>19.6</td><td>15.3</td></tr><tr><td>UT&amp;T</td><td>53.5</td><td>97.1</td><td>78.3</td><td>77.5</td><td>63.1</td><td>57.5</td><td>25.4</td><td>49.2</td><td>54.9</td><td>69.9</td><td>50.7</td><td>50.5</td><td>50.6</td><td>22.0</td><td><strong>60.0</strong></td><td>59.0</td><td>46.3</td><td><strong>36.9</strong></td><td>13.8</td></tr><tr><td>CROMA</td><td>60.1</td><td>97.1</td><td>87.1</td><td>88.9</td><td>64.2</td><td>86.5</td><td>41.1</td><td>64.5</td><td>74.2</td><td>76.5</td><td><strong>51.2</strong></td><td>74.3</td><td><strong>61.3</strong></td><td>26.1</td><td>53.3</td><td>59.1</td><td>45.9</td><td>22.4</td><td>9.1</td></tr><tr><td>Scale-MAE</td><td>42.2</td><td>97.1</td><td>84.2</td><td>83.3</td><td>56.4</td><td>53.6</td><td>15.1</td><td>24.1</td><td>54.9</td><td>39.1</td><td>27.5</td><td>43.9</td><td>41.5</td><td>12.3</td><td>48.1</td><td>34.2</td><td>30.4</td><td>12.5</td><td>2.6</td></tr><tr><td>OmniSat</td><td><strong>69.9</strong></td><td><strong>97.2</strong></td><td><strong>90.1</strong></td><td><strong>92.2</strong></td><td><strong>82.6</strong></td><td><strong>89.0</strong></td><td><strong>64.7</strong></td><td><strong>75.0</strong></td><td><strong>82.7</strong></td><td><strong>88.1</strong></td><td>48.5</td><td><strong>81.3</strong></td><td>56.7</td><td><strong>52.6</strong></td><td>56.9</td><td><strong>85.7</strong></td><td><strong>48.4</strong></td><td>34.8</td><td><strong>31.7</strong></td></tr></tbody></table>
discriminative features: (i) multimodality allows us to train the modality combiner network *C* with more data, or (ii) our cross-modal and token-wise alignment-based losses provide a strong supervisory signal. We propose an experiment to verify which mechanism is the leading reason of our scheme's strong performance.

We pre-train OmniSat on TreeSatAI-TS in mono- and multimodal settings with a constant amount of tokens. More precisely, we pre-train OmniSat using all input tokens from the S2 modality *only*, and using all 3 modalities but only 33% of patches. This means that each experiment considering the same number *P* of input tokens. We then train a single linear layer to map these representations to class scores (linear probing) using 10 and 100% of the annotated S2 data. Finally, we evaluate the quality of these linear mappings on the test set using only the S2 modality.

The model trained with a multimodal pretext task reaches a F1-score of 44.7 for 10% and 46.3 for 100% of the training data. The model trained only with S2 performs significantly worse: 26.9 for 10% and 29.8 for 100% of data. This result suggests that the key to the efficacy of our pretraining scheme is the supervisory signal of per-patch contrastive and reconstruction objectives, rather than just increasing the number of tokens viewed by the transformer backbone.

## A-2 Adapting Competing Methods

We adapt competing methods to allow them to handle single images and time series at different resolutions. We performed multiple tests for each approach and kept the configurations leading to the competing approach' highest performance.

- **Multimodality.** We train methods that are not natively multimodal (PSE [35], ViT [25], DOFA [98], SatMAE, ScaleMAE) using a late-fusion scheme [48] by concatenating the embeddings learned in each modality, as suggested by Ahlswede et al. [2]. For UT&T [31], initially designed for VHR images and Sentinel-2 time series, we add a branch for Sentinel-1 integration, which is identical to the Sentinel-2 branch except for the first layer.

- **Handling Temporal Data.** To evaluate image models (SatMAE, ScaleMAE, CROMA) on time series, we convert image sequences to single images by concatenating for each pixel and channel channel-wise the median observation for the four seasons: spring, summer, fall, and winter [57].

- **Handling VHR Data.** To evaluate methods designed for low-resolution images (PSE, LTAE [32]) in a multimodal setting that includes VHR images, we concatenate their final embedding to the one of a ResNet network.

- **Scaling Models.** The considered datasets are smaller than the ones typically used to train large ViT-based models, making them prone to overfitting. To address this issue we select a ViT-Small [25] backbone for SatMAE, ScaleMAE and CROMA. For DOFA, we use a ViT-Base, the smallest pretrained model available.

- **Multi-Class Prediction.** To evaluate ViT-based models on classification experiments, we insert a linear layer that maps the embedding of the class token $\langle CLS \rangle$ to a vector of class scores. For the UT&T model, we compute a spatial average of the last feature map, followed by a similar linear projection.
<figure><img src="image_9.png" /><figcaption>Figure 2: **Failure Cases.** We provide hard instances for the TreeSatAI-TS, FLAIR, and PASTIS-HD datasets, and report the multiclass prediction for OmniSat as well as two competing methods.</figcaption></figure>
A-3 Supplementary Results

We report the performance of different approaches for each class for the two datasets graphically in Figure A-1 and as a table in Table A-2. OmniSat is able to parse complex scenes including mixed forest, cultures, and complex urban areas. In particular, Omnisat leverage temporal dynamics to distinguish between different vegetation species.

**Failure Case.** We report in the bottom half of Figure A-2 hard examples from our three datasets and compare the prediction of OmniSat and other models. For the TreeSatAI-TS example, the Sentinel-2 optical time-series is highly occluded: over 80% of acquisitions are covered by clouds. Furthermore, the forest tile contains a large variety of tree species organized in densely connected canopy, making its classification particularly hard. Indeed, the texture of the images in closed forests does not bring additional discriminative information.

The example from FLAIR is a scrap yard, which is almost entirely covered by broken vehicles. Since FLAIR's annotations focus on the ground rather than transient or stationary objects, identifying the actual land cover in such scenarios is very challenging.

The image taken from PASTIS contains a mix of several different crop types, including the class *mixed cereal* which can already correspond to a parcel with various cereal types. This leads to a hard classification problem for all methods.
<figure><img src="image_9.png" /><figcaption>**Fig. A-1: Class-Wise Performance.** We plot the performance of different models for each class, sorted by decreasing frequency. OmniSat improves the performance across the board, and for rare classes in particular.</figcaption></figure>
## TreeSatAI-TS

<table><thead><tr><th>Method</th><th>Macro F1</th><th>Abies</th><th>Acer</th><th>Alnus</th><th>Betula</th><th>Cleared</th><th>Fagus</th><th>Fraxinus</th><th>Larix</th><th>Picea</th><th>Pinus</th><th>Populus</th><th>Prunus</th><th>Pseudotsuga</th><th>Quercus</th><th>Tilia</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>2.5</td><td>6.5</td><td>6.5</td><td>7.1</td><td>11.2</td><td>22.6</td><td>5.7</td><td>9.7</td><td>21.8</td><td>23.0</td><td>1.8</td><td>0.8</td><td>8.9</td><td>22.8</td><td>0.5</td></tr><tr><td>UT&T</td><td>48.8</td><td>52.8</td><td>43.4</td><td>49.8</td><td>36.8</td><td>53.7</td><td>60.7</td><td>45.9</td><td>43.7</td><td>63.6</td><td>72.1</td><td>44.8</td><td>32.4</td><td>45.4</td><td>62.5</td><td>25.0</td></tr><tr><td>Scale-MAE</td><td>47.3</td><td>53.2</td><td>32.4</td><td>42.7</td><td>38.5</td><td>64.4</td><td>63.8</td><td>38.3</td><td>51.7</td><td>71.5</td><td>78.2</td><td>25.4</td><td>11.8</td><td>55.1</td><td>64.0</td><td>18.2</td></tr><tr><td>OmniSat</td><td><strong>73.4</strong></td><td><strong>74.1</strong></td><td><strong>58.9</strong></td><td><strong>62.5</strong></td><td><strong>56.5</strong></td><td><strong>74.9</strong></td><td><strong>72.9</strong></td><td><strong>60.0</strong></td><td><strong>69.5</strong></td><td><strong>77.6</strong></td><td><strong>85.4</strong></td><td><strong>64.6</strong></td><td><strong>46.2</strong></td><td><strong>68.5</strong></td><td><strong>76.1</strong></td><td><strong>43.4</strong></td></tr></tbody></table>

## FLAIR

<table><thead><tr><th>Method</th><th>Macro F1</th><th>building</th><th>perv. surface</th><th>imperv. surface</th><th>bare soil</th><th>water</th><th>coniferous</th><th>deciduous</th><th>brushwood</th><th>vineyard</th><th>herbaceous</th><th>agri. land</th><th>plowed land</th><th>other</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>27.6</td><td>45.0</td><td>37.7</td><td>8.0</td><td>12.5</td><td>36.0</td><td>84.2</td><td>49.6</td><td>6.5</td><td>74.1</td><td>39.4</td><td>7.1</td><td>11.0</td></tr><tr><td>UT&T</td><td>57.3</td><td>52.4</td><td>63.3</td><td>60.4</td><td>32.7</td><td>46.9</td><td>64.4</td><td>91.2</td><td>66.3</td><td>57.1</td><td>84.7</td><td><strong>71.7</strong></td><td>27.1</td><td>26.2</td></tr><tr><td>Scale-MAE</td><td>70.0</td><td>90.1</td><td>72.0</td><td>87.1</td><td>47.1</td><td><strong>81.3</strong></td><td>65.1</td><td><strong>95.2</strong></td><td>72.3</td><td>53.7</td><td><strong>88.8</strong></td><td>70.2</td><td><strong>39.7</strong></td><td>45.7</td></tr><tr><td>OmniSat</td><td><strong>75.8</strong></td><td><strong>91.8</strong></td><td><strong>73.2</strong></td><td><strong>88.2</strong></td><td><strong>54.4</strong></td><td><strong>81.3</strong></td><td><strong>70.6</strong></td><td><strong>95.2</strong></td><td><strong>73.4</strong></td><td><strong>72.1</strong></td><td>87.5</td><td>71.1</td><td>36.0</td><td><strong>48.5</strong></td></tr></tbody></table>

## PASTIS-HD

<table><thead><tr><th>Method</th><th>Macro F1</th><th>Meadow</th><th>Soft winter wheat</th><th>Corn</th><th>Winter barley</th><th>Winter rapeseed</th><th>Spring barley</th><th>Sunflower</th><th>Grapevine</th><th>Beet</th><th>Winter triticale</th><th>Winter durum wheat</th><th>Fruits, vegetables, flowers</th><th>Potatoes</th><th>Leguminous fodder</th><th>Soybeans</th><th>Orchard</th><th>Mixed cereal</th><th>Sorghum</th></tr></thead><tbody><tr><td>Proportion in %</td><td></td><td>94.4</td><td>63.7</td><td>61.9</td><td>46.4</td><td>29.0</td><td>19.4</td><td>17.3</td><td>20.0</td><td>12.3</td><td>19.8</td><td>15.1</td><td>22.6</td><td>12.7</td><td>42.7</td><td>14.9</td><td>17.7</td><td>19.6</td><td>15.3</td></tr><tr><td>UT&T</td><td>53.5</td><td>97.1</td><td>78.3</td><td>77.5</td><td>63.1</td><td>57.5</td><td>25.4</td><td>49.2</td><td>54.9</td><td>69.9</td><td>50.7</td><td>50.5</td><td>50.6</td><td>22.0</td><td><strong>60.0</strong></td><td>59.0</td><td>46.3</td><td><strong>36.9</strong></td><td>13.8</td></tr><tr><td>CROMA</td><td>60.1</td><td>97.1</td><td>87.1</td><td>88.9</td><td>64.2</td><td>86.5</td><td>41.1</td><td>64.5</td><td>74.2</td><td>76.5</td><td><strong>51.2</strong></td><td>74.3</td><td><strong>61.3</strong></td><td>26.1</td><td>53.3</td><td>59.1</td><td>45.9</td><td>22.4</td><td>9.1</td></tr><tr><td>Scale-MAE</td><td>42.2</td><td>97.1</td><td>84.2</td><td>83.3</td><td>56.4</td><td>53.6</td><td>15.1</td><td>24.1</td><td>54.9</td><td>39.1</td><td>27.5</td><td>43.9</td><td>41.5</td><td>12.3</td><td>48.1</td><td>34.2</td><td>30.4</td><td>12.5</td><td>2.6</td></tr><tr><td>OmniSat</td><td><strong>69.9</strong></td><td><strong>97.2</strong></td><td><strong>90.1</strong></td><td><strong>92.2</strong></td><td><strong>82.6</strong></td><td><strong>89.0</strong></td><td><strong>64.7</strong></td><td><strong>75.0</strong></td><td><strong>82.7</strong></td><td><strong>88.1</strong></td><td>48.5</td><td><strong>81.3</strong></td><td>56.7</td><td><strong>52.6</strong></td><td>56.9</td><td><strong>85.7</strong></td><td><strong>48.4</strong></td><td>34.8</td><td><strong>31.7</strong></td></tr></tbody></table>

**Table A-2: Class-Wise Performance.** We report the F1-score for each class for TreeSatAI-TS, FLAIR, and PASTIS-HD for multilabel classification. We also report the unweighted class-averaged F1-score (Macro-F1). We can observe that OmniSat outperforms UT&T [31] and Scale-MAE [76] on nearly all classes for both datasets. In particular, we observe the most significant gains for classes with discriminative temporal dynamics, such as broadleaf tree species and the vineyards class.
<figure><img src="image_10.png" /><figcaption>**Fig. A-2: Qualitative Results.** We report predictions of OmniSat and two competing models on tiles from our datasets, including a failure case (bottom). OmniSat can detect classes with recognizable temporal dynamics such as agricultural lands or mixed forest areas with both coniferous ♀ and deciduous trees 🍂. Other methods, and in particular ScaleMAE, struggle to detect these classes.</figcaption></figure>