CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders
Authors: Anthony Fuller
Koreen Millard
James R Green
Venue: NeurIPS 2023 poster (2023)
Keywords: Remote Sensing
Earth Observation
Self-supervised learning
Multimodal
TL;DR: We jointly leverage cross-modal contrastive learning and masked autoencoding to learn SoTA representations for Earth Observation.

Abstract: A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples—aligned in space and time—and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to $17.6\times$ larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks—finetuning (avg.$\uparrow$ 1.8%), linear (avg.$\uparrow$ 2.4%) and nonlinear (avg.$\uparrow$ 1.4%) probing, $k$NN classification (avg.$\uparrow$ 3.5%), and $K$-means clustering (avg.$\uparrow$ 8.4%); and three segmentation benchmarks (avg.$\uparrow$ 6.4%). CROMA’s rich, optionally multimodal representations can be widely leveraged across remote sensing applications.

CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders
Anthony Fuller',; Koreen Millard', James R. Green' 'epaarmment of Systems and Computer Engineering 2 Department of Geography and Environmental Studies Carleton University, Ottawa, Canada
Abstract
A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes supe-ssue learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multi- modal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar aligned in space and time-ane and performs cross-modal contrastive learning. Another encoder fuses these sen- sors, producing joint multimodal encodings that used predict the masked patches via a lightweight decoder. We show that these objectives are comple- mentary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models effectively ex- trapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks- finetuning (avg.1 1.8%), linear (avg. 1 2.4%) and nonlinear (avg. t 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg.f 6.4%). CROMA's 's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.
Introduction
Deep learning has led to rapid advances in remote sensing, augmenting our ability to understand and monitor our planet. The remote sensing community has developed many application-specific deep learning models, specifically for satellite imagery: identifying heavily polluting brick kilns 2 or illegal airstrips [3]; monitoring deforestation [
5
6, 7 or crops [8] 9 10] detecting floods 11 12
or wildfires 13 [5]; even estimating houschold income [16 17 or poverty I8 19 20 Deep
learning-based remote sensing is playing growing role in tackling our climate crisis 22 22 23
Recently, researchers leveraged self-supervised learning to pretrain remote sensing models that can be employed these tasks, and more 24 25 26 27 Self-supervised methods invaluable for remote sensing, as there are petabytes of publicly available raw data from which to learn general representations, while only limited annotated data exists for downstream applications.
Self-supervised representations are often learned via contrastive approaches
or recon-
struction approaches [31 32 33 Contrastive approaches encourage the representations of positive pairs of samples- built by producing another view of a sample, for instance, from another sensor B4 or time (35), or by augmentations [30]-to be similar, and the representations of negative pairs to be dissimilar; this process can learn descriptive, object-focused representations. Models trained with
All correspondence should be addressed to Anthony Fuller: anthony www.errcarreeeoo.c..
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

contrastive objective learn to discard information not shared between views [7, regardless of the information's usefulness on downstream tasks; this makes the representations they learn sensitive to how positive pairs are built [30
38
39
40
Conversely, models trained with reconstruction or
autoencoding objectives- for instance, predicting hidden pixels [3] 41 learn to capture as much
information as possible [4]
43
44
Reconstruction approaches do not rely on multiple views and
but they learn representations that require significant finetuning to 45 Park et al. [47) show vision transformers (VITs (48) trained with contrastive learning focus more on shapes and wow-freuenccy information than ViTs trained with reconstruction approaches, which focus more textures and high-frequency information. They show that combining both objectives may achieve a sweet spot that learns better representations than either objective alone. Several other frameworks have been developed that leverage both objectives to learn however, none are designed for spatially aligned multimodal
scale incredibly well [1 45
46
be useful on downstream tasks 43
SoTA representations data.
49 50 51 52
Researchers developing foundation models for remote sensing have yet to take advantage of the multimodal data ubiquitous to remote sensing. For instance, the Sentinel missions the Earth's landmass multiple times per month since 2015-consse of multispectral optical imagery acquired by Sentinel-2 and Synthetic Aperture Radar (SAR) data acquired by Sentinel-1. By exploiting differences in how electromagnetic radiation interacts with Earth surface materials and measuring the radiation at many wavelengths (ranging from 440 to 2,200 nm), Sentinel-2 multispectral optical imagery can be used to characterize the material composition of objects [53]. By actively transmitting and receiving longer wavelength (5.5 cm) electromagnetic pulses, Sentinel-1 SAR can
be used to characterize the geometry, roughness, and electrical properties of objects
[4 These
Importantly for our work, they are spatially aligned, allowing multiple views of the same feature on the ground. Moreover, because data from the Sentinel missions are freely available, they have become the most widely used source of satellite imagery in research; thus, models with useful representations of Sentinel-1 & imagery can be immediately leveraged in scientific research, improving our ability to understand our planet.
modalities have proven complementary across remote sensing applications [55 56 57
These observations motivate Contrastive Radar-Optica Masked Autoencoders (CROMA): frame- work for learning rich representations of multimodal, spatially aligned. 2D data; which we leverage to pretrain ViT models Sentinel-1 & 2 data, providing the most valuable foundation models for Earth Observation, to date. We highlight three contributions: O CROMA significantly outperforms the current SoTA multispectral model, SatMAE (26). under an extensive evaluation. CROMA learns representations that are optionally multimodal (i.e., they can be effectively leveraged when either or both modalities are available) and extrapolate larger images at test-time (i.e., they can be effectively leveraged on images larger than those on which the model was trained). 2 CROMA's effectiveness is driven by two innovations: our complementary pretraining objectives and our novel relative position encoding (RPE) strategies that contribute to the quality of learned representations.
2 Method
In this section. "optical" refers to 12-channel multispectral optical imagery acquired by Sentinel-2, and "radar" refers to 2-channel SAR backscatter data acquired by Sentinel-1.
Architecture Background. Most work combining contrastive and reconstruction objectives learning 59
joint multimodal representations is in the image-text domain [51]
58
60
61
62
[3
[4
6S
;
heavily inspiring our architecture, Contrastive Captioning (CoCa 49 learns SoTA image- text
representations that are optionally multimodal. The CoCa framework consists of two unimodal
encoders-on for images, one for text-and multimodal decoder that receives text encodings at the bottom of the network and cross-attends to image encodings. CoCa is trained with two objectives: an image<-ttet contrastive objective between unimodal encoders and an image-captioning objective at the output of the multimodal decoder. Our framework significantly adapts CoCa to aligned multimodal 2D data by masking both modalities, introducing multimodal encoder, lightweight decoder (only used during pretraining, inspired by masked autoencoders (T]), and novel cross-attention and self-attention positional biases.
A unimodal radar ViT fx. that encodes radar inputs JR E 2x into I. patch encodings ER E BLLD i.e., &R = (R) @ A unimodal optical ViT fo that encodes optical inputs Lo e 1x into L patch encodings
Model Architecture. CROMA consists of three encoders (Fig. 1
2

---

Learning Representations vadoresoptical contrasnve leanning
Reconstruction Loss
Leveraging Representations
Multinodal Decoder
Breetoiing probing, clustering, and sxprert neighbor
nultioudal musked antorscoding
cluselfiction segmestation
MMM M MM
aptionally woliiode
larger imsages test time
ems--ttsiiie with sporii Ntwes
Multinodal Ficcoder
coss-iterion with, spelid Viace
Multimodal Engoder
JOL
pool
Mulrispectral Optical Encoder
Multispectral Optical Encoder
Radar Encoder
Rodar Encoder
Contrastive Loss
Figure 1: (Left) Our CROMA framework jointly leverages radar contrastive learning and masked autoencoding to learn rich, self-supervised representations. (Right) Leverage representations on: unimodal or multimodal data, larger images at test-time, and diverse tasks and methods.
Eo E RIXDD i.e., Eo = fo(lo). @ A multimodal radar- optical transformer fRO that encodes L radar- optical patches, ERO E R LxD i.e., ERO = FRO(EE Eo). For each set of unimodal encodings, build full-image representations R E IR by processing the mean pooled patch encodings through a feedforward network (FFNN, i.e RR FFN& (MeanPool(ER)) and Ro = FF (MeanPool(EEn) Our patch size is 8 x 8 pixels for both modalities, and our default image size is 120 x 120 pixels. Our radar encoder has N/2 transformer layers, and our optical encoder has N transformer layers (N is 12 for ViT-B and 24 for VIT-L backbones). All unimodal encoder layers are composed of self- attention and FFN sublayers. Our multimodal 1/22-layer attention, and FFN sublayers-encode both modalities into single sequence of L patch encodings; this encoder receives radar patch encodings the bottom of the network and learns multimodal representations by cross-attending to optical patch encodings. Multimodal representations can be built via pooling multimodal patch encodings, i.e., Rgo McanPool Our ViT backbones do not use sinusoidal position embeddings; instead, we bias the self-attention and cross-attention matrices with the distances between patches.
of self-attention, cross-
ALiBi Background. ALiBi (66) is simple and intuitive RPE method for transformers that biases the self-attention matrix based on the distance between tokens in ID sequence. Each self-attention head receives positional biases with different strengths, called slopes m. With 16 attention heads, the geometric sequence defines these scalar slopes starting at vद. i.c., 2015 st ,..r.... ..... # Biases are subtracted from the attention matrix before the softmax is calculated. Specifically, the sre-ssotmax attention matrix A 1 (h is the number of heads, L is the sequence length) is populated with attention scores anij for the ith query qhi E R and jth key knj E TR (d is the head dimension):
ahij = Vd qhi knj. without ALiBi; and Gnij = Vd qui knj - distance(i, (i. mm),, with ALiBi. Crucially, ALiBi does not add position embeddings the bottom of the network; instead, the relative positions between tokens are encoded in the attention matrix itself. To date, ALiBi is the only position encoding method for transformers that has been demonstrated to extrapolate at test-time to sequences far longer than those on which it was trained.
k 2o kg K4 Kg Kg ky Kg kg
2D-ALiBi and X-ALiBi.
We extend ALiBi to 2D inputs by biasing the self-attention matrix based on the Euclidean distance
41
0.0
1.0 2.0 1.0 2.2
Q2
1.0
1.001.0110 1.4
1.4
2.2
between query-key pairs ViT-we call this 2D-ALiBi (Fig.
C3
2.0
2.2
1.4
1.0
2.8
2.2
2). We also extend ALiBi to cross-attention by biasing the cross-attention matrix based on the Euclidean distance between
1.
0.0
1.00020
1.0
95 1.41..1
1.0
1.444
call this X-ALiBi. (In cross- attention (67], queries are built from the previous layer, whereas keys and values are built from optical encodings.) For both 2D and X-ALiBi, we calculate our attention head slopes with the same geometric sequence as ALiBi. Since our two modalities
cross-modal query-key pairs-
2.2.244400
1.0
1.0
47 2.02.22..
1.4
2.0
Q8
1.4 2.2
1.0
2.8 2.2 2.0 1.4 1.0
1.0
Figure 2: 2D-ALiBi matrix for an
image with 9 patches (32 before flat- tening): g is for query, k is for key, and m is a fixed scalar that biases attention heads at different rates.
3

---

are aligned 2D sensor data, our 2D-ALiBi and X- matrices are identical. The primary motivation of 2D-ALiBi to learn representations that can generalize across image sizes; this is particularly useful in remote sensing and is likely useful in other domains. The primary motivation of X-ALiBi is to improve sensor fusion by inserting positional information the cross-attention sublayer of our multimodal encoder, SRO- These position encoding techniques are rotation and translation invariant, which are desirable properties for the overhead imagery in Earth Observation.
MAE Background. A masked autoencoder (MAE 31 rearranges an image into a sequence of non-overlapping patches, then randomly samples large portion of patches to be held-out. The "visible" patches, that are not held-out, are encoded by ViT. This MAE-style masking is ingenious: leverages sparse computation while only requiring dense operations that run efficiently on modern hardware. MAE introduces lightweight decoder that receives visible patch encodings and hidden mask embeddings, which are both added to 2D-sinusoidal position embeddings. The decoder outputs predictions of the pixel values of the held-out patches. Both the encoder and decoder are pretrained end-to-end to minimize the mean squared error between patch predictions and the originally held-out patches. The pretrained encoder can then be leveraged on downstream applications.
Reconstruction Objective. We independently mask 75% of radar and optical patches and en- code the unmasked patches with our three encoders, i.e., & = [x(Ix"",, ), E0TH = fo( and Eam (Eg _ where um means unmasked. We introduce lightweight 1- layer transformer decoder fDEC that receives multimodal patch encodings and mask embed- dings after adding 2D-sinusoidal position embeddings and predicts a target image, i.c., i = (McNCCA([[ Emb + Emppesition). We split the channels of to form predictions for each sensor, lo and lR; the loss is only applied at the locations of the masked-out patches:
N
M
Norm(T")))
ΣS
M
Norm(('))
.MAE = N
1
Σ
+
M
M
optical reconstruction
radar reconstruction
where N is the batch size, M is the number of masked patches, and Norm sets the mean to 0 and standard deviation to 1 for target patches (following MAE). Along with learning unimodal repre- sentations, this objective spatially fuses our two sensors, i.e., it builds multimodal patch encodings that represent information from both sensors in the patch, corresponding to an 80 m x 80 m square on the ground (8 8 patches at 10m resolution). Finally, 2D and X-ALiBi can be easily adapted to MAE-style masking by removing the masked-out columns and rows from the bias matrix.
Contrastive Learning Background. Contrastive learning aims to classify the correct pairings of samples derived from a batch. Logits are formed by measuring the similarity between the projected representations of samples. As result, the representations of positive pairs are pulled together, and the representations of negative pairs are pushed apart. Very recently, FLIP 68 performs contrastive learning with masked-out representations via MAE-style masking. This speeds up pretraining and enables larger batches due to the reduced memory per sample. FLIP performs on par with CLIP [69]- the foundational work that learns rich representations via images+text contrastive learning- but can be pretrained at half the cost.
We perform radar contrastive learning across the unimodal repre- sentations of our masked-out sensor data using the InfoNCE loss (28). For an optical anchor image, the positive sample is the geographically and temporally matched radar sample, and the negative samples are all other radar samples from the batch; likewise, our radar representations are pulled towards (positives) or pushed apart (negatives) from optical representations:
Contrastive Objective.
Liso
1
Σ
N
log
exp
2R
TT 26/0r
Σ log
exp (20 2
2NV
1iT 20/0
Σ exp (o =//0))
exp
radar-to- to-trcaa
optical-to-rade -adarr
where ER and 20 are lg normalized linear projections of radar and optical representations, respectively, i.., øR Norm form(Linearg, (RR)) and zo = Norm(Linearo (Ro)). is the softmax temperature, and

---

N is the batch size, Crucially, we only encode small portion of input patches, which form our representations. This masking provides advantages: it enables larger batches, speeds up pretraining. and enables our multimodal reconstruction objective with the same computational graph. This radare-opticcal contrastive objective encourages representations to be sensor-invariant i.e., to capture information shared between sensors.
Combined Pretraining Objective. We combine contrastive learning and masked sensor modeling pretraining objectives: -= ACon LCon + AMAE We set both task weights (i.e., Aoon and XMAB) to and ablate them in Appendix SA.1
3 Experiments
Pretraining. We pretrain CROMA models on the SSL4EO dataset 77 - large geographically and seasonally diverse unlabeled dataset. 4SI consists of 1 million paired Sentinel-1 GRD & Sentinel-2 1.2A samples of 264 x 264 pixels. Sentinel-1 channels consist of VV and VH backscatter. Sentinel-2 channels consist of 12 surface reflectance multispectral bands (the cirrus band is removed). We pretrain CROMA-B (ViT-B backbone) for 300 epochs and CROMA-L (ViT-L backbone) for 600 epochs. Crucially, single pretraining run trains all three encoders (optical, radar, and joint radar-optical) end-to-end; users can then finetune one or multiple encoders, depending on their task and data availability. We perform all pretraining experiments on an NVIDIA DGX server (8x A100-80 GB), including ablations. Please see Appendix $A.3 for more details.
Comparisons. We compare CROMA to all available multispectral optical foundation models, which include two models pretrained by 71] using radart>oppticl contrastive learning: two models pretrained by [70] using the MAE 31 and DINO (72) frameworks; and two models pretrained by [26] using their multispectral representation learning framework, SatMAE. We also compare CROMA to SoTA method for learning visual representations of natural -angge--maae joint embedding predictive architecture (I-JEPA, 1 we leverage to pretrain a ViT-B model on SSL4EO's s optical imagery for 300 epochs. To enable fair comparison between models, we evaluate all models under identical conditions and This is necessary because the originally reported results of these models occurred under inconsistent evaluation conditions for instance, data splits or training data amounts. We use the latest publicly available models for all evaluations and preprocess data according to official repositories. For radar and radar-optical datasets, we compare CROMA to SatViT-V2 74). a model pretrained using MAE 31 stacked Sentinel-1 1 & 2 imagery; and DeCUR, model pretrained coorained-cooncureentty with this work-by 77 using their multimodal representation learning framework.
budgets (please see Appendix $A.4.2 for details).
3.1 Multispectral Optical Experiments
Classification Setup. We evaluate CROMA by finetuning, frozen linear and nonlinear probing. ANN classifyyng, and K -means clustering pretrained representations across four Sentinel-2 classification benchmarks. 1 The multi-label BigEarthNet dataset [76] (35,420 train samples and 118,065 vali- dation samples); this is 10% of the complete BigEarthNet training set that is now used by default 25 (26] to reduce the costs of finetuning and is better suited for remote sensing benchmark (22). dataset 26 (71,287 train samples and 84,939 validation samples); this is also 10% of the complete training set. Following BigEarthNet's use, believe this smaller training set is a more appropriate benchmark for model evaluation, but we show results on the complete training set in Appendix 1 The EuroSAT dataset (77) (16,200 train samples and 5,400 validation sam- ples). The Canadian Cropland dataset (78 (53,884 train samples and 23,088 validation samples); this is a new benchmark inspired by EuroSAT but is more challenging, as the crop types (barley, canola, corn, etc.) can be visually similar. For finetuning and linear probing, we add a linear layer for these tasks atop the full-image representations, i.c., Linear (Ro). For nonlinear probing, we use an MLP with one hidden 2048-d layer, i.e., Lincar(ReLU (Linerr (Ro))). Additionally, we perform non-pan non-paamettric I:N classification (k- 20) and K-means clustering for single-label benehmarks to evaluate frozen representations. 779 shows that no single method of evaluating representations the best; they recommend including kNN and K-means alongside linear probing. Other studies [80 72 81 show a rank mismatch between KNN and linear probing evaluations Sanlutiios-indiicating they offer complementary estimates of representation quality. Please see Appendix $A.4. and A.4.2 for implementation, data splits, and parammee details.
The fMoW-Sentine
5

---

Table I: Classification results on four benchmarks, under finetuning (FT), and frozen linear (LP) and nonlinear (MIP) probing. * denotes originally reported results; we obtain all other results under identical conditions.
BigtarthNet (10%)
IMoW-Sentinel (10%)
EaroSAT Top Acc.
Canadian Cropland Top Acc.
mAP
Top 1 Ace.
Method
Backhone
FT
MI.P
T.P
FT
MI.P
1.P
FT
MI.P
L.
FT
MI.P
1.P
tadaresuptiaal
ResNet50
77.65
6.01
radureoopicall
Swin T
86.41
78.65
77.93
52.01
28.54
31.06
98.09
93.50
94,78
70.98
60.36
57.05
MAE 31
70
VII-S
86.15
81.70
75.94
$1.79
31.70
27.69
98.78
94.46
91.80
74.02
59.07
48.38
DINO 72
70
VII-S
87.04
84.96
81.58
$2.79
35.62
32.64
98.63
97.07
96.07
75.27
67.35
59.94
I-JEPA
73
NIT-B
85.92
84.27
SatMAE 26
VIT-B
85 94
83.48
79.36
57.20
37.28
35.17
99.20
97.28
96.61
73.58
66,02
60.40
CROMA
VIT-B
87.58
86,29
85.04
54.47
39.67
38 42
99.22
97 97.89
97.59
76.17
67.62
63.39
SarMAE 26
VIT-
86.18/
84.01
80,29
58.19
38.18
36.76
99.35/
97.67
97.65
74.06
67.03
61.75
82.62
98. 98
CROMA
VIT-L
88.29
86.46
85.01
59.02
40.07
39.17
99.46
98.04
98.02
78.07
67.94
64.02
Classification Results. CROMA ranks
Table 2: Von-parametric ANN classification and K- means clustering results. denotes 10% of the training
first, averaged across four Sentinel-2 (mul- tispectral optical) benchmarks, under fine- tuning. frozen linear and nonlinear probing, kNN classification, and K-means cluster- ing (Table 2) The only case where a Sat- MAE model outperforms CROMA model of the same backhone is finetuning on the fMoo-Sentinel dataset. However, Sat- MAE was pretrained on -MoW-Sentinel- meaning there no distribution shift be- tween pretraining and finetuning. This gives SatMAE an advantage on the fMoW- Sentinel benehmark since downstream per- formance impacted by the similarity be- tween upstream and downstream data [22
[MoW Sent. Sen.
EuroSAT
Can. Crop.
Top I Acc.
Top I Acc.
Top Ace.
Method
Backhone
RNN
K- means
ANN
K- ANN
K.
means
means
raduresoptical eudaresoptical
ResNet50
7.07
3.94
52.09
23.52
49.82
23.15
71
Swin-T
19.46
7.93
85.07
56.91
48.22
21.57
MAE
31
70
VITS
22.25
7.54
87.33
41.50
56.01
18.31
72
ViT-S
IJEPA 73
VIT-B
25.45
8.01
89.02
42.89
57.48
18.06
SaMAA 26
VIT-B
26.76
8.98
89.28
44.87
56.20
20.14
CROMA
VII-B
32.03
11.35
95.26
76.06
61.25
22.55
CROMA
WI-L
29.54
10.12
94.70
61.24
59.41 21.27
Despite this observation, CROMA-B outperforms SatMAF-B on MoW--Sentine under linear probing (1 3.3%), nonlinear probing (f 2.4%), ANN than 4x faster during fine-
83 84
(1 5.3%), and K-means 2.4%). Additionally, CROMA models tuning and inference than their SatMAE counterparts (please see Appendix 44.4.2 for a comparison). On all evaluations, CROMA outperforms a ViT-B model pretrained using the I-JEPA framework on the SSL4EO dataser- I-JEPA is the current SoTA framework for learning self-supervised represen- tations of ImageNet 773 We also plot UMAP separation of EuroSAr AT classes.
85 embeddings (Fig. 3); CROMA shows strong
CROMA (VTT B): 76.0%
SaIMEE (VTT B): 44.9%
DINO (VIT S): 45.8%
radar rrdpticoptical (Swin T): 56.9%
Permanent Crup Sea & Lake
Residential
Industrial
Highway Anmual Crop
Pusture Herhaceous Vegetation
Forest
River
Figure 3: UMAP embeddings and K-means clustering accuracies of CROMA (ViT-B), SatMAE (ViT-B) 26). DINO (ViT-S) [70], and radaca (Swin-T) [71] models EuroSAT (77).

---

Sparse Probing. Inspired by the sparse probing of language models [86], we sparsely probe the 768-dimensional (ViT-B backhone) image representations by restricting the linear probe to & dimensions. For each class, we rank all dimensions and then train a linear probe on the top k to perform binary classifi- cation (this setup follows [86], please see Appendix $AA.4.2 for more details). This experiment helps us understand the infor- mation contained in the learned representations. For example, in Fig. 4 (B), there is large drop in Fl score when decreas- ing k -iddicaatinng that BigEarthNet's "beaches, dunes, sands" class is not well represented by individual dimensions. Rather, this class is represented by a composition of many features that correlated with beaches. Conversely, BigEarthNet's "agriculture with natural vegetation" class maps well to single dimension of CROMA's representations, achieving F1 score of 49% when k = (Fig. 4 (A)). In Appendix { we show sparse probing results for all classification tasks and for all classes.
(A) Agriculture
0.6
Fl Score
0.4
CROMA SaMAA
I-JEAA
0.2-
(B) Beaches,
dunes, sands
10
10
10a
10a
k
Figure
Sparse probing two BigFarthNet classes.
Segmentation Setup. We evaluate all ViT-based models on three Sentinel-2 segmentation bench- marks. O The DFC2020 dataset [87] (46,152 train samples and 8,874 validation samples). Ø A subset of the Dynamic World dataset (88] that was annotated by experts; hence we call it DW-Expert (20,422 train samples and 51,022 validation samples). DW-Expert is new, high-quality benchmark that was annotated with the help of high-resolution satellite and street-level imagery. 2 The MARIDA dataset [89] (1,682 train samples and 1,615 validation samples), which is small, sparsely labeled dataset marine debris. For all tasks, we linear probe frozen patch encodings, i.e., Lincar (Eo). We crop images of 96 96 from the original images (256 256 for DFC2020, 510 x 510 for DW-Expert, and 256 x 256 for MARIDA). We train and evaluate all models on these 96 x 96 images, which is the default image size for SatMAE, enabling fair comparison with the SoTA.
Segmentation Results CROMA outperforms Sat- MAE by averages of 5.4% and 6.4% for ViT-B and ViT-L backbones, respectively (Table 3.. These re- sults demonstrate that CROMA effectively learns fine- grained patch-level features useful for dense predic- tion tasks like semantic segmentation.
Table 3: Semantic segmentation (mloU) re- sults on three Sentinel-2 benchmarks.
Methed Backbone DFC2020
DW-Expert MARIDA
MAE
VIT-S
35.63
46.63
48.06
DINO
VIT-S
48.34
49,38
IJEPA
VIT-B
36.72
50.82
53.85
SaMMAE
VIT-B
45.53
51.03
58.17
CROMA
VITBB
46.67
58.55
65.56
SaMAR
VIT-I.
51.50
$7.12
3.2 Radar and Radar-Optical Experiments
CROMA
VIF-
49.78
58.71
63.32
Multimodal Setup. BigEarthNet and DFC2020 also contain spa- tially and temporally aligned Sentinel-1 samples alongside Sentinel- 2. This allows us to evaluate our multimodal representations and directly compare them with optical-only representations on the same benchmark. We linear probe frozen image representations for BigEarthNet, i.e.. Lincar Lincar(CONCATT (CONCAT (RR. Ro. Ruo)), and patch en- codings for DFC2020, i.e., Linear CONCAT(ER. &o, ERO)). Con- current with this work, DeCUR (75) learns radar and radar-optical representations via their novel multimodal framework. They fit a linear probe on top of DeCUR's 's representations on 1 1% of the Bigharth training set and report results on the BigEarthNet validation set; we follow this experimental procedure with our CROMA-B model for direct comparison.
Table 4: Multimodal (Sentinel- & 2) linear probing results on classification and segmen- tation benchmarks.
BigEarthNer
DFC2020 mlol!
Model
mAP
SaIVIE-22 74
79.80
46.20
CROMA
86.24
$1.58
CROMA-L
53.24
Multimodal Results. CROMA significantly outperforms the joint multi- modal representations learned by SatViT-V2 [74] (Table Our joint multi- modal representations outperform our optical-only representations by 1. 1.2% on BigEarthNet (both CROMA-B and CROMA-L) and by 4.9% (CROMA- B) and 3.5% (CROMA-L) on DFC2020 (Table 3 justifying our multimodal approach. CROMA'S radar-only and radar-optical represen- tations also outperform the concurrent DeCUR under a linear probing experiment (Table
Table 5: Linear prob- ing radar and radar- optical representations.
Bag (1%) mAP
Method Radar Radar-Opt.
DECUR 73.7
79.4
CROMA
75.7
81.8
7

---

Table 6: Linear probing ablation results on radar-only ("R"), optical-only ("O"), and joint radar- optical ("RO") inputs. We consider both performance and cost to select our design. All rows below "all default" report the performance differences between ablated cases and the default.
batch size
Clusiifictiinn (mAP)
Segmentaatin (mloU)
Case (default)
Ablation
cost
R
RO
R
0
RO
Avg
all default
7.2k
1.0x
78.2
84.5
84,8
40.8
56.0
56.5
66.8
objectives (both)
MAF-only
7.2k
1.0x
-10.4
-6.0
-5.6
-9.2
-8.4
-5.1
-7.5
contrast-only
14k
10.6%
-3.2
-2.9
position emcrding (2D-ALiBi X-ALiBi)
2D-ALiB
7.2k
1.0%
-0.2
-0.2
-0.3
0.1
0.0
-0.9
-0.2
PEG 90)
6.9k
1.0%
-0.3
0.0
-0.1
-0.6
-1.0
-1.9
-0.7
2D-simusodaal
7.2k
-1.3
-2.
shared 25%
2.4k
3.0x
-0.5
-1.0
-1.1
1.1
0.5
-0.2
-0.2
masking (independent 75%)
50%
3.7k
-0.3
-0.7
0.6
0.1
0.0
3
shared 75%
7.2k
-0.8
-0.3
0.1
0.0
-0.2
independent 50%
3.7k
1.8%
-0.2
-0.5
-0.5
0.9
0.7
0.3
0.1
target
radar-only optical.only
7.2k
-0.2
(radar optical)
7.2k
1.0%
-0.1
-0.2
-0.2
-0.1
-0.1
-0.6
-0.2
5
MAE decodar (depth=1, dim=512)
depth=6, dim=768
4.3k
1.7x
0.1
-0.1
0.1
0.2
0.2
0.1
scale
ViT-B, epochs=300
7.2k
3.0%
1.6
0.5
0.4
1.7
1.3
0.5
1.0
(VTT-B, e0ochs=
I., epochs=600
3k
15x
2.5
0.4
0.5
0.9
0.1
1.2
4
Ablation Analysis
4.1
CROMA Design
Ablation Setup. We ablate the CROMA design by pretraining CROMA-B models for 100 epochs unless stated otherwise. For each ablation, we set the maximum batch size that can fit into 640 GB of VRAM (with 6 precision); adjusting the batch size results in better comparisons between approaches. For classification on BigEarthNet, we linear probe frozen unimodal and multimodal representations, i.c., Linear(Ra, Linear (Ro),, Linear (Roo). For segmentation DW-Expert 120, we linear probe frozen patch encodings, i.c., Linear(Ex). Linear(Eo)) Linear E.o) For BigEarthNet, train the same 10% split as $3.1 but report results the combined validation and test sets (236, 130 samples). For DW-Expert-120, we select 120 x 120 images (CROMA's default image size) from Dynamic World's [88] original Sentinel-2 images and match them, in space and time, with Sentinel-1 images-forme a high-quality multimodal segmentation dataset. DW-Expert-120 consists of 10,200 train and 45,367 validation samples.
bllaionn Results. Removing either self-supervised objective significantly degrades accuracy on both classification and segmentation tasks. This justifies fundamental design choice, i.c., combining contrastive and reconstruction approaches for aligned multimodal data. 2 Although our primary motivation for 2D-ALiBi is enable input size extrapolation (see $4.2). we find it also improves linear probing accuracy over a SoTA RPE method for ViTss, even when no extrapolation occurs. Averaged across six evaluations, 2D-ALiB without X-ALiBi outperforms Position Encoding Generator (PEG, (00) by 0.5% and 2D- sinuiiuso embeddings by 1.5%. (We adapt PEG to MAE-style masking by zero-filling the masked-out patches during pretraining, inspired by 191). Leveraging X-ALiBi with 2D-ALiBi further improves average performance by 0.2% (Table @ppatticularly improving multimodal performance, which is our motivation for X-ALiBi. We believe there are two reasons for 2D-ALiBi's superior serformance--vvidence for both is provided in Appendix $A.2 First, 2D>-Aiii learns inationninvarianl representations despite not being trained with rotation-invariant objectives; this is desirable property of satellite imagery that likely improves classification performance. Second, 2D-ALiBi prevents patch-wise representational collapse (i.e., the representations of patches within an image become similar, losing local information) often observed with contrastive objectives 47]: preserving patch-wit wiihe-wiie diversity likely improves segmentation performance. ① Lower mask ratios hurt classification and help segmentation but increase costs. For both 50% and 75% mask ratios, independently masking our modalities (i.e., optical and radar samples are masked differently) slightly outperforms shared masking. Although 50% independent masking outperforms 75% masking by 0.1%, we select the higher mask ratio because it offers 1 1.8x speedup. O multimodal target (14 channels) outperforms optical-only target (12 channels) by 0.2%, which outperforms radar-only
8

---

target (2 channels) by 0.3% (Table this implies that reconstructing more information leads to more general representations. A multimodal target especially important for learning rich multimodal representations. $ A larger decoder improves segmentation but costs 1.7x more. MAF showed that a deep decoder improves linear probing accuracy [1I, but CROMA is very robust to decoder size. Therefore, we select the most efficient, i.e., a 1 1-layer, 512-d decoder. @ We design CROMA by pretraining ViT-B models for 100 epochs and consider linear probing performance and cost. Thus, our design may not be optimal when scaling up. Nevertheless, scaling to more epochs and larger model improves representations, especially radar-only representations.
In Appendix A we also experiment with all showing minimal or negative impacts iapacts-task task weights, more decoder sizes, VICReg [92]. mean squared error loss between cross-modal patch encodings (inspired by 933, ) InfoNCE loss between cross-modar patch encodings (inspired by [94)), hard-
negative mixing and lower-masked tuning at the end of pretraining (68)
4.2 Extrapolating Representations to Larger Images
Extrapolation Setup. For three CROMA-B models
PEG 190]. and 2D-ALiBi) pretrained for 100 epochs, we finetune all parameters (along with linear head) on Sentinel-2 images from DW-Expert-1 120 (120 x 120 pixels). Then, we directly evaluate the models images of varying sizes to test their ability to extrapolate at test-time. We create validation sets with different image sizes by cropping from the original 510 x 510 images in Dynamic World [88] Regardless of image size, we retain the 10m per pixel spatial resolution on which the models were trained-extrapollee smaller or larger geographic areas test-time. For 2D-sinusoidal embeddings, we also evaluate an
embedding interpolation algorithm often used when applying ViT' on larger images 3 26 48 96
Extrapolation Results. 2D-ALiBi outper- forms PEG by 1.7% on 504 x 504 pixel only see
Table 7: Segmentation results (mloU) of models trained on 120 120 resolution and evaluated on various reso- lutions.
images (Table 7) Amazingly, a 0.7% drop in mIoU when testing on ar- 17.6 x larger than those which our model was trained-effectivee effectively generaliz- ing from 225 to 3,969 patches per image. We achieve this by extending ALiBi [66] to 2D inputs by penalizing attention scores based on the Euclidean distance between
Train Ros.
Method
Test Res.
48
96
120
224
384
448
504
Simssiall
54.4
58.5
59.2
38.2
24.9
21.8
19.5
2D-Sinusoidal w/ interp.
50.7
58.6
59.2
58.3
562
55,5
54.9
PEG (90)
55.1
58.0
58.3
58.5
57.7
57.4
57.1
AD.
56.3
59.1
59.0
58.8
query-key pairs. We may achieve even better results by encoding directions in a subset of attention heads or learning scalars [97]: we leave these investigations to future work. We believe X- and 2D-ALiBi have tremendous potential beyond our CROMA framework, for instance, by extending these methods to additional modalities, viewing angles, or 3D data.
5 Related Work
Remote Sensing Representations. Deep learning for remote sensing has been an active research area for many years. Researchers have leveraged self- supervueee learning frameworks in the last few years to learn representations of remote sensing data that can be widely leveraged across societally important downstream tasks. In designing contrastive pretext tasks, remote sensing researchers built positive pairs by leveraging spatial [98 99
100.
101
102
103
104
105 [06 107 [08], temporal
24 25 109], spectral [10], or modal 111
112
113
114
115
116 information. In designing
reconstructive pretext tasks, researchers held-out spectral bands 117 pixels 12) and
resolutions [27]. However, these studics including the concurrent Scale-MAE [27] and influential studies tile2vec [08). GASSL (24), and SeCo [25] [25e- either performed at smaller scales or only included wavelengths from the visible (red, green, and blue) spectrum; this limits their utility on downstream applications since wavelengths from the non-visible spectrum contain information
critical to many remote sensing tasks [53] 122 123 124 For example, the ability to measure the reflectance of objects in the near-infrared portion of the wavelength is extremely valuable in applications related to vegetation identification [25], health and productivity [26], as well as
identifying water bodies [127, soil moisture 128 and vegetation water content 129
Relative Position Encoding for ViTs. SoTA transformers in natural language processing use RPE 130 131 [32]. but fixed 2D-sinusoidal embeddings still predominate ViTs. The improved inductive

---

bias of RPE over absolute position encoding can offer improved performance and, sometimes, the ability extrapolate larger images at test-time, i.e. directly applying a model trained on one image size to another without further training. This extrapolation ability would add considerable value to remote sensing foundation models since image sizes vary widely-images are often cropped from large scenes down to smaller sizes chosen idiosyncratically. Positional Encoding Generator (PEG, (90)) is a SoTA RPE method that uses a convolution between ViT layers; PEG was demonstrated to significantly outperform other RPE methods when tested on larger images. iRPE [133 is another, more complex, RPE method for ViTs; however, it demonstrates no extrapolation ability. We found no prior work that leverages RPE in cross attsnnnt
Conclusion
We propose a novel framework for learning unimodal and multimodal representations for Earth Observation by jointly leveraging contrastive and reconstruction self-supervised objectives. We extend a SoTA position encoding method for ID sequences to 2D inputs and cross-attention; to the best of our knowledge, this is the first time explicit position encoding has been leveraged in cross-attention. These strategies allow our models to extrapolate to larger images at test-time and improve performance on both unimodal and multimodal data. We extensively evaluate our pretrained models on diverse tasks and methods, outperforming the previous SoTA. Although our method is designed for multimodal satellite imagery, it can be leveraged on other applications that offer spatially aligned multimodal data, for instance, medical imaging or autonomous vehicles.
The main limitation of our work is our focus we will explore other sensors that offer higher spatial or spectral resolutions and time-series data. Despite this limitation, Sentinel-1 & 2 are the most widely used sources for satellite imagery in research-makee our models an incredible resource for the remote sensing community and users of remote sensing-derived products, for instance, geographers, economists, or environmental scientists.
matic-in-time Sentinel-1 & 2 data; in the future,
Acknowledgements
This work was made possible with compute provided by Cyxtera Technologies and the NVIDIA Academic Hardware Grant Program. Anthony thanks the Vector Institute for their financial support during his master's program.
References
[1] Jihyeon Lee, Nina R. Brooks, Fahim Tajwar, Marshall Burke, Stefano Ermon, David B. Lobell, Debashish Biswas, and Stephen P. Luby. Scalable deep learning to identify brick kilns and aid regulatory capacity. Proceedings of the National Academy of Sciences, 2021.
[2] Arati Paul, Soumya Bandyopadhyay, and Uday Raj. Brick kiln detection in remote sensing imagery using deep neural network and change analysis. Spatial Information Research, 2021.
[3] Fen Chen, Ruilong Ren, Tim Van de Voorde, Wenbo Xu, Guiyun Zhou, and Yan Zhou. Fast Automatic Airport Detection in Remote Sensing Images Using Convolutional Neural Networks. Remote Sensing. 2018
[4] Raian V. Maretto, Leila M. G. Fonseca, Nathan Jacobs, Thales S. Körting, Hugo N. Bendini, and Leandro L. Parente. Spatio- Temporl Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters, 2021.
[5] Seong-Hyeok Lee, Kuk-Jin Han, Kwon Lee, Kwang-Jae Lee, Kwan- Young Oh, and Moung-Jin Lee. Classification of Landscape Affected by Deforestation Using High- Resolueeo Remote Sensing Data and Deep-Learnig Techniques. Remote Sensing. 2020.
[6] Pablo Pozzobon de Bem, Osmar Abílio de Carvalho Junior, Renato Fontes Guimarães, and Roberto Arnaloo Trancoso Gomes. Change Detection of Deforestation in the Brazilian Amazon Using Landsat Data and Convolutional Neural Networks. Remote Sensing, 2020.
10

---

[7] Bruno Menini Matosak, Leila Maria Garcia Fonseca, Evandro Carrijo Taquary, Raian Vargas Maretto, Hugo do Nascimento Bendini, and Marcos Adami. Mapping Deforestation in Cerrado Based on Hybrid Deep I learnigg Architecture and Medium Spatial Resolution Satellite Time Series. Remote Sensing, 2022.
[8] Hannah Kerner, Gabriel Tseng, Inbal Becker- Reshef, Catherine Nakalembe, Brian Barker, Blake Munshell, Madhava Paliyam, and Mchdi Hosseini. Rapid Response Crop Maps in Data Sparse Regions. CoRR, abs/2006. 2020.
[9] Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov. Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data. IEEE Geoscience and Remote Sensing Letters, 2017.
[10] Foyez Ahmed Prodhan, Jiahua Zhang, Fengmei Yao, Lamei Shi, Til Prasad pangali Sharma, Da Zhang. Dan Cao, Minxuan Zheng, Naveed Ahmed, and Hasiba Pervin Mohana. Deep Learning for Monitoring Agricultural Drought in South Asia Using Remote Sensing Data. Remote Sensing, 2021.
[11] R. BBentivoglio, E. Isufi, S. N. Jonkman, and R. Taormina. Deep learning methods for flood mapping: review of existing applications and future research directions. Hydrology and Earth System Sciences, 2022.
[12] Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy. FloodNet: High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding. IEEE Access, 2021.
[13] Fantine Huot, R. Lily Hu, Nita Goyal, Tharun Sankar, Matthias Ihme, and Yi-Fan Chen. Next Day Wildfire Spread: A Machine Learning Dataset to Predict Wildfire Spreading From Remote-Sensing Data. IEEE Transactions on Geoscience and Remote Sensing, 2022.
[14] Dmitry Rashkovetsky, Florian Mauracher, Martin Langer, and Michael Schmitt. Wildfire Detection From Multisensor Satellite Imagery Using Deep Semantic Segmentation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021.
[15] Panagiotis Barmpoutis, Periklis Papaioannou, Kosmas Dimitropoulos, and Nikos Grammalidis. A Review Early Forest Fire Detection Systems Using Optical Remote Sensing. Sensors, 2020.
[16] Arman Khachiyan, Anthony Thomas, Huye Zhou, Gordon Hanson, Alex Cloninger, Tajana Rosing, and Amit K. Khandelwal. Using Neural Networks Predict Microspatial Economic Growth. American Economic Review: Insights, 2022.
[17] Esra Suel, Samir Bhatt, Michael Brauer, Seth Flaxman, and Majid Ezzati. Multimodal deep learning from satellite and street-level imagery for measuring income, overcrowding, and environmental deprivation in urban areas. Remote Sensing of Environment, 2021.
[18) Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning understand economic well-being Africa. Nature Communications, 2020.
[19] Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping. In AAAI, 2016.
[20] Luna Yue Huang, Solomon M Hsiang, and Marco Gonzalez-Navarro. Using Satellite Imagery and Deep Learning to Evaluate the Impact of Anti-Poverty Programs. Working paper, National Bureau of Economic Research, 2021.
[21] David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman- Brown, Alexandra Sasha Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Kording, Carla P. Gomes, Andrew Y. Ng. Demis Hassabis, John C. Platt, Felix Creutzig, Jennifer Chayes, and Yoshua Bengio. Tackling Climate Change with Machine Learning. ACM Comput. Surv., 2022.

---

[22] Alexandre Lacoste, Evan David Sherwin, Hannah Kerner, Hamed Alemohammad, Björ Lütjens, Jeremy Irvin, David Dao, Alex Chang, Mehmet Gunturkun, Alexandre Drouin, Pau Rodríguez, and David Vázquez. Toward Foundation Models for Earth Monitoring: Proposal for Climate Change Benchmark. NeurlPS Workshops, 2021.
[23] Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources. IEEE Geoscience and Remote Sensing Magazine, 2017.
[24] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography- Self-Supervised Learning. ICCV, 2021.
[25] Oscar Mañas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodríguez. Seasonal Contrast: Unsupervised Pre-Training From Uncurated Remote Sensing Data. In ICCV, 2021.
[26] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon. SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery. In NeurlPS, 2022.
[27] Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning. CoRR, abs/2212.14532, 2023.
[28] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. CoRR, abs/1 8877 0011188777444 .0744,,78880 2018.
[29] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximizing Mutual Information Across Views. In NeurlPS, 2019.
[30] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In JCML, 2020.
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR, 2022.
[32] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEÎT: BERT Pre-Training of Image Transformers. ICLR, 2022.
[33] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACI, 2019.
[34] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Multiview Coding. In ECCV, 2020.
[35] Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucie. Self-Supervised Learning of Video-Induced Visual Invariances. In CVPR, 2020.
[36] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What Makes for Good Views for Contrastive Learning? In NewrIPS, 2020.
[37] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What Should Not Be Contrastive in Contrastive Learning. In ICLR, 2021.
[38] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.
[39] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR, 1021.
12

---

[40] Jean-Bastien Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent- A New Approach Self- Self-Super Learning. In NeurIPS, 2020.
[41] Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling. In CVPR, 2022.
[42] Baruch Epstein and Ron Meir. Generalization Bounds For Unsupervised and Semi-Supervised Learning With Autoencoders. CoRR, 0tbs/1902.01449) 2019.
[43] Johannes Lehner, Benedikt Alkin, Andreas Fürst, Elisabeth Rumetshofer, Lukas Miklautz, and Sepp Hochreiter. Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget. CoRR, abs/2304.10522 2023.
[44] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive Auto-Encoders: Explicit Invariance during Feature Extraction. In ICML, 2011.
[45] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of MAE pre-pretraining for billion-scale pretraining, CoRR, abs/2303. 13496, 2023.
[46] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. In CVPR, 2023.
[47] Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What Do Self-Supervised Vision Transformers Learn? In ICLR, 2023.
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In ICLR, 2021.
[49] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. Transactions on Machine Learning Research, 2022.
[50] Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive Audio-Visual Masked Autoencoder. In ICLR, 2023.
[51] Haoxuan You, Mandy Guo, Zhecan Wang, Kai-Wei Chang, Jason Baldridge, and Jiahui Yu. CoBIT: A Contrastive Bi-directional Image-Test Generation Model. CoRR, 0tts/2333.134555 2023.
[52] Ya-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haogi Fan, Yanghao Li, Shang Wen Li, Gargi Ghosh, Jitendra Malik, and Christoph Feichtenhofer. MAViL: Masked Audio- Video Leamers. CoRR, abs/2212.08071, 2022.
[53] Gary A Shaw and Hsiaohua K Burke. Spectral imaging for remote sensing, Lincoln laboratory journal, 2003.
[54] Alberto Moreira, Pau Prats-Iraola, Marwan Younis, Gerhard Krieger, Irena Hajnsek, and Konstantinos P Papathanassiou. A tutorial on synthetic aperture radar. IEEE Geoscience and remote sensing magazine, 2013.
[55] Cle Pohl and John Van Genderen. Review article multisensor image fusion in remote sensing:
concepts, methods and applications. International journal of remote sensing, 1998.
13

---

[56] Pedram Ghamisi, Behnood Rasti, Naoto Yokoya, Qumming Wang, Bernhard Hofle, Lorenzo Bruzzone, Francesca Bovolo, Mingmin Chi, Katharina Anders, Richard Gloaguen, Peter M. Atkinson, and Jon Atli Benediktsson. Multisource and Multitemporal Data Fusion in Remote Sensing: Comprehensive Review of the State of the Art. IEEE Geoscience and Remote Sensing Magazine, 2019.
[57] Michael Schmitt and Xiao Xiang Zhu. Data Fusion and Remote Sensing: An ever-growing relationship. IEEE Geoscience and Remote Sensing Magazine, 2016.
[58] Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Chen Wu, Xiujun Shu, and Bo Ren. VLMAE: Vision-Language Masked Autoencoder. CoRR, abs/2208.09374, 2022.
[59] Ziyang Luo, Yadong Xi, Rongsheng Zhang, Gongzheng Li, Zeng Zhao, and Jing Ma. Condi- tioned Masked Language and Image Modeling for Image- Text Dense Retrieval. In EMNLP, 2022.
[60] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In NeurIPS, 2021.
[61] Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision- Vision-Language Pre-training with Triple Contrastive Learning. In CVPR, 2022.
[62] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics Aligned Pre-training for Vision-Language Tasks. In ECCV, 2020.
[63] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Iman Pre-training for Unified Vision-Language Understanding and Generation. In ICML, 2022.
[64] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks. CoRR, abs/2303. 16839, 2023.
[65] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. In ICML, 2022.
[66] Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In ICLR, 2022.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurlPS, 2017.
[68] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling Language-Imare Image Pre-training via Masking. In CVPR, 2023.
[69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
[70] Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu. SSL4EO-SI S12: A Large-Scale Multi-Modal, Multi Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation. CoRR, abs/2211 1. 1.07044, 2022.
[71] Linus Scheibenreif, Joëlle Hanna, Michael Mommert, and Damian Borth. Self- Self-Supervised Vision Transformers for Land-Cover Segmentation and Classification. In CVPR Workshops, 2022.
[72] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, Emerging Properties in Self-Supervised Vision Transformers. ICCV, 1021.
14

---

[73] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning from Images with Joint-Embedding Predictive Architecture. In CVPR, 2023.
[74] Anthony Fuller, Koreen Millard, and James R. Green. Transfer Learning with Pretrained Remote Sensing Transformers. CoRR, 0ts//2299.14666 2022.
[75] Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham, Chenying Liu, Zhitong Xiong, and Xiao Xiang Zhu. DeCUR: decoupling common & unique representations for multimodal self-supervision. CoRR, abs/2309. .055000,,, 2023.
[76] Gencer Sumbul, Amne de Wall, Tristan Kreuziger, Filipe Marcelino, Hugo Costa, Pedro Benevides, Mário Caetano, Begüm Demir, and Volker Markl. BigEarthNerrMM: A Large- Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets]. IEEE Geoscience and Remote Sensing Magazine, 2021.
[77] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[78] Amanda A Boatswain Jacques, Abdoulaye Baniré Diallo, and Etienne Lord. Towards the Creation of a Canadian Land-Use Dataset for Agricultural Land Classification. In 42nd Canadian Symposium on Remote Sensing: Understanding Our World: Remote Sensing for Sustainable Future, 2021.
[79] Matthew Gwilliam and Abhinav Shrivastava. Beyond Supervised vs. Unsupervised: Represen- tative Benchmarking and Analysis of Image Representation Learning. In CVPR, 2022.
[80] Jae-Hun Lee, Doyoung Yoon, ByeongMoon Ji, Kyungyul Kim, and Sangheum Hwang. Re- thinking Evaluation Protocols of Visual Representations Learned via Self-supervised Learning. CoRR, flos/3044034566. 2023.
[81] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT Pre-training with Online Tokenizer. In ICLR, 2022.
[82] Colorado Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, al. Self-Supervised Pretrain- ing Improves Self-Supervised Pretraining. In WACV, 2022.
[83] Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting Contrastive Self-Supervised Representation Learning Pipelines. In ICCV, 2021.
[84] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the Limits of Large Scale Pre-training. In ICLR, 2022.
[85] Leland McInnes, John Healy, Nathaniel Saul, and Lukas GroBberger. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 2018.
[86] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding Neurons in a Haystack: Case Studies with Sparse Probing. CoRR, abs/2305.01610, 2023.
[87] Naoto Yokoya, Pedram Ghamisi, Ronny Hänsch, and Michael Schmitt. 2020 IEEE GRSS Data Fusion Contest: Global Land Cover Mapping With Weak Supervision. IEEE Geoscience and Remote Sensing Magazine, 2020.
[88] Christopher F Brown, Steven P Brumby, Brookic Guzder- Williams, Tanya Birch Saman- tha Brooks Hyde, Joseph Mazzariello, Wanda Czerwinski, Valerie Pasquarella, Robert Haertel, Simon llushchenko, et al. Dynamic World, Near real-time global 10 m land use land cover mapping. Scientific Data, 2022.
[89] Katerina Kikaki, Ioannis Kakogeorgiou, Paraskevi Mikeli, Dionysios E Raitsos, and Konstanti- nos Karantzalos. MARIDA: benchmark for Marine Debris detection from Sentinel-2 remote sensing data. PloS one, 2022.
15

---

[90] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang. and Chunhua Shen. Conditional Positional Encodings for Vision Transformers. In ICLR, 2023.
[91] Peng Gao, Teli Ma, Hongsheng Li, Ziyi Lin, Jifeng Dai, and Yu Qiao. MCMAE: Masked Convolution Meets Masked Autoencoders. In NewrIPS, 2022.
[92] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varaancelivarrnne-CCraainnn Regularization for Sulf-Supe Learning. In ICLR, 2022.
[93] Adrien Bardes, Jean Ponce, and Yann LeCun. VICRegL: Self-Supervised Learning of Local Visual Features. In NeurlPS, 2022.
[94] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense Contrastive Learning for Self-Supervised Visual Pre-Training, In CVPR, 2021.
[95] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard Negative Mixing for Contrastive Learning. In NeurlPS, 2020.
[96] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolies, and Herve Jegou. Training data-efficient image transformers distillation through attention. In ICML, 2021.
[97] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language. CoRR, abs/2212.07525,, 2022.
[98] Gengchen Mai, Ni Lao, Yutong He, Jiaming Song, and Stefano Ermon. CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Vinn Visual Representations. In ICCV, 2023.
[99] Chao Tao, Ji Qi, Weipeng Lu, Hao Wang, and Haifeng Li. Remote Sensing Image Scene Clas- sification With Self- Superviuee Paradigm Under Limited Labeled Samples. IEEE Geoscience and Remote Sensing Letters, 2022.
[100] Heechul Jung, Yoonju Oh, Seongho Jeong, Chaehyeon Lee, and Taegyun Jeon. Contrastive Self-Supervised Learning With Smoothed Representation for Remote Sensing. IEEE Geo- science and Remote Sensing Letters, 2022.
[101] Jian Kang, Ruben Brrmandde--eelrraan Puhong Duan, Sicong Liu, and Antonio J. Plaza. Deep Unsupervised Embedding for Remotely Sensed Images Based on Spatially Augmented Mo- mentum Contrast. IEEE Transactions on Geoscience and Remote Sensing, 2021.
[102] Yuxing Chen and Lorenzo Bruzzone. Self-Supervised SAR-Optical Data Fusion of Sentinel- 1/-2 Images. IEEE Transactions on Geoscience and Remote Sensing, 2022.
[103] Lucas Kruitwagen. Towards DeepSentinel: An extensible corpus of labelled Sentinel-1 and -2 imagery and general-purpose sensor-fusion semantic embedding model. CoRR, abs/2102.06260,, 2021.
[104] Xin Huang, Mengjie Dong, Jiayi Li. and Xian Guo. A 3-D-Swin Transformer-Bate Hierarchi- cal Contrastive Learning Method for Hyperspectral Image Classification. IEEE Transactions Geoscience and Remote Sensing, 2022.
[105] Johan Bjorck, Brendan H. Rappazzo, Qinru Shi, Carrie Brown-Lima, Jennifer Dean, Angela Fuller, and Carla Gomes. Accelerating Ecological Sciences from Above: Spatial Contrastive Learning for Remote Sensing. In AAAI, 2021.
[106] Haifeng Li, Yi Li, Guo Zhang, Ruoyun Liu, Haozhe Huang, Qing Zhu, and Chao Tao. Global and Local Contrastive Self Supervieee Learning for Semantic Segmentation of HR Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2022.
[107] Xinye Wanyan, Sachith Seneviratne, Shuchang Shen, and Michael Kirley. DINO-MC: Self- supervised Contrastive Learning for Remote Sensing Imagery with Multi-sized Local Crops. CoRR, 0e////333066770 2023.
16

---

[108] Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tele Unsupervised Representation Learning for Spatially Distributed Data. In AAAI, 2019.
[109] Haozhe Huang, Zhongfeng Mou, Yunying Li, Qiujun Li, Jie Chen, and Haifeng Li. Spatial- Temporal Invariant Contrastive Learning for Remote Sensing Scene Classification. IEEE Geoscience and Remote Sensing Letters, 2022.
[110] Vladan Stojnic and Vladimir Risojevic. Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding. In CVPR Workshops, 2021.
[111] Gencer Sumbul, Markus Müller, and Begüm Demir. A Novel Self-Supervised Cross-Modal Image Retrieval Method in Remote Sensing. In ICIP, 2022.
[112] Isaac Corley and Peyman Najafirad. Supervising Remote Sensing Change Detection Models With 3d Surface Semantics. In ICIP, 2022.
[113] Keumgang Cha, Junghoo Sco, and Yeji Choi. Contrastive Multiview Coding With Electro- Optics for SAR Semantic Segmentation. IEEE Geoscience and Remote Sensing Letters, 2022.
[114] L. Scheibenreif, M. Mommert, and D. Borth. Contrastive Self-supervised Data Fusion for Satel- lite Imagery. ISPRS Annals of the Photogrammetry. Remote Sensing and Spatial Information Sciences, 2022.
[115] Umangi Jain, Alex Wilson, and Varun Gulshan. Multimodal contrastive learning for remote sensing tasks. CoRR,
2022.
[116] Chenfang Liu, Hao Sun, Yanjie Xu, and Gangyao Kuang. Multi-Source Remote Sensing Pretraining Based on Contrastive Supervuue Learning. Remote Sensing. 2022.
[117] S. Vincenzi, A. Porrello, P. Buzzega, M. Cipriano, P. Fronte, R. Cuccu, C. Ippoliti, A. Conte, and S. Calderara. The color out of space: learning self-supervised representations for Earth Observation imagery. In ICPR, 2021.
[118] Anthony Fuller, Koreen Millard, and James R. Green. SatViT: Pretraining Transformers for Earth Observation. IEEE Geoscience and Remote Sensing Letters, 2022.
[119] Xian Sun, Peijin Wang, Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong, Zhujun Yang, Hao Chang, Qinglin He, Guang Yang, Ruiping Wang, Jiwen Lu, and Kun Fu. RingMo: A Remote Sensing Foundation Model with Masked Image Modeling. IEEE Transactions Geoscience and Remote Sensing, 2022.
[120] Yuan Gao, Xiaojuan Sun, and Chao Liu. A General Self-Supervised Framework for Remote Sensing Image Classification. Remote Sensing, 2022.
[121] Hongmiao Wang, Cheng Xing. Junjun Yin, and Jian Yang. Land Cover Classification for Polarimetric SAR Images Based on Vision Transformer. Remote Sensing, 2022.
[122] Kai Wang, Steven Franklin, Xulin Guo, and Mare Cattet. Remote Sensing of Ecology, Biodiversity and Conservation: A Review from the Perspective of Remote Sensing Specialists. Sensors, 2010.
[123] Sahel Mahdavi, Bahram Salehi, Jean Granger, Meisam Amani, Brian Brisco, and Weimin Huang. Remote sensing for wetland classification: A comprehensive review. GIScience & Remote Sensing, 2018.
[124] Freek D Van der Meer, Harald MA Van der Werff,, Frank JA Van Ruitenbeek, Chris A Hecker, Wim H Bakker, Marleen F Noomen, Mark Van Der Meijde. E John M Carranza, J Boudewijn De Smeth, and Tsehaie Woldai. Multi-and hyperspectral geologic remote sensing: A review. International Journal of Applied Earth Observation and Geoinformation, 2012.
[125] John Wilson Rouse, Rüdiger H Haas, John A Schell, Donald W Deering, et al. Monitoring vegetation systems in the Great Plains with ERTS. NASA Spec. Publ, 1974.
17

---

[126] E RRaymond Hunt. Jr. Relationship between woody biomass and PAR conversion efficiency for estimating net primary production from NDVI. International Journal of Remote Sensing. 1994.
[127] Bo-Cai Gao. NDWI- normalized difference water index for remote sensing of vegetation liquid water from space. Remote sensing of environment, 1996.
[128] Zhao-Liang Li, Pei Leng, Chenghu Zhou, Kun-Shan Chen, Fang-Cheng Zhou, and Guo-Fei Shang. Soil moisture retrieval from remote sensing measurements: Current knowledge and directions for the future. Earth-Science Reviews, 2021.
[129] Compton J Tucker. Remote sensing of leaf water content in the near infrared. Remote sensing of Environment, 1980.
[130] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176B-Parameter Open-Aceess Multilingual Language Model. CoRR, abs/221 1.05100, 2022.
[131] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. Training Compute-Optimal Large Language Models. In NeurlPS, 2022.
[132] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, al. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302.13371, 2023.
[133] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and Improving Relative Position Encoding for Vision Transformer. In ICCV, 2021.
[134] David Arthur and Sergei Vassilvitskii. K-Means++: The Advantages Careful Seeding. In Proceedings of the Eighteenth Annual ACM-SIAN Symposium Discrete Algorithms, 2007.
[135] H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
18

---

A
Appendix
Code and pretrained models: httpge///ggiibbooo////oootllltlp
A.1 Other Pretraining Experiments
In this section, we experiment with additional CROMA settings. We use the same experimental conditions as of our paper; i.e., linear probe representations on BigFarthNet [76 (reporting mAP on the combined validation and test sets) and patch encodings on DW-Expert-120 88 (reporting
mloU on the validation set). We use the linear probing hyper-parameters listed in A.2.2 of this Appendix.
Table 8: Linear probing results on radar-only ("R"), optical-only ("O"), and joint radar-optical ("RO") inputs. Across all experiments, we use 2D-ALiBi with X-ALiBi, 75% shared masking, ViT-B backbones, and 100 pretraining epochs.
Modal
Obj Weights
Mixing
Classification (mAP)
Segmentation (mlol)
Image Obj.
Patch Obj.
Depth, Dim
Acoo AMAE
(1024, O. ri)
Cost
R
RO
R
0
RD
InfoNCE
MSE
1,512
1x
77.4
83.9
84.3
40.5
56.0
56.7
InfaNCE
MSB
I, 768
1,1
X
1x
77.4
83.9
84.3
40.7
56.1
56.6
InfoNCE
MSE
3,512
1.1
K
1,2x
77.5
83.9
84.4
40.5
56.3
57.1
InfoNCE
1.3x
InfoNCE
MSR
6,512
1.4x
77.5
83.9
84.4
40.3
56.0
56.7
InfoNCE
MSB
7,768
1,1
X
1.6
77.6
83.8
84.5
40.6
56.2
56.7
InfoNCE
1,512
1.1
1X
77.4
84.0
84.5
40.8
56.1
56.4
InfoNCE
InfoNCE
3,512
1.2X
77.6
84.1
84.5
40.8
56.2
56.7
InfoNCE
3,768
1.1
X
1.3
77.0
83.9
84.5
40.6
56.1
56.5
InfNNC
6,512
1.1
14X
77.3
84.1
84.5
40.8
56.1
56.5
InfoNCE
77.5
40.6
56.8
InfoNCE
InfONCE
3,512
2.2
72.8
80,9
82.4
39.0
$5.1
55.2
InfoNCE
1,512
1.2
X
LX
77.5
84.3
84.2
40.7
55.9
56.2
InfoNCE
1,512
1X
77.5
84.3
84.1
40.6
InfoNCE
InfoNCE
1,512
1X
77.6
83.9
84.5
40,7
55.8
56.8
InfoNCE
1,512
128
1X
73.6
81.6
83.0
38.0
53.2
55.0
InfaNCE
1,512
1X
73.0
81.0
InfoNCE
1,512
72.5
80.2
82.4
37.6
52.6
54.4
VICReg
MSB
1,768
1.1
X
1.1
70.7
78.7
83.3
40.0
55.5
55.1
Self-supervised Objectives. Inspired by the local objective of VICRegL 933, we experiment with mean squared error (MSE) objective between cross-modal patch encodings, i.e., MMo &o). This attracts patch encodings if they match locations, i.e., if they represent the same 80 30m m square on the ground. We find this does not improve representations. Next, we experiment with the VICReg (92 objective (calculating VICReg statistics based on a batch size of 800) between cross-mode image representations, i.e., Rg and Ro; we find it underperforms InfoNCE (28) Finally, we experiment with the InfoNCE objective between cross-modal patch encodings; positive pairs encodings that match locations across modalities, and negative pairs are all other encodings from the matched sample and encodings from all other samples in the batch. This does not improve representations and slows pretraining by 2.2x (Table 8).
Objective Weights. We find that weighting the contrastive loss term MAE 31 loss term does uniformly improve representations; hence, we select equal weights.
Hard Negatives. We find that hard-negative mixing 955 (N=1024, s=0, s'=n, 8=0.5, with of 1 128, 256, or 512) degrades performance when used in our framework.
Decoder Sizes. At least in these experiments, CROMA is not sensitive to the decoder size; tiny decoder with a 1-layer, 512-d transformer performs similarly to much larger 6-layer, 768-d transformer.
19

---

Position Encoding with Shared Masking, We find
Table 9: Linear probing results with shared 75% masking, ViT-B, 100 epochs.
that using 2D-sinusoidal embeddings or PEG [90]
with shared masking performs poorly. These two
Classifiation
Segmentation mloti
methods of position encoding store positional infor-
Method
mAP
mation in the internal representations, which can help solve the contrastive objective if both modal- ities share masks; 2D-ALiBi instead stores positional information in the attention matrix, which may pre-
R
0
RO
R
0
RO
[90]
67.9
75.9
79.0
32.6
49.8
51.0
2D-Sin.
69.4
75.6
79.8
29.0
44.1
50.7
vent this from occurring. In our paper (Table 6 show that 2D-sinusoiaal or PEG can perform well in our framework if modalities are masked independently, although 2D-ALiBi still outperforms these approaches.
Lower Masked Tuning. FLIP 68 performs con- trastive learning using the representations of masked- out samples; after this masked pretraining, it lever- ages unmasked tuning to increase accuracy by 1.3% on zero-shot ImageNet-IK K. Unmasked tuning contin- ues FLIP pretraining by performing contrastive learn- ing using the representations of unmasked samples to reduce the distribution gap between pretraining and inference [68]. We cannot perform fully unmasked
Table 10: Lower masked tuning for 5 epochs after pretraining CROMA-L.
Classificatio mAP
Segmentation mloU
Mask Ratio
RO
RO
10%
80.8
84.7
84.7
43.8
56.8
56.6
80.8
84.8
43.9
56.8
56.6
50%
80.8
84.8
85.0
43.9
56.8
56.6
tuning because must mask patches for reconstruction objective. However, can lower our mask ratio and perform lower masked tuning. Following FLIP, initializing parameters with our train for 5 additional epochs using base learning rate of 8e-8, warmup over the first epoch, and cooldown for 4 epochs using a cosine decay schedule. We explore mask ratios {10%, 25%, 50% } and find that lower masked tuning does not improve linear probing accuracy for CROMA (Table 10p.
pretrained CROMA-L model,
A.2 Two Reasons for 2D-ALiBi's Performance
Our primary reason for introducing 2D-ALiii is to enable the test-time extrapolation demonstrated in $4.2 But even when no extrapolation occurs -training and testing on the same image size-213
ALiBi outperforms both the most commonly used position encoding method (2D-sinusoidal) and SoTA relative position encoding method (PEG [90).. We believe 2D-ALiBi outperforms these two methods for two reasons.
Table 11:
Cosine similarity between the representations of images and the representations of trans- formed versions of the same images. Higher similarity means the representations are less influenced by the image transformation.
Transformation (cosine similarity)
Method
V.Flip
rotate(90")
rolate(180")
rotale(270")
2D-ALiBI X-ALiBi
0.992
0.992
0.992
0.992
0.992
2D-ALiBi
0.992
0.992
0.992
0.992
0.992
PEG 90
0.992
0.992
0.988
0.992
0.988
2D-sinssoddl
0.641
0.629
0.523
0.559
0.523
2D-ALiBi learns image representations invariant to rotating and flipping. Learning represen- tations that are invariant to certain transformations is often desirable. In CROMA, the contrastive objective between optical and radar data encourages sensor-invariant representations. Invariances to flipping and rotating are desirable properties of the representations of satellite imagery. How- ever, CROMA'S pretraining objectives do not explicitly encourage these invariances. To investigate if CROMA's optical representations (i.e., Ro) are invariant to flipping and rotating, we produce representations of 5, 5,000 optical images from DW-Expert- for four position encoding strategies 2D-ALiBi, 2D-ALiBi with X-ALiBi, PEG, and 2D-sinusoidal. Then, for each model, we produce representations of the same samples but transformed. We measure the cosine similarity between the original and transformed images' representations. When used in our CROMA franework, 2D-ALiBi learns representations invariant to flipping and rotating (average cosine similarity of 0.992, Table III Notably, 2D-sinusoidal learns representations that are not invariant to flipping and rotating
20

---

(average cosine similarity of 0.575, Table II]: we believe this contributes to the poor performance of 2D-sinusoidal embeddings on image classifcction. PEG performs similarly to 2D-ALiBi on image classification and also learns representations invariant to flipping and rotating.
Table 12: (Middle column) Cosine similarity between patch encodings at different locations within an image, averaged across 5,000 images. (Right column) Cross entropy loss of an MLP probe trained to predict patch locations given patch encodings.
Method
Patch Encoding Similarity (cosine similarity)
Encoding Position Probing (cross entropy loss)
2D-ALiBi + X-ALiBi
0.546
2D-ALiBi PEG 3
0.582
4,41
0,701
4.13
2D-sinusoidal
0.493
0.00
2D-ALiBi learns patch representations that retain local information. 47 show that models trained with contrastive objectives can lose significant local information at deeper ViT layers that harm performance on dense prediction tasks. They show that the cosine similarity between the representations of patches different locations within an image becomes high, indicating a patch- wise representational collapse. At every VIT layer, 2D-ALiBi injects a local bias in each attention head, for each patch location. To investigate if 2D-ALiBi can successfully prevent this collapse from occurring, we measure the cosine similarity between different patch encodings of the same sample and take the average across 5,000 optical images. We find that 2D-ALiBi learns to represent patches with greater spatial diversity than PEG, and X-ALiBi further improves diversity (Table Interestingly, 2D-sinusoidal learns the most diverse patch representations. We also train MLP probes on patch encodings to classify the patch location; this measures the amount of positional information represented patch encodings. We find that the patch encodings of models that use 2D-sinusoidal embeddings fully specify the location of the patch within the image (Table [2): this is undesirable property of patch encodings, which should represent the content of the patch, rather than its location.
A.3 Pretraining Details
A.3.1 Data
We use the SSL4EO dataset [70. which consists of Sentinel-1 & imagery acquired at 250K locations around the world; each location (a 2.64 km 2.64 km square) is imaged four times, spread out year. We use these million samples of 264 x 264 pixels for pretraining. Please see the SSI4EO paper [70 for more details.
A.3.2 Implementation
We use an NVIDIA DGX server (8 88xA100-800G 80GB), the maximum batch size that can fit into 640 GB of VRAM (7,200 for our default ViT-B), bfloatl6 precision, a base learning rate of 4e-6, warmup for 5% of the total epochs, and cooldown via a cosine decay schedule. We use the same normalization procedure as SatMAE (26). For data augmentation, we randomly crop 60-180 pixel squares from the original 264 x 264 pixels and resize the crops to 120 x 120 pixels (our default image size). We also perform vertical and horizontal flipping, 90-degree rotations, and mixup=0.3. Crucially, apply these transformations identically to both modalities; if we applied them to each modality independently, our spatial alignment would break. We use the Adam W optimizer with 3,=0.9 and 32=0.999, and a weight decay of 0.01.
A.4 Evaluation Details
The evaluation of foundation models for Earth Observation is less mature than in other fields. We do our best to re-use the experimental conditions of the SoTA, i.e., SatMAE (26), and improve upon them where possible. One such condition is to report results from held-out validation set; precisely, the best validation performance measured after each finetuning epoch is reported. No test sets are used. To enable fair comparisons with prior work, we copy this approach. In trying to improve the evaluation of foundation models for Earth Observation, we detail our approach this Appendix, share code and preprocessed datasets, re-evaluate all near-So TA models under identical
21

---

conditions, and evaluate models in more ways than prior work (i.e., linear and nonlinear probing, KNN classification, and K K-means clustering).
We initialize parameters from publicly shared pretrained weights, evaluating all models ourselves under identical conditions. Although this process is laborious, we believe it significantly improves the value of our paper, several prior studies have often evaluated their models in different ways, using different data splits that cannot be directly compared. When downloading pretrained weights, we use the latest weights that are publicly available. For instance, SatMAE 2 released improved versions of their multispectral ViT-B and VIT-L models, pretrained for 200 epochs, after their manuscript was accepted for publication (edited on arxiv on January 15th, 2023). We exclusively evaluate these improved models throughout our paper, ensuring we compare CROMA to the best models available. For multispectral benchmarks with 13 channels (with cirrus included), we simply drop the cirrus band for models pretrained without it.
A.4.1 Data
BigEarthNet. [76 We use the same splits for training (10% of the complete training set) and evaluating (the entire validation set) SatMAB (26] and SeCo [25]. However, we use the combined validation and test sets (236,130 samples) in our ablation studies to increase the reliability of our findings with minimal added cost. Images are 120 120 pixels.
[26] Inspired by how the BigEarthNet benchmark is used (i.c., training on 10% of the complete training set of 354,200 samples), we create 10% split of the complete iMoW-Sentie Sentinel training set of 71 712,874 samples. We share the IDs of the 10% of (MoW-Sentinel training samples that we randomly selected. We believe this smaller training set should be used in future work to reduce searches- a single finetuning run of SatMAE on the complete training
the costs of set requires 192 hours on a V100 GPU (26). Following SatMAE, we use the full validation set for evaluation. Images vary in size; the mean height is 45 pixels, and the mean width is 60 pixels.
In our paper, we benchmark this new split. However, we report results obtained by our CROMA models on the complete training set in Table 13 Due to the costs of finetuning on the complete training set (712,874 samples), we decide allocate resources else- where and not perform any hyper-parameter tuning. Instead, we select apper-parameters we believe be reasonable and finetune CROMA-B and CROMA-L once. For finetuning, we use base learning rate of le-5 and all other hyper-parameters from $A.4.2
Table 13:
fMoW-Sentinel results (top 1 ac- curacy) using the complete training set. * denotes results reported in SatMAF (updated on arxiv on January 15th, 2023).
Method
Backhone
Finetuning
Lineur Probing
SaIMAE
VIT-B
62.65*
37.40
CROMA
VIFB
61.00
40.94
SaMMA
VIT-L
63.84*
39.19
CROMA
WIL
63.59
41,96
EuroSAT. 77 We use the same training and validation sets as SatMAE. Images are 64 64 pixels.
Canadian Cropland. (78) We are the first benchmark this dataset of Canadian agricultural croplands, consisting of 10 classes (barley, canola, corn, mixedwood, oats, orchard, pasture, potato, soybean, and spring wheat). We select this dataset because it is large dataset that evaluates different capabilities from the other benchmarks that typically consider croplands as single class. Following EUToSAT (77), the authors selected an image size of 64 64 pixels [78]; therefore, models evaluated on EAroSAT can be evaluated on Canadian Cropland with minimal modifications. We use the training set and combine their validation and test sets to form a single held-out set for evaluation. We share these complete training and validation sets. The performance (Table [ and representation visualizations (Fig. and 7 in this Appendix) indicate that the 10 classes present in this dataset are challenging separate.
DFC2020. 88 This dataset is used for evaluation in diverse ways both the choice of data split and image size. The original dataset comprises 6,114 samples of 256 x 256 pixels. These samples are typically split into two: so-called "validation set" of 986 samples and a so-called "test set" of 5,128 samples. Some studies use the "validation set" for training and the "test set" for validation; others use the "test set" for training and the "validation set" for validation. Some studies use the full 256 x 256 pixels as inputs to their models, while others use smaller inputs. We select the split of 5,128 samples for training, which we divide into 46, 152 images of 96 x 96 pixels-leaving us with the split of 986 samples for validation, which divide into 8,874 images of 96 96 pixels. We select this final
22

---

resolution because it is the default image size of SatMAE, enabling a fair comparison to the SoTA. We share these complete training and validation sets.
DW-Expert. [88 The data collected by Dynamic World 88 is a new high-quality dataset annotated
by experts with the help of auxiliary information. Thus, it should be used in the future when benchmarking models. Our work uses the expertly annotated data from Dynamic World, which we split into 20,422 train samples and 51,022 validation samples. All images are 96 x 96 pixels to enable a fair comparison with SatMAE. We share these complete training and validation sets. We also create a version of this dataset that consists of 120 120 pixel images (i.e., DW-Expert- 120) that we only use for ablations because it is the default image size of CROMA.
MARIDA. 89 We use the training set and combine the validation and test sets to form single held- out set for evaluation. Following our approach for DFC2020 and DW-Expert, we divide the original images into images of 96 x 96 pixels. Because it is a sparsely labeled dataset (i.e., only fraction of pixels per image are labeled), we include images with at least one labeled pixel. We select this dataset because it evaluates different capabilities from the other semantic segmentation benchmarks. It consists of the following classes: marine debris, dense Sargassum, sparse Sargassum, natural organic material, ship, clouds, marine water, sediment-laden water, foam, turbid water, shallow water, waves, cloud shadows, wakes, and mixed water. We share these complete training and validation sets.
A.4.2 Implementation
Finetuning, We select reasonable syper-parameters that we use for all models and datasets unless otherwise stated and sweep across learning rates. This learning rate sweep is essential to creating fair evaluation conditions across models since each model is given the same search budget (in terms of finetuning runs, not compute hours), and different models have different optimal learning rates. Models pretrained with reconstruction approaches tend to require higher base learning rates during finetuning than models pretrained with contrastive learning. For instance, MAE [31 lists a base learning rate of le-3, FLIP (68) lists base learning rate of 5e-5, CoCa [4] lists base learning rates from le-5 to 5e-4, depending on the downstream dataset.
No single learning rate would enable fair comparison across all models and datasets. Therefore, we sweep learning rates across an extensive range (3e-5, 5c-5, 8c-5, 1e-4, 3e-4, 5e-4, 8c-4, 1e-3) ] and report the best single evaluation result obtained for each dataset; this sweep is performed for CROMA models and all other models. We convert these base learning rates to actual learning rates via the widely used linear scaling rule: Ir = base_lr _e x butch_size/ /25.. We use the largest batch size that can fit on an A100-40GB GPU (using 6 precision), the AdamW optimizer with 31 =0.9, $2=0.999, and a weight decay of 0.01. We warmup for 5 epochs and cooldown for 30 epochs using a cosine decay schedule (other than EuroSAT, which we cooldown for 150 epochs); this follows SatMAE (26). For classification tasks, we use mixup=0.8, cutmix=1.0, switch probability=0.5, label smoothing=0.1, vertical and horizontal flipping, and 90-degree rotations. We enlarge images to the default image size of the model we are finetuning (i.e., the image size which the model was pretrained), with one exception. The default image size of SatMAE is 96 x 96; however, BigEarthNet images are 120 x 120 (76), requiring that we either crop BigEarthNet samples (losing information) or adapt SatMAE to larger images. We achieve better performance by adapting SatMAE to 120 x 120 images via the widely used position embedding interpolation algorithm than cropping BigEarthNet samples down to 96 x 96. This allowed us to achieve an mAP of 86.18 for SaIMAE, a significant improvement over the 82.62 reported in the SatMAE paper. All other datasets use images of 96 x 96, or shaller-thus, there is no reason to use this technique for other datasets.
I Linear and Nonlinear Probing. We encode each image without data augmentation then train linear and nonlinear probes on the frozen representations. Since each model only encodes each image once, can sweep through large range of learning rates (11, 3, 4,5, 6,7,8, 9,,,,,,4,, -3, -2)) very quickly. Unlike finetuning, we do not evaluate probes after every epoch; instead, we evaluate trained probes after all epochs are complete. We use batch size of 1024, bfloat16 precision, the AdamW W optimizer with 31=0.9, 3,=0.999, and a weight decay of 0.01. We warmup for 5 epochs and cooldown for 100 epochs using cosine decay schedule.
Mon-parametric ANN and K- K-means. For kNN, we use the implementation from This consists of encoding all training and validation samples and then using the representations of validation samples as queries and training samples as keys to fetch training labels. These fetched training labels are used to classify validation samples. We use k=20, other values for & (i.c., 10, 50) ranked
23

---

Table 14: CROMA Vs SatMAE training and inference throughput on an A100-40GB GPU.
Backhone
Image Size
Train Imgs's
Inference Imgsls
SaMAA
VIT-B
96x96
249.3
692.5
CROMA
VIT.B
96x96
1,079.3
2,957.7
CROMA
x20x120
555.0
1,532.1
SaIMAE
VIT-L
96x96
84.2
263.2
CROMA
VIF-
96x96
389.1
1,168.2
CROMA
VII-L
120x120
209.6
640.3
models the same order as k=20. For K -meas, we use the implementation from 77]. This consists of encoding all training and validation samples, then clustering training samples with K K-means (K-means++ [[34] initialization run 10 times). Then, we assign validation samples to clusters and assign clusters to classes via the Hungarian matching algorithm [135].
Sparse Probing. For each model and dataset, we rank the dimensions of representations R. using the mean difference between classes- [86] shows that the mean difference performs on-par with more complex ranking methods and is simple implement. Specificallyy, this is our procedure to sparsely probe the representations of CROMA for BigEarthNet's "beaches, dunes, sands" class: (i) compute the average representation, R Beaches E R676, of all samples in the BigEarthNet training set that contain the class "beaches, dunes, sands"; (ii) compute the average representation, COOMA R 768, of all samples in BigEarthNet that do not contain the class "beaches, dunes, sands"; (ii) compute the difference between these averaged representations, Raif E 76 (iv) rank all 768 dimensions in Rai by the absolute value, i.e., the dimension with the greatest absolute difference between classes is ranked first; (v) train separate linear probe to perform binary classification on the top & dimensions, we sweep many values of & between and 768; (vi) using these trained probes, evaluate them on the BigEarthNet validation set. Again, this procedure is performed for all three ViT-B models (CROMA, SatMAE, and I-JEPA), all classification datasets (BigEarthNet, fMoW-Senting and AT), and classes-thed plots are displayed at the end of this Appendix $A.6 We also sparsely probe radar-only, Rg, and radar-optical, Rgo- representations for BigEarthNert, since we have access to Sentinel-1 samples.
Canadian Cropland,
SatMAE Specifics. SatMAE [26) divides spectral bands into three groups and outputs patch encod- ings for every group; thus, SatMAE outputs three patch encodings per patch location. To be fair as possible to SatMAE, we explore four ways of merging these co-located patch encodings to perform segmentation: unnormalized spatial concatenation, normalized spatial concatenation, unnormalized spatial pooling, and normalized spatial pooling. We find unnormalized spatial concatenation (i.e., concatenating the patch encodings co-located patches before the LayerNorm) performed best. Thus, we use the unnormalized spatially concatenated patch encodings for all segmentation datasets. Conversely, CROMA does not divide spectral bands into groups-rrsulting in 3x shorter sequence lengths. The computation required to process sequence of tokens with transformer increases with increasing sequence lengths. This makes CROMA much more computationally efficient than SatMAE for given ViT backbone and image size (Table 14). We also pretrain SatMAE-B model for 300 epochs on the SSL4EO dataset. However, this model performs poorly, so we do not report these results; this experiment indicates that SatMAE's pypre-parmmeters may not transfer well to different pretraining datasets.
A.5 Societal Impact
Since we pretrain our models on the SSL4EO dataset 770,, our models may be biased towards the distribution from which SSL4EO data were sampled. Although SSL4EO samples are geographically diverse (please see Fig. 2 from the SSL4EO paper (70)), locations human settlements. As a result, large geographic areas that are sparsely populated- Amazon rainforest, the Sahara desert, and the Australian outback are underrepresented. This could negatively impact the quality of representations in these locations and any decisions made on their basis.
sampled from areas surrounding instance, the
Another distribution shift-thi, time, between finetuning and inference-is is our primary concern. For example, finetuning a model on the imagery of one geography and then making predictions on the imagery of another geography creates a distribution shift. As a result, biases from the finetuning geography may be realized in the predictions made by the finetuned model. This is particularly
24

---

problematic when these predictions are used in decision decision-makinn making, for instance, allocating poverty assistance. However, it is well-demonstrated that pretrained models are more robust distribution shifts than models trained from scratch, Additionally, as we develop better foundation models for Earth Observation, we reduce the need for annotated data; this may allow practitioners to be more selective of the data they wish to leverage during finetuning.
We do not expect our pretrained models be particularly valuable for military applications, as militaries likely have access to higher resolutions (spatially, spectrally, and temporally) than Sentinel- 1 & 2 provide. However, our framework may be leveraged pretrain models on higher-resolutin imagery, which could be useful for military applications, although this is risk of all novel learning algorithms.
A.5.1 Compute
approximate the computational resources we use for pretraining and finetuning (frozen represen- tation evaluations are negligible in comparison). For pretraining, estimates are in A A100-80GB GPU hours; for finetuning, estimates are A A100-40GB GPU hours. Please see Table T5
Table 15: Estimated GPU hours used for developing and validating CROMA.
Method
Backhone
Task
GPU Hours
rudare+optical
ResNet50
Classificaiin Finetuning
radar ropticll
Swin-T
Classification Finctuning
25
MAE 31 70
VITS
Classificatinn Finetuning
20
DINO 72 20
VIT
Classification Fineluning
Classification Finetuning
CROMA
VIT-B
Classificuton Fineluning
SaIMAE 20
VIT-
Classification Finctuning
215
CROMA
VIFL
Classification Finetuning
CROMA
VIT-B
Pretraining 300 epochs
CROMA
VIT-
Pretraining 600 epochs
380
CROMA
VIT-B
Pretranning Ablations
1,100
A.6 Visualizations
We visualize representations and patch encodings using UMAP and t-SNE. For both segmentation datasets (DFC2020 (87) and DW-Expert (88), visualize patch encodings of 50,000 randomly sampled patches and use the most dominant class patch as its label. We also plot sparse probing results for binary classifiers trained on the top & representation dimensions.
CROMA (VIT-B)
SarMAE (ViT-B)
DINO (VIT-S)
radare+optical (Swin-T)
Permancot Crop Sea & Lake
Residential Forest
Industrial River
Highway Annual Crop
Pasture Herbaceous Vegetation
Figure 5: I-SNE plots of EuroSAT representations.
25

---

CROMA (NTT-B)
SarMAE (VTT-B)
DINO (VIT-S)
rudar++optical (Swin-T)
Barley Orchard
Canola Pasture
Corn
Mixedwood Soy
Potato
Spring Wheat
Figure 6: UMAP plots of Canadian Cropland representations.
CROMA (VTT-B)
SarMAE (NTT-B)
DINO (VT--S)
radare+optical (Swin-T)
Barley Orchard
Canola Pasture
Com
Mixedward Soybean
Out Spring Wheat
Potato
Figure 7: t-SNE plots of Canadian Cropland 78 representations.
CROMA-B (UMAP)
CROMA-B (-SNE)
(UMAP)
SaMAAEB (-SNE)
Water Shrub & Scrub
Trees Built-up
Grass Barren
Flooded vegetation
Crops
Snow & loc
Figure 8: UMAP and t-SNE plots of DW-EEppert
88
patch encodings.
CROMA-B (UMAP)
CROMA-B (-SNE)
SarMAE-B (UMAP)
SaIMAE-B (-SNE)
Forest Built-up
Shrubland
Grassland Water
Wetlands
Croplands
Barren
Figure 9: UMAP and t-SNE plots DFC2020 8 patch encodings.
26

---

Utban Eabric
commercial units
Arable land
Permanent crops
0.7
0.4
0.8
Seor
0.22
10.3-
0.5
02.77
0.1
0.2
0.0
10a
10a
102
103
10
10a
102
103
10
10a
102
10a
10a
101
102
10a
Complex culivcation patierns
Agricalture with
Pistures
natural vegesation
Agro forestry areas
0.6
F1 Stooo
0.6
Fl Scorr
Fl Score
Fl Score
0.5
0.00
0.4
0.4
111E
1100
1100
11100]
10
10a
10
10
10a
10
10
10
10a
10a
10
10
10a
10
10a
10
Natural grassland
Coniferous forest
sparsely vegetated
0.7
0.8
F3 SSore
Score
0.7
0.7
0.6
100
10
10
10
10
10
10
10a
10a
10'
100
10
10
10'
10"
10a
Mooes, beathland
Traasitional woodlandshhah
Beaches, dunes. sands
sclerophyllons vegetation
Ialand wellands
0.4
Scone
0.6
Score
Scoge
0..3
0..2-
0.0.
0.2
0.2
10
10
10
101
10
10
101
10a
10a
10"
10a
103
Coustal wetlands
Inlamd waters
Marine
CROMA (radar)
Score
21 Scoore
CROMA (optical)
CROMA iaadar-pticol
D..8
SaIMAE
I-JERA
0.7
10
10a
10
10a
10a
10
10
Figure 10: Sparse Probing all classes in BigEarthNet
27

---

Barley
Canola
Corn
Mixedwood
0.25-
0.6
0.6
0.0
0.4
0.4
0.15
10
10a
102
10a
10
10a
102
10a
10
10a
10
10a
10
10a
202
10a
Orchard
Pasture
Potato
0.5
0.8-
0.44
0.25
Sane
Scnte
0.3
0.16
100
102
10a
10a
10
10
10a
10a
10a
102
10a
10a
20
10a
Soybean
Spring
0.4
0.35
F1 Soon
00.4
CROMA
SaMMAE
FI
I-JEPA
0.3
1111
10
101
10a
10a
10
101
10a
10a
Figure 11: Sparse Probing all classes in Canadian Cropland
28

---

Permanent crop
Residential
Industrial
Highway
0.8
0.8
Saxe
115cee
Scoec
0..8
0.66
0.6
0.66
0.4
100
10
102
10a
10
10a
102
10a
10
10a
10
10a
10
10a
202
10a
Pastuee
Lake
Forest
River
0.8
Sane
0.
0.96
Saoe
0.8
10.8-
0.9
0.4
0.85
0.6
0.4
100
102
10a
10
10a
102
10
10a
20
102
crop
vegetation
Soger
F1 SSooree
CROMA
SaMMAE
0.6
I-JEPA
0.4-
1111
10
101
10a
10a
10
101
10a
10a
Figure 12: Sparse Probing all classes in EuroSAT
29

---

Airport
Airport hangar
Airpont terminal
Amusetenn park
0.6
0.2-
0.4-
F1 Store
0.1
0.33
0.44
FI
F1
0.05
0.1
1811
111E
1101
110
1T1II
10
10
10a
10a
10
10
10a
10a
10
10a
10
10a
10
10
10
10a
Aquaculture
Ancharological site
Burn
Border checkpoint
0.6
0.06
50.4
0.35
F1
0.004
100
10a
10
10a
10
10a
10
10a
10
10a
20
10a
10
10a
10
10a
Burial site
Car dealenshpp
Construction
Crup field
0.07
0.15
Scoe
0.004
0.5
0.05
0.03
0.3
100
10
10
10a
10
10a
10
10
10a
10
10a
10a
20
10a
institution
0.3
0.12-
0.07
$0.06
0.1-
0.08
0.04
1111
1000
100
10a
10a
10
10a
10a
10a
10
10
10
10a
10
10a
102
10a
Factory or powerplant
Pire station
Hlooed road
0.08
0.06
0.07
0.10
CROMA
0.1
SaIMAR
0.044
I-JEPA
0.05
0.05
100
10a
102
10a
100
10a
102
10a
100
10a
102
10a
Figure 13: Sparse Probing all classes in fMoW-Senting (part i)
30

---

Fountain
Gas station
Golf course
Ground transportation statco
0.07
0.12
00.06
F1 Store
0.06
0.05
0.1-
F1
0.04
0.08
0.04
TETEE
111E
1111
110
1110
1110
100
10
10
10a
10
10
10a
10a
10
10a
10a
10a
10
10
102
10a
Helipad
Nespial
Impoverished setilemenn
Interchamge
0.2
Scoto
0.066
0.44
12
2.01-
0.05
0.2
0.04
10
10
102
10a
10
10a
10
10a
10
10a
10
10a
10
10a
20
10a
Lake
Lighthouse
Miliary facility
Muli residential
0.2
0.2
0.3
Sont
Soge
20.06
0.1
0.1
10
10
10a
10
10a
10
10a
10
10a
10a
10a
20
Nuclear powerplant
building
gas facility
0.0
0.06
0.4
Scon
0.2
0.06
20..5
F1
0.2
0.04-
1111
1111
1000
100
10a
10a
10
10
10
10a
10
10
10
10a
10a
10a
102
10a
Parking for oc garage
Place of weeship
Police station
0.25
0.14
0.07
CROMA
SaIMAR
0.12
Fl
I-JEPA
0.05
100
10a
102
10a
100
10a
102
10a
10P
10a
102
10a
Figure 14: Sparse Probing all classes in fMoW-Senting (part ii)
31

---

Por
Prison
Race track
Railway bridge
0.1
F1 Stoxeee
F1 Scoee
0.06-
FI
F1
0.05
1811
111E
1111
110
100
10
10
10a
10
10
10a
10a
10
10
10
10a
10
10
10a
10a
Recreational facility
Road bridge
Ronway
Shipyard
0.4
0.32
0.0
0.4-
0.3
0.0.8
0.002
0.06
0.2
0.2
100
10
10
10a
10
10
10
10a
10
10a
10
10a
10
10a
102
10a
Shopping
Single-unit residearia
Snokestack
Solar
0.36
0.08
Scote
0.25
0.1-
0.006
0.05
0.15
0.04
100
10
102
10
10a
10
10a
10
10a
10
10a
10a
10a
20
Space facility
Storage
0.4
0.2
F1 Soore
Score
Scote
0..15
0.2
0.1
0.1-
0.04
0.05
1111
1111
1000
101
10a
10a
10
10a
10
10a
10
10a
10
10a
10
10a
102
10a
Swimming pool
Toll booth
Tower
0.15
0.16
CROMA
SaIMAR
I-JEPA
0.06
0.05
100
10a
102
10a
100
10a
102
10a
100
10a
102
10a
Figure 15: Sparse Probing all classes in MMoW-Sentine (part ii))
32

---

Tunnel opening
Waste disposal
treatment facility
0.12
Soore
Fl
22008
0.06
1111
1111
100
10a
202
10a
100
10a
20
10a
100
10a
20
10a
100
10a
202
10
Wind Farm
7o0
0.4
0.1
0.3
CROMA
27
0.05
00.2
1111
TTTTEE
110
10
10
10
10a
100
10
10
10a
Figure 16: Sparse Probing all classes in MMoW-Sentine (part iv)
33


Reviews:
[{"review_id": "LkWfwqsycN", "reviewer": "Reviewer_iZ73", "summary": "The paper presents a self supervised representation learning model for multimodal sentinel images. The model learns from geographically aligned optical and radar (sentinel-2 and sentinel-1, respectively) representations that are then used for downstream tasks, such as classification and segmentation. In the paper, classification and segmentation tasks are illustrated by using different approaches, namely, fine tuning, linear probing and nonlinear probing (MLP). Authors also show other quantitative evaluations (knn, kmeans over classes, a UMAP) to show the quality of the learned representation over SatMAE [26], which is the main competing method.\n", "questions": "\n- L91: I found the concept of \"optionally multimodal\" very interesting, and I think it could be better framed. I am not sure whether the proposed CROMA it is indeed optionally multimodal, but the fact that some image pairs are not available concurrently is a very common issues which is often solved by temporal composition, but might not be optimal for some fine grained monitoring applications. \n- L109: it is not completely clear to me why (beyond the ablations) RPE offer, sometimes, good performance when extrapolating over larger images. At the same time, i think, contrarily what stated afterwards, the wild variability of remote sensing images is not an issue: images are referenced absolutely to geographical coordinates and can be cropped following the constraints of the task and hardware. \n- What is not so clear to me, is how the choice of spectral channels for S2 is done, and how resolution is dealt with. S2 channels have a ground sampling distance that varies between 10m, 20m and 60m, while radar can vary depending on the processing level. It is unclear how patches are sampled and how the mismatches in resolution is dealt with, since if sampling eg a 80x80 pixels patch, the actual content varies a lot unless interpolated and upsampled, which could introduce artefacts in itself. I think these data preprocessing steps should be better explained. \n- It is unclear at which level of processing the data is used, whether L2A corrected or L1C Sentinel-2 data, and if so, how the processing is performed. It is also mentioned that 12 bands are used, but in facts S2 has 13. Just that some of these bands are used for sensing properties of the atmosphere such as clouds and aerosols, and do not help in performing land-cover / land-use related modelling. Again, I think that some of these aspects should be clarified in the main text. \n- I wonder why [86] has not been included in the baselines, as it is one of the main papers highlighted in the related work section dedicated to RS representaiotns. \n- In light of the above points, some datasets used to test the model have very different properties and characteristics. how is the CROMA model pretrained on 12 S2 channels, retrained for each of the benchmark making use of the specific data provided? eg the fMoW, as far as i remember, only make use of 8 channel and not 12. \n- L203 and following: It is unclear why no other optical image participates in the definition of negative samples, this could be maybe beneficial to encode differences in landcover at different locations, or account for seasonality effects. \n- L240: it is unclear what \"single label benchmarks\" are. \n\n\n\n", "limitations": "- Limitations shortly mentioned in the conclusion section. I agree with those highlighted. \n- I had the feeling when reading the paper, the fact that CROMA was performing better than SatMAE, that is all it was needed. I think that SatMAE comes with pros and cons that are not very well discussed and framed. I think that since the paper is comparing and improving directly upon SatMAE, these aspects could have been better presented. \n- I really missed a test on non Sentinel data. I do agree that Sentinel is a great data source, but it is not the only source used, particular when studies need to go back before 2015/2016. It would have been nice to see some results on other satellite data, but I understand that this could have been too much work or out of scope, but I would nonetheless mention it (this goes beyond higher spatial and spectral resolution, it is often interesting to transfer models to lower spatial and spectral resolution). \n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper deals with an important topic, and the models and results presented in the paper are significant for a variety of applications making use of sentinel data. \n- Results are validated on well known datasets in the field, and show superiority over a series of strong baselines and several metrics. Ablation study is very complete and shows how the model performs under different changes in the modules. \n- The approach of combining masked out reconstruction and contrastive losses in learning SSL representations is, to the best of my knowledge, novel in the field of geographically aligned data. The use of different modalities is also interesting, although the synergistic use of optical and radar images is well known and studied, but for specific applications.\n- The paper is very dense but clear enough, well written and well structured. \n", "weaknesses": "I think that the paper is sound and although it touches upon a niche application of computer vision that might not be of wide interest for the NeurIPS audience, it could be a good contribution. However I have a series of comments that I think could be addressed and improve the paper. In general: \n- The data description and the explanation of the different levels of preprocessing for each dataset (eg atmospheric corrections from L1C to L2A), and how those influence the model, are not well explained. For instance, sentinel 2 has 13 channels, but of which only a subset are useful for land cover applications. Also, spatial resolution of the different S2 channels is very different (from 10 to 60m) and the one from S1 changes depending on the processing levels. \n- The different benchmark tested are characterised by different preprocessing and channel subsets, and it is unclear how the models are fine tuned in this setting and what is the dependency on those aspects. \n- I felt that the related work section at L72 makes plenty of references but it is not so good at clarifying some main lines of research, pros and cons of those. There are many papers, maybe too many, and it is hard to get some information out of that. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. We believe we can address all the concerns you raise.\n\n**Q**: *Optionally multimodal.*\n\n**A**: CROMA is indeed optionally multimodal. In sections 4.1 & 4.2, we only use the optical encoder (Sentinel-2-only tasks). In section 4.3, we use all three encoders (multimodal tasks). And in section 5.1, we use all three combinations (optical-only, radar-only, and joint). Crucially, each of the three combinations of encoders requires a single pretraining run that jointly pretrains these three encoders end-to-end.\n\n**Q**: *RPE.*\n\n**A**: RPE captures the relative locations between tokens in a transformer rather than their absolute locations in the sequence. As ALiBi [119] (ICLR ‘22) shows, RPE methods often fail to extrapolate to longer sequences effectively. PEG (ICLR ‘23) showed that a simple convolution between the 1st and 2nd ViT layers (along with discarding position embeddings) improves performance and enables extrapolation. For our application, we show that 2D-ALiBi outperforms PEG with and without extrapolation. 2D-ALiBi is a true RPE method that is both translation and rotation equivariant, these properties are desirable for EO imagery.\n\nThe sizes of imagery in RS benchmarks vary considerably. Still, we agree that users can query and crop imagery to their liking if they have access to the original data sources. Besides this, image-size extrapolation would be a significant feature for RS foundation models. By increasing or decreasing the number of patches, users can vary the amount of compute they wish to spend on a sample. To illustrate this, we finetune CROMA-B and SatMAE-B on EuroSat for 50 epochs on **96x96px** images—each model is trained once using the default hyper-parameters of each model. Then, we evaluate each model on the EuroSAT validation set at various resolutions:\n\n| Model\t|  Test Resolution -> | 32x32 | 64x64 | 96x96 | 120x120 | 224x224 |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| SatMAE-B  | \t| 64.1% | 84.2% | 99.1% | 98.3% | 61.4% |\n| CROMA-B |  \t| 90.7% | 97.6% | 99.2% | 98.3% | 83.1% |\n\nFor instance, if a user requires fast predictions only possible on 32x32px images, CROMA will outperform SatMAE by 26.6% on EuroSAT.\n\n**Q**: Data preprocessing.\n\n**A**: For pretraining, we use the SSL4EO dataset assembled, preprocessed, and published by another lab [85]. This dataset provides paired Sentinel-2 L2A data (atmospheric corrections removing the cirrus band, resulting in 12 bands) and Sentinel-1 GRD data—this is briefly stated on L220 of our paper. The data is already upsampled to a 10m per pixel spatial resolution for all relevant channels. According to the SSL4EO codebase, using Google Earth Engine, they query: (i) the 'COPERNICUS/S2_SR' collection, filtering images with greater than 10% cloud coverage, and bilinear spatial upsampling, and (ii) the 'COPERNICUS/S1_GRD' collection in interferometric wide mode with VV and VH channels. We selected the SSL4EO dataset for pretraining because it is the largest preprocessed Sentinel-1 & 2 imagery collection and can be easily downloaded, making replication feasible. Furthermore, given the data collected by SSL4EO, before feeding them into our models, we normalize all channels exactly like SatMAE [26]. In our revised appendix we will clarify how data is preprocessed.\n\n**Q**: *Handling 13 channels.*\n\n**A**: The Sentinel-2 benchmarks contain either 12 (with cirrus removed) or 13 channels (with cirrus included). When applying CROMA to benchmarks that contain 13 channels, we drop the cirrus band, giving our model 12 of the 13 available channels. We do not re-train CROMA for each benchmark—all CROMA models (CROMA-B, CROMA-L, and ablations) are pretrained on the SSL4EO dataset described above. fMoW-Sentinel contains 13 channels [26].\n\n**Q**: *SatViT-V2 comparison.*\n\n**A**: SatViT-V2 [86] only processes Sentinel-1 & 2 data stacked along the channel dimension—it is not optionally multimodal. Therefore, it cannot process the optical-only datasets we evaluate in sections 4.1 & 4.2. In section 4.3, we compare CROMA with SatViT-V2 on two multimodal benchmarks, BigEarthNet [126] and DFC2020 [137]—CROMA significantly outperforms SatViT-V2, which was pretrained using masked autoencoding.\n\n**Q**: *Negative optical samples in contrastive learning?*\n\n**A**: We selected this cross-modal contrastive learning (CMCL) framework because it is the standard in the literature. The representations of non-matching samples of the same modality are much closer to each other than the representations of matching samples of different modalities. This has been widely observed [*1, *2, *3] and is now coined the “modality gap”—nicely illustrated in Figure 1 of “Mind the Gap” [*1]. We observe this same modality gap between optical and radar representations in CROMA (both at initialization and convergence). Including optical samples as negatives in the CMCL calculation amounts to introducing very difficult negatives. Although “hard negatives” can improve representations, overly difficult negatives can hurt representations [*4, 36, 145].\n\nIn our appendix, we did experiment with hard negatives via hard negative mixing (HNM, [145])—this mixes optical representations with radar representations to build hard negatives. We show HNM hurts representations across all 6 evaluations. Using optical representations as negatives would create even more difficult negatives than mixed optical and radar representations.\n\n[*1] “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning,” in NeurIPS ‘22\n\n[*2] “Towards understanding the modality gap in CLIP,” in ICLR ‘23 Workshop\n\n[*3] “Understanding and constructing latent modality structures in multi-modal representation learning,” in CVPR ‘23\n\n[*4] “Contrastive learning with hard negative samples,” in ICLR ‘21\n"}, {"review_id": "RxZI8JACoq", "reviewer": "Reviewer_too9", "summary": "This paper proposes CROMA to align optical and SAR modal images via contrastive learning and reconstruction. Comprehensive experiments on three datasets have demonstrated the effectiveness of CROMA.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces a multi-modal representation using contrastive learning and reconstruction.\n2. The proposed CROMA has exceeded the SatMAE, which only used unimodal images.\n3. CROMA is more faster and effective than SatMAE.\n", "weaknesses": "1. CROMA only constructs pos-neg samples from different modal images. Why are image patches in different regions of the same modality not used as negative samples?\n2. There miss lots of important details. For example, the number of positive and negative samples is not discussed. \n3. How is the sampling ratio of positive and negative samples affected? What is the relationship between positive and negative sample sampling and effective inputs in reconstruction?\n4. I am worried about the theoretical innovation of the paper. The paper mainly focuses on the application of contrastive learning loss and reconstruction loss to remote sensing multimodal modeling, which is also very common in medical multimodal and multitemporal. As an extension of SatMAE, I don't know whether the innovation of the paper is enough for NeurIPS. Because there already exists many related papers for remote sensing[1][2].\n[1] Ayush K, Uzkent B, Meng C, et al. Geography-aware self-supervised learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10181-10190.\n[2] Manas O, Lacoste A, Giró-i-Nieto X, et al. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 9414-9423.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. We appreciate the recognition of the strengths of our work: the introduction of learning multimodal representations by jointly leveraging reconstruction and contrastive learning and the superiority over the current SoTA. We believe we can answer the concerns raised, but we’d like some clarification on one question. Please see our replies below:\n\n**Q**: *Why not use samples from the same modality as negatives in contrastive learning?*\n\n**A**: Please see our response to reviewer iZ73, \"**Q**: *Negative optical samples in contrastive learning?*\"\n\n**Q**: *Clarification on the number of positives and negatives.*\n\n**A**: We believe that the paragraph starting at L203 discusses this, including the equation below L207—but we acknowledge this could have been more clear. We use the most popular contrastive learning framework in the literature; this uses one positive sample (in our case, matching radar-optical representations), and negatives are all other samples from the batch (in our case, mismatching radar-optical representations)—SimCLR [99] best exemplifies this method. For CROMA-B, we use a batch size of 7200 (the largest that can fit onto a DGX server consisting of 640 GB of VRAM using bfloat16 precision); this means we use 7199 negatives. If our paper is accepted, we will have more space (with a 10th page) to clarify this in our main text.\n\n**Q**: *“How is the sampling ratio of positive and negative samples affected? What is the relationship between positive and negative sample sampling and effective inputs in reconstruction?”*\n\n**A**: We are not sure what is meant by these questions and would appreciate some clarification.\n\n**Q**: *Clarification on the novelty of our CROMA self-supervised learning algorithm.*\n\n**A**: We agree that the main innovation of our paper is combining reconstruction and contrastive losses for learning multimodal representations. Since these two objectives learn different types of representations, we believed combining them would learn more general representations than alone. And we demonstrate this experimentally in our ablations. We have outlined the related work in our paper, but admittedly, it was dense—if our paper is accepted, we will have an extra page of space to further clarify how CROMA differs from related methods. The vast majority of work that leverages reconstruction and contrastive learning to learn joint multimodal representations is in the image-text domain. For example, ALBEF [107] (NeurIPS ‘21), TCL [108] (CVPR ‘22), BLIP [110] (ICML ‘22), MaMMUT [111] (TMLR ‘23) and CoCa [49] (TMLR ‘22)—these were all clever iterations on prior work. In general, these frameworks independently encode full images (no masking) and masked text, then perform a contrastive objective between matching cross-modal samples. Next, they reconstruct the masked text (either with a masked language modeling or autoregressive language modeling objective) conditional on image representations. But none of these approaches were designed for 2D multimodal data. We are primarily inspired by CoCa and significantly adapt it to 2D multimodal data. The simplest way to adapt these frameworks to 2D data would have been to replace the text encoder with another ViT—but encoding mask tokens in a ViT hurts performance and is much slower than hiding patches and leveraging a decoder to predict the hidden patches (this is the main innovation of MAE [31] (CVPR ‘22)). Thus our CROMA framework can best be considered a combination of CoCa and MAE. By leveraging MAE-style masking, we save on compute and provide a target for reconstruction; the efficiency of performing contrastive learning on masked-out cross-modal samples was also found in concurrent work, FLIP [123] (CVPR ‘23). These innovations are significant and justify the broad audience that NeurIPS attracts. Two papers came to similar conclusions in parallel with us, CAV-MAE [50] (ICLR ‘23) and MaViL [52](available on arxiv, also under review), but these papers are in the audio-visual domain and are not designed for spatially aligned data that is ubiquitous to RS and other applications. Our multimodal decoder is designed explicitly for spatially aligned multimodal data as tokens in the decoder predict both hidden radar and optical patches corresponding to a precise location on the ground—this is not possible in the CAV-MAE or MaViL frameworks.\n\nAdditionally, with X-ALiBi, we bias the cross-attention matrix based on the relative locations between cross-modal patches—this is only possible with spatially registered data and is the first time position encoding has ever been leveraged in cross-attention. Finally, by providing true relative position encoding that is both translation and rotation equivariant, 2D-ALiBi outperforms the SoTA in position encoding for ViTs, PEG [118] (ICLR ‘23). ALiBi [119] (ICLR ‘22) is a groundbreaking method in NLP that is now the method of choice in many SoTA LLMs because of its extrapolation abilities—an extension of it to 2D data is also novel.\n\nOverall, CROMA builds and improves on methodology recently published in NeurIPS and conferences with similar audiences and impact. Not to mention, CROMA substantially outperforms SatMAE [26] (NeurIPS ‘22) and provides a more thorough evaluation of learned representations. We thank you for asking this question, as it demonstrates that the novelty of CROMA was not sufficiently articulated—we will integrate this discussion in our revision.\n"}, {"review_id": "SCapdpHv49", "reviewer": "Reviewer_vsHk", "summary": "This paper presents a CROMA, a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multi-modal representations. CROMA separately encodes masked-out multispectral optical and synthetic aperture radar samples and performs cross-modal contrastive learning. X- and 2D-ALiBi are also introduced to ensure the performance, which spatially biases the cross and self-attention matrices.", "questions": "1.\tThe main contributions are not clear. Many multi-model learning models have been proposed, what are your advantages and own features compared with them?\n2.\tWhat is FFT? I cannot find its full name in this manuscript.\n3.\tThere are three encoders in CROMA. What are the relations between them? What is the rationale behind the freed settings and parameters? How do you decide the input patch sizes?\n4.\tWhy do you extend ALiBi to a 2D version? Please explain the necessity.\n5.\tHow do you decide the MASK value (i.e., 75%)?\n", "limitations": "1.\tIntroduction is chaotic. There is no clear, logical flow between paragraphs, resulting in a disjointed content presentation. \n2.\tA meaningful literature review is missing. The authors only display many published literature. However, the relations between them and CROMA are not clear. Also, the inner relationships of the reviewed literature are confusing.\n3.\tThe experimental settings are unclear, preventing the readers from simulating your method.\n4.\tThe compared models are not enough, limiting the reliability of the results.\n5.\tThe experimental results are discussed as shallow. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "CROMA aims to address the multi-model learning problem in the remote sensing (RS) community, which is an important and hot topic. Also, many advanced techniques are adopted and combined properly to ensure the final results for different tasks. In sum, the proposed method is feasible.", "weaknesses": "However, due to the poor statements and organization, the main ideas of this work are hard to follow. Also, as mentioned above, CROMA combines some existing techniques to deal with its tasks. Thus, its novelty is limited for NIPS. Some detailed comments can be found in “Questions.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. Please see our replies below:\n\n**Q**: *Clarification of main contributions.*\n\n**A**: Please see our comments to all authors that clarify our contributions.\n\n**Q**: *What is FFT?*\n\n**A**: The term FFT does not appear in our document; perhaps the reviewer is referring to FFN? FFN stands for feedforward network—this is the standard way of referring to the feedforward network in a transformer. We will define the term in our revision.\n\n**Q**: *Clarification on CROMA’s three encoders and choice of patch size.*\n\n**A**: CROMA uses three encoders, (i) a unimodal radar encoder, (ii) a unimodal optical encoder, and (iii) a multimodal (i.e., radar-optical) encoder. The unimodal encoders use a standard ViT-B architecture (other than position encoding) with 8x8 patches. ViTs often use 16x16x3 or 8x8x3 patches (HxWxC); since our optical samples have 12 channels, 16x16x12 patches would lose information when linearly projected to the width of the transformer (i.e., 768 for ViT-B or 1024 for ViT-L). Conversely, selecting patches that are too small (i.e., when the number of pixels per patch is far fewer than the width of the transformer) leads to inefficient models [136]. Thus, an 8x8 patch size is a reasonable choice for CROMA.\n\nThe multimodal encoder in CROMA is a transformer that receives radar patch encodings at the bottom of the network and cross-attends to optical patch encodings. This approach is similar to CoCa and the original seq2seq transformer design in “Attention Is All You Need”. This explanation is best understood by consulting Figure 1 in our paper.\n\nPlease provide additional clarification on what is meant by “freed settings and parameters.” To be clear, all parameters in CROMA are learned end-to-end.\n\n**Q**: *Why invent 2D-ALiBi?*\n\n**A**: ALiBi (ICLR ‘22) is a SoTA relative position encoding method for transformers modeling 1D inputs. Its primary advantage is it can be trained on sequences of, say, 1024 tokens and make inferences on much longer sequences without needing to be re-trained or finetuned on that longer sequence. Because the sizes of imagery in remote sensing datasets vary considerably, we feel that foundation models would benefit from being able to process images with varying numbers of patches. We show how 2D-ALiBi outperforms a SoTA relative position encoding method for ViTs, called PEG (ICLR ‘23), that leverages a convolution between the 1st and 2nd ViT layers and discards position embeddings that are typically added to patch embeddings at the bottom of the network. Furthermore, a model that can flexibly handle images of different sizes can offer a superior trade-off between compute and performance. To illustrate this, we finetune CROMA-B and SatMAE-B on EuroSat for 50 epochs on **96x96px** images—each model is trained once using the default hyper-parameters of each model. Then, we evaluate each model on the EuroSAT validation set at various resolutions:\n\n| Model\t|  Test Resolution -> | 32x32 | 64x64 | 96x96 | 120x120 | 224x224 |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| SatMAE-B  | \t| 64.1% | 84.2% | 99.1% | 98.3% | 61.4% |\n| CROMA-B |  \t| 90.7% | 97.6% | 99.2% | 98.3% | 83.1% |\n\nThis demonstrates that CROMA is more robust to differences between train/test-inference image sizes. This robustness permits users to select more compute-efficient image sizes without dramatically sacrificing performance. For instance, if a user requires fast predictions only possible on 32x32px images, CROMA will outperform SatMAE by 26.6% on EuroSAT.\n\nBeyond extrapolating to larger images, 2D-ALiBi outperforms 2D-sinusoidal embeddings and PEG without extrapolation, i.e., testing on the exact resolution as training. We believe this is due to the true relative position encoding of 2D-ALiBi, which is both translation and rotation equivariant. These properties complement data in the remote sensing domain that is overhead imagery whose representations should be translation and rotation invariant.\n\n**Q**: *Why a 75% mask ratio?*\n\n**A**: We select our hyperparameters by considering both performance and cost. As shown in Table 5, a mask ratio of 50% outperforms a mask ratio of 75% by 0.1%, averaged across 6 evaluations. But a 50% mask ratio is 1.8x slower to train due to the encoders processing more patches. This highlights a benefit of CROMA—it is robust to the choice of mask ratio.\n\n**Q**: *Experimental settings.*\n\n**A**: We provide experimental conditions in our appendix, as we focused on areas of most interest to a NeurIPS audience in our main text. We also anonymously share all code, pretrained models, and preprocessed datasets. Please see the appendix.\n\n**Q**: *Model comparisons.*\n\n**A**: We compare CROMA to all relevant foundation models in our application. Furthermore, some of our ablations are equivalent to other SoTA methods that have yet to be explored for remote sensing. For example, only leveraging contrastive learning amounts to adapting Fast Language Image Pretraining [124] (FLIP, published at CVPR ‘23) to our domain, as FLIP performs cross-modal contrastive learning with masked-out samples. An ablation in our appendix with VICReg and a patch-wise invariance loss objective is equivalent to adapting VICRegL [143] (NeurIPS ‘22) to our domain. In our revision, we will clarify the connection between these ablations and other SoTA computer vision methods not yet explored in remote sensing.\n\n**Q**: *Discussion of experimental results.*\n\n**A**: We agree that our submission would benefit from a more detailed discussion of the results. Our main text focused on using our space to validate CROMA as thoroughly as possible experimentally—hence the probing, kNN, K-means, and segmentation experiments. If our paper is accepted, we will have more room to incorporate a deeper discussion in a revised version.\n"}, {"review_id": "p46cH2M5qd", "reviewer": "Reviewer_FExC", "summary": "The paper presents a new SSL representation learning framework remote sensing and earth observation data. The presented framework combines a contrastive objective with a reconstruction objective working on single or multi-modal inputs i.e. multispectral satellite data and synthetic aperture radar data. Cross-modal learning is done by cross-attention of both individually encoded modalities, which are fused and decoded by one lightweight decoder.\n\nIn a wide range of experiments, the authors are able to demonstrate the proposed approach capability to outperform baseline and current SOTA approaches.\n\nAlthough this work displays rather a novel combination of already known approaches, I think it is interesting given the insightful adaptation of these approaches to the remote sensing and earth observation domain. I really enjoyed reading it. What I really like is the idea to use RPE as presented by the extension of ALiBi towards multispectral 2dim signals allowing to deal with different resolutions of satellite data. This particular characteristic of satellite data is very often neglected.", "questions": "- (Q1) How would you extend to more than two modalities? I am asking since in remote sensing and earth observation there are very often multiple modalities / sensors available. How would you model the cross-attention?", "limitations": "- (L1) The paper does not show the presented approach being capable of generalizing to other problem domains beyond remote sensing or earth observation. I could imagine that there exist other problem domains where multiple sensors are available. Showing how this approach performs in such a scenario would strengthen this work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- (S1) The use of RPE with its 2d extension of ALiBi including X-ALiBi. I think this is interesting since it aims to tackle the multi-resolution nature of individual bands of remote sensing data.\n\n- (S2) multi-modal representation is optional i.e. it performs well with only one modality if needed. This is in particular interesting for remote sensing scenarios such as natural disasters, where fast response is important but one of the two satellites is not available but will need a couple of days to fly over the target region.\n\n- (S3) The wide range of experiments including multiple datasets and downstream tasks all able to demonstrate the outperformance of the presented approach.\n\n- (S4) A broad set of ablation studies providing insights about the inter-working and contribution of each component of the proposed method.", "weaknesses": "- (W1) see (L1) under limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments on our work. We appreciate you recognizing the strengths of our work: (i) the introduction of X- and 2D- ALiBi, (ii) the optionally multimodal nature of CROMA, (iii) the extensive evaluation across methods, tasks, and datasets, and (iv) the thorough ablation. Although our paper is dense, we appreciate that the paper was really enjoyable to read. We will address two points individually:\n\n**Q**: *How would you model the cross-attention with more than two modalities?*\n\n**A**: Extending CROMA to more than two modalities is very exciting. Let’s imagine we are provided with higher-resolution RGB imagery spatially aligned with Sentinel-1 and 2 data. In this case, we could encode this high-res sample separately with a unimodal encoder, just like CROMA encodes radar and optical samples. Then a multimodal encoder could cross-attend to both sets of optical encodings (high-res and Sentinel-2) and bias the cross-attention matrix based on the distance between patches via X-ALiBi. In practice, this would mean repeating the X-ALiBi bias matrix along the key dimension, i.e., from a shape of (batch_size, heads, queries, keys) to (batch_size, heads, queries, 2*keys). If the patches between modalities do not perfectly align—for example, if the high-res sample used patches that were 16x16x3 at 1m spatial resolution—then we’d have to build another X-ALiBi matrix by calculating the relative distance between each query and key patch and concatenate this high-res bias matrix with the Sentinel-2 bias matrix along the key dimension. If there were 196 high-res patches, then the new X-ALiBi bias matrix would be of shape (batch_size, heads, queries, keys+196). In general, we believe that as long as the relative locations between cross-modal patches are known, then X-ALiBi can be leveraged. For example, imagine we have ground-level imagery with known coordinates. We could bias the cross-attention matrix based on the 2D distance between a ground-level representation and various patches in the satellite image. This idea could be extended to multiple ground-level images for a single satellite image. This research direction excites us—thank you for prompting this discussion.\n\n**Q**: *Validation on other domains with multiple sensors.*\n\n**A**: We agree that validating CROMA on other sensor data would strengthen our work. However, our paper is already quite dense. We look forward to seeing other groups apply CROMA to their applications. The potential for CROMA to be used broadly is one of our main reasons for targeting NeurIPS.\n"}]

Author rebuttal:
We thank all reviewers for their feedback. Before replying to reviewers individually we will restate our contributions:

* We leverage reconstructive and contrastive objectives to learn joint multimodal representations. This is not only novel for Earth Observation (EO) but is novel for multimodal 2D data. Of the many multimodal learning frameworks recently published, only two also learn multimodal representations of 2D data via these combined pretraining objectives: CAV-MAE [50] (ICLR ‘23) and MaViL [52] (available on arxiv, also under review). Both CAV-MAE and MaViL were developed concurrently with our work. Both frameworks are designed for audio-visual signals, not spatially aligned sensor data that we focus on. This feature lets us fuse corresponding cross-modal patches spatially—learning joint multimodal patch representations. We are glad that reviewers iZ73, too9, and FExC acknowledge this contribution.
* We extend ALiBi [119] (ICLR ‘22) to 2D data and show that 2D-ALiBi outperforms PEG [118] (ICLR ‘23) for our application and is novel to ViTs. Additionally, our X-ALiBi method is the first time position encoding has been leveraged in cross-attention, not just in our application. These methods allow CROMA to effectively generalize to smaller or larger images at test-time without further training. This means that CROMA models—and ViTs that will leverage our work by using 2D-ALiBi—offer a superior trade-off between accuracy and compute. We are glad that reviewer FExC “really likes” this contribution and seems interested in building on it.
* We thoroughly demonstrate that CROMA significantly outperforms the previous SoTA, SatMAE [26] (NeurIPS ‘22). Our evaluation consists of finetuning, linear probing, nonlinear probing, kNN classifying, and K-means clustering the learned representations of CROMA and all other SoTA foundation models for Earth Observation under identical conditions. Additionally, two of our ablations are equivalent to adapting other SoTA algorithms in computer vision to our domain: (i) Fast Language Image Pretraining (FLIP [124], published at CVPR ‘23) performs cross-modal contrastive learning with masked-out samples to efficiently outperform CLIP [124], and (ii) VICRegL [143] (NeurIPS ‘22) combines the VICReg [142] objective between image representations and an MSE patch-wise invariance objective to learn SoTA local representations (see this experiment in our appendix). CROMA outperforms both. We are glad that reviewers iZ73, too9, and FExC view our extensive experiments and ablations as a strength.

We acknowledge that our paper is dense and are glad reviewers iZ73 and FExC liked our writing and presentation. Our extensive experiments and ablations were only made possible by this density and by assuming our readers had a prior understanding of related work—specifically, vision transformers, masked autoencoders, contrastive learning, and prior foundation models for EO. Without this background, we completely understand how our paper may be challenging to comprehend. Should our paper be accepted to this conference, we will be granted a 10th page which we will dedicate to providing more background, clarifying our contributions, and including more experimental conditions in our main text (as of now, much of our experimental conditions are in the appendix).

Our paper improves on methods recently published at this conference and conferences with similar audiences and impact. As reviewer FExC correctly points out, many other domains feature multiple sensors whose data are spatially registered. Our framework can be directly used for these applications. We also see tremendous potential for X- and 2D-ALiBi, (or variants building on them, e.g. 3D data) that can be broadly applied to transformers. We are targeting this conference to publish our work for these reasons, not only because we outperform SatMAE.

We are happy to continue our discussions with all reviewers.

Meta-review: This paper introduces a framework for self-supervised, multi-modal remote sensing, showing that it outperforms SatMAE on a range of tasks. While there is disagreement among the reviewers, the main objection raised is "novelty" which is difficult to evaluate in settings where application-specific challenges lead to innovations that adapt existing ML tools. Of the two dissenting reviewers, one lowered their review from a borderline accept after raising a request for new experiments only at the end of the discussion phase, when authors were unable to respond. The other dissenting review did not provide sufficient detail or support for their concerns. Accordingly, I recommend acceptance.
Decision: Accept (poster)