SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Classification results on four benchmarks, under finetuning (FT), and frozen linear (LP) and nonlinear (MLP) probing. * denotes originally reported results; we obtain all other results under identical conditions.
Method	Backbone	BigEarthNet (10%) mAP	fMoW-Sentinel (10%) Top 1 Acc.	EuroSAT Top 1 Acc.	Canadian Cropland Top 1 Acc.
FT	MLP	LP	FT	MLP	LP	FT	MLP	LP	FT	MLP	LP
radar↔optical [71]	ResNet50	77.65	78.79	77.44	32.03	6.46	6.01	96.31	86.35	78.81	57.44	58.09	55.55
radar↔optical [71]	Swin-T	86.41	78.65	77.93	52.01	28.54	31.06	98.09	93.50	94.78	70.98	60.36	57.05
MAE [31, 70]	ViT-S	86.15	81.70	75.94	51.79	31.70	27.69	98.78	94.46	91.80	74.02	59.07	48.38
DINO [72, 70]	ViT-S	87.04	84.96	81.58	52.79	35.62	32.64	98.63	97.07	96.07	75.27	67.35	59.94
I-JEPA [73]	ViT-B	85.92	84.27	80.80	53.54	35.76	32.35	99.20	96.60	95.63	75.13	66.69	60.17
SatMAE [26]	ViT-B	85.94	83.48	79.36	57.20	37.28	35.17	99.20	97.28	96.61	73.58	66.02	60.40
CROMA	ViT-B	87.58	86.29	85.04	54.47	39.67	38.42	99.22	97.89	97.59	76.17	67.62	63.39
SatMAE [26]	ViT-L	86.18/82.62*	84.01	80.29	58.19	38.18	36.76	99.35/98.98*	97.67	97.65	74.06	67.03	61.75
CROMA	ViT-L	88.29	86.46	85.01	59.02	40.07	39.17	99.46	98.04	98.02	78.07	67.94	64.02
Classification Results. CROMA ranks first, averaged across four Sentinel-2 (multispectral optical) benchmarks, under finetuning, frozen linear and nonlinear probing, kNN classification, and  $K$  -means clustering (Table 1, 2). The only case where a SatMAE model outperforms a CROMA model of the same backbone is finetuning on the fMoW-Sentinel dataset. However, SatMAE was pretrained on fMoW-Sentinel—meaning there is no distribution shift between pretraining and finetuning. This gives SatMAE an advantage on the fMoW-Sentinel benchmark since downstream performance is impacted by the similarity be
tween upstream and downstream data [82, 83, 84]. Despite this observation, CROMA-B outperforms SatMAE-B on fMoW-Sentinel under linear probing ( $\uparrow$ $3.3\%$ ), nonlinear probing ( $\uparrow$ $2.4\%$ ), kNN ( $\uparrow$ $5.3\%$ ), and  $K$ -means ( $\uparrow$ $2.4\%$ ). Additionally, CROMA models are more than  $4\times$  faster during fine-tuning and inference than their SatMAE counterparts (please see Appendix §A.4.2 for a comparison). On all evaluations, CROMA outperforms a ViT-B model pretrained using the I-JEPA framework on the SSL4EO dataset—I-JEPA is the current SoTA framework for learning self-supervised representations of ImageNet [73]. We also plot UMAP [85] embeddings (Fig. 3); CROMA shows a strong separation of EuroSAT classes.
Table 2: Non-parametric kNN classification and  $K$  - means clustering results. * denotes  $10\%$  of the training set.
Method	Backbone	fMoW-Sent.* Top 1 Acc.	EuroSAT Top 1 Acc.	Can. Crop. Top 1 Acc.
kNN	K-means	kNN	K-means	kNN	K-means
radar↔optical [71]	ResNet50	7.07	3.94	52.09	23.52	49.82	23.15
radar↔optical [71]	Swin-T	19.46	7.93	85.07	56.91	48.22	21.57
MAE [31, 70]	ViT-S	22.25	7.54	87.33	41.50	56.01	18.31
DINO [72, 70]	ViT-S	28.89	8.23	94.20	45.83	60.70	20.22
I-JEPA [73]	ViT-B	25.45	8.01	89.02	42.89	57.48	18.06
SatMAE [26]	ViT-B	26.76	8.98	89.28	44.87	56.20	20.14
CROMA	ViT-B	32.03	11.35	95.26	76.06	61.25	22.55
SatMAE [26]	ViT-L	26.77	9.17	89.57	38.48	56.93	19.65
CROMA	ViT-L	29.54	10.12	94.70	61.24	59.41	21.27
Figure 3: UMAP embeddings and  $K$ -means clustering accuracies of CROMA (ViT-B), SatMAE (ViT-B) [26], DINO (ViT-S) [70], and radar  $\leftrightarrow$  optical (Swin-T) [71] models on EuroSAT [77].
Sparse Probing. Inspired by the sparse probing of language models [86], we sparsely probe the 768-dimensional (ViT-B backbone) image representations by restricting the linear probe to  $k$  dimensions. For each class, we rank all dimensions and then train a linear probe on the top  $k$  to perform binary classification (this setup follows [86], please see Appendix §A.4.2 for more details). This experiment helps us understand the information contained in the learned representations. For example, in Fig. 4 (B), there is a large drop in F1 score when decreasing  $k$ -indicating that BigEarthNet's "beaches, dunes, sands" class is not well represented by individual dimensions. Rather, this class is represented by a composition of many features that are correlated with beaches. Conversely, BigEarthNet's "agriculture with natural vegetation" class maps well to a single dimension of CROMA's representations, achieving an F1 score of  $49\%$  when  $k = 1$  (Fig. 4 (A)). In Appendix §A.6, we show sparse probing results for all classification tasks and for all clas
Figure 4: Sparse probing two BigEarthNet classes.
Segmentation Setup. We evaluate all ViT-based models on three Sentinel-2 segmentation benchmarks. ① The DFC2020 dataset [87] (46,152 train samples and 8,874 validation samples). ② A subset of the Dynamic World dataset [88] that was annotated by experts; hence we call it DW-Expert (20,422 train samples and 51,022 validation samples). DW-Expert is a new, high-quality benchmark that was annotated with the help of high-resolution satellite and street-level imagery. ③ The MARIDA dataset [89] (1,682 train samples and 1,615 validation samples), which is a small, sparsely labeled dataset of marine debris. For all tasks, we linear probe frozen patch encodings, i.e.,  $\mathrm{Linear}(\mathcal{E}_0)$ . We crop images of  $96 \times 96$  from the original images ( $256 \times 256$  for DFC2020,  $510 \times 510$  for DW-Expert, and  $256 \times 256$  for MARIDA). We train and evaluate all models on these  $96 \times 96$  images, which is the default image size for SatMAE, enabling a fair comparison with the SoTA.
Segmentation Results. CROMA outperforms SatMAE by averages of  $5.4\%$  and  $6.4\%$  for ViT-B and ViT-L backbones, respectively (Table 3). These results demonstrate that CROMA effectively learns fine-grained patch-level features useful for dense prediction tasks like semantic segmentation.
Table 3: Semantic segmentation (mIoU) results on three Sentinel-2 benchmarks.
Method	Backbone	DFC2020	DW-Expert	MARIDA
MAE	ViT-S	35.63	46.63	48.06
DINO	ViT-S	32.34	48.34	49.38
I-JEPA	ViT-B	36.72	50.82	53.85
SatMAE	ViT-B	45.53	51.03	58.17
CROMA	ViT-B	46.67	58.55	65.56
SatMAE	ViT-L	44.13	51.50	57.12
CROMA	ViT-L	49.78	58.71	63.32
3.2 Radar and Radar-Optical Experiments
Multimodal Setup. BigEarthNet and DFC2020 also contain spatially and temporally aligned Sentinel-1 samples alongside Sentinel-2. This allows us to evaluate our multimodal representations and directly compare them with optical-only representations on the same benchmark. We linear probe frozen image representations for BigEarthNet, i.e., Linear(CONCAT( $\mathcal{R}_{\mathrm{R}}$ ,  $\mathcal{R}_{\mathrm{O}}$ ,  $\mathcal{R}_{\mathrm{RO}}$ ), and patch encodings for DFC2020, i.e., Linear(CONCAT( $\mathcal{E}_{\mathrm{R}}$ ,  $\mathcal{E}_{\mathrm{O}}$ ,  $\mathcal{E}_{\mathrm{RO}}$ ). Concurrent with this work, DeCUR [75] learns radar and radar-optical representations via their novel multimodal framework. They fit a linear probe on top of DeCUR's representations on  $1\%$  of the BigI results on the BigEarthNet validation set; we follow this experimental model for a direct comparison.
Table 4: Multimodal (Sentinel1 & 2) linear probing results on classification and segmentation benchmarks.
Model	BigEarthNet mAP	DFC2020 mIoU
SatViT-V2 [74]	79.80	46.20
CROMA-B	86.24	51.58
CROMA-L	86.20	53.24
Multimodal Results. CROMA significantly outperforms the joint multimodal representations learned by SatViT-V2 [74] (Table 4). Our joint multimodal representations outperform our optical-only representations by  $1.2\%$  on BigEarthNet (both CROMA-B and CROMA-L) and by  $4.9\%$  (CROMA-B) and  $3.5\%$  (CROMA-L) on DFC2020 (Table 1, 3, 4)—justifying our multimodal approach. CROMA's radar-only and radar-optical representations also outperform the concurrent DeCUR under a linear probing experiment (Table 5).
Table 5: Linear probing radar and radar-optical representations.
Method	BigEarthNet (1%)
mAP
Radar	Radar-Opt.
DeCUR	73.7	79.4
CROMA	75.7	81.8
Table 6: Linear probing ablation results on radar-only ("R"), optical-only ("O"), and joint radar-optical ("RO") inputs. We consider both performance and cost to select our design. All rows below "all default" report the performance differences between ablated cases and the default.
Case (default)	Ablation	batch size	cost	Classification (mAP)	Segmentation (mIoU)	Avg
R	O	RO	R	O	RO
all default	7.2k	1.0×	78.2	84.5	84.8	40.8	56.0	56.5	66.8
1	objectives (both)	MAE-only	7.2k	1.0×	-10.4	-6.0	-5.6	-9.2	-8.4	-5.1	-7.5
contrast-only	14k	0.6×	-3.2	-3.0	—	-2.9	-3.7	—	—
2	position encoding (2D-ALiBi + X-ALiBi)	2D-ALiBi	7.2k	1.0×	-0.2	-0.2	-0.3	0.1	0.0	-0.9	-0.2
PEG [90]	6.9k	1.0×	-0.3	0.0	-0.1	-0.6	-1.0	-1.9	-0.7
2D-sinusoidal	7.2k	1.0×	-3.8	-2.9	-1.3	-2.2	-0.1	-0.2	-1.7
3	masking (independent 75%)	shared 25%	2.4k	3.0×	-0.5	-1.0	-1.1	1.1	0.5	-0.2	-0.2
shared 50%	3.7k	1.8×	-0.3	-0.7	-0.7	1.0	0.6	0.1	0.0
shared 75%	7.2k	1.0×	-0.8	-0.5	-0.3	0.0	0.1	0.0	-0.2
independent 50%	3.7k	1.8×	-0.2	-0.5	-0.5	0.9	0.7	0.3	0.1
4	MAE target (radar & optical)	radar-only	7.2k	1.0×	-0.1	-0.2	-0.6	0.3	-0.2	-1.4	-0.5
optical-only	7.2k	1.0×	-0.1	-0.2	-0.2	-0.1	-0.1	-0.6	-0.2
5	MAE decoder (depth=1, dim=512)	depth=6, dim=768	4.3k	1.7×	0.1	-0.1	0.4	0.2	0.2	0.1
6	scale (ViT-B, epochs=300)	ViT-B, epochs=300	7.2k	3.0×	1.6	0.5	0.4	1.7	1.3	0.5
ViT-L, epochs=600	3k	15×	2.5	0.4	0.5	2.8	0.9	0.1
4 Ablation Analysis
4.1 CROMA Design
Ablation Setup. We ablate the CROMA design by pretraining CROMA-B models for 100 epochs unless stated otherwise. For each ablation, we set the maximum batch size that can fit into 640 GB of VRAM (with bfloat16 precision); adjusting the batch size results in better comparisons between approaches. For classification on BigEarthNet, we linear probe frozen unimodal and multimodal representations, i.e.,  $\mathrm{Linear}(\mathcal{R}_{\mathrm{R}})$ ,  $\mathrm{Linear}(\mathcal{R}_{\mathrm{O}})$ ,  $\mathrm{Linear}(\mathcal{R}_{\mathrm{RO}})$ . For segmentation on DW-Expert-120, we linear probe frozen patch encodings, i.e.,  $\mathrm{Linear}(\mathcal{E}_{\mathrm{R}})$ ,  $\mathrm{Linear}(\mathcal{E}_{\mathrm{O}})$ ,  $\mathrm{Linear}(\mathcal{E}_{\mathrm{RO}})$ . For BigEarthNet, we train on the same  $10\%$  split as §3.1, but report results on the combined validation and test sets (236,130 samples). For DW-Expert-120, we select  $120 \times 120$  images (CROMA's default image size) from Dynamic World's [88] original Sentinel-2 images and match them, in space and time, with Sentinel-1 images—forming a high-quality multimodal segmentation dataset. DW-Expert-120 consists of 10,200 train and 45,367 validation samples.
Ablation Results. ① Removing either self-supervised objective significantly degrades accuracy on both classification and segmentation tasks. This justifies a fundamental design choice, i.e., combining contrastive and reconstruction approaches for aligned multimodal data. ② Although our primary motivation for 2D-ALiBi is to enable input size extrapolation (see §4.2), we find it also improves linear probing accuracy over a SoTA RPE method for ViTs, even when no extrapolation occurs. Averaged across six evaluations, 2D-ALiBi without X-ALiBi outperforms a Position Encoding Generator (PEG, [90]) by  $0.5\%$  and 2D-sinusoidal embeddings by  $1.5\%$ . (We adapt PEG to MAE-style masking by zero-filling the masked-out patches during pretraining, inspired by [91]). Leveraging X-ALiBi with 2D-ALiBi further improves average performance by  $0.2\%$  (Table 6)—particularly improving multimodal performance, which is our motivation for X-ALiBi. We believe there are two reasons for 2D-ALiBi's superior performance—evidence for both is provided in Appendix §A.2. First, 2D-ALiBi learns rotation-invariant representations despite not being trained with rotation-invariant objectives; this is a desirable property of satellite imagery that likely improves classification performance. Second, 2D-ALiBi prevents patch-wise representational collapse (i.e., the representations of patches within an image become similar, losing local information) often observed with contrastive objectives [47]; preserving patch-wise diversity likely improves segmentation performance. ③ Lower mask ratios hurt classification and help segmentation but increase costs. For both  $50\%$  and  $75\%$  mask ratios, independently masking our modalities (i.e., optical and radar samples are masked differently) slightly outperforms shared masking. Although  $50\%$  independent masking outperforms  $75\%$  masking by  $0.1\%$ , we select the higher mask ratio because it offers a  $1.8 \times$  speedup. ④ A multimodal target (14 channels) outperforms an optical-only target (12 channels) by  $0.2\%$ , which outperforms a radar-only
target (2 channels) by  $0.3\%$  (Table 6); this implies that reconstructing more information leads to more general representations. A multimodal target is especially important for learning rich multimodal representations. A larger decoder improves segmentation but costs  $1.7\times$  more. MAE showed that a deep decoder improves linear probing accuracy [31], but CROMA is very robust to decoder size. Therefore, we select the most efficient, i.e., a 1-layer, 512-d decoder. We design CROMA by pretraining ViT-B models for 100 epochs and consider linear probing performance and cost. Thus, our design may not be optimal when scaling up. Nevertheless, scaling to more epochs and a larger model improves representations, especially radar-only representations.
In Appendix §A.1 we also experiment with all showing minimal or negative impacts—task weights, more decoder sizes, VICReg [92], mean squared error loss between cross-modal patch encodings (inspired by [93]), InfoNCE loss between cross-modal patch encodings (inspired by [94]), hard-negative mixing [95], and lower-masked tuning at the end of pretraining [68].
4.2 Extrapolating Representations to Larger Images
Extrapolation Setup. For three CROMA-B models (2D-sinusoidal, PEG [90], and 2D-ALiBi) pretrained for 100 epochs, we finetune all parameters (along with a linear head) on Sentinel-2 images from DW-Expert-120 ( $120 \times 120$  pixels). Then, we directly evaluate the models on images of varying sizes to test their ability to extrapolate at test-time. We create validation sets with different image sizes by cropping from the original  $510 \times 510$  images in Dynamic World [88]. Regardless of image size, we retain the  $10\mathrm{m}$  per pixel spatial resolution on which the models were trained—extrapolating to smaller or larger geographic areas at test-time. For 2D-sinusoidal embeddings, we also evaluate an embedding interpolation algorithm often used when applying ViTs on larger images [31, 26, 48, 96].
Extrapolation Results. 2D-ALiBi outperforms PEG by  $1.7\%$  on  $504\times 504$  pixel images (Table 7). Amazingly, we only see a  $0.7\%$  drop in mIoU when testing on areas  $17.64\times$  larger than those on which our model was trained—effectively generalizing from 225 to 3,969 patches per image. We achieve this by extending ALiBi [66] to 2D inputs by penalizing attention scores based on the Euclidean distance between
Table 7: Segmentation results (mIoU) of models trained on  $120 \times 120$  resolution and evaluated on various resolutions.
Test Res. Method	Train Res.
48	96	120	224	384	448	504
2D-Sinusoidal	54.4	58.5	59.2	38.2	24.9	21.8	19.5
2D-Sinusoidal w/ interp.	50.7	58.6	59.2	58.3	56.2	55.5	54.9
PEG [90]	55.1	58.0	58.3	58.5	57.7	57.4	57.1
2D-ALiBi	56.3	59.0	59.3	59.5	59.1	59.0	58.8
query-key pairs. We may achieve even better results by encoding directions in a subset of attention heads or learning scalars [97]; we leave these investigations to future work. We believe X- and 2D-ALiBi have tremendous potential beyond our CROMA framework, for instance, by extending these methods to additional modalities, viewing angles, or 3D data.
Remote Sensing Representations. Deep learning for remote sensing has been an active research area for many years. Researchers have leveraged self-supervised learning frameworks in the last few years to learn representations of remote sensing data that can be widely leveraged across societally important downstream tasks. In designing contrastive pretext tasks, remote sensing researchers built positive pairs by leveraging spatial [98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108], temporal [24, 25, 109], spectral [110], or cross-modal [111, 112, 113, 114, 115, 116] information. In designing reconstructive pretext tasks, researchers held-out spectral bands [117], pixels [118, 119, 120, 121], and resolutions [27]. However, these studies—including the concurrent Scale-MAE [27] and influential studies tile2vec [108], GASSL [24], and SeCo [25]—were either performed at smaller scales or only included wavelengths from the visible (red, green, and blue) spectrum; this limits their utility on downstream applications since wavelengths from the non-visible spectrum contain information critical to many remote sensing tasks [53, 122, 123, 124]. For example, the ability to measure the reflectance of objects in the near-infrared portion of the wavelength is extremely valuable in applications related to vegetation identification [125], health and productivity [126], as well as identifying water bodies [127], soil moisture [128], and vegetation water content [129].
Relative Position Encoding for ViTs. SoTA transformers in natural language processing use RPE [130, 131, 132], but fixed 2D-sinusoidal embeddings still predominate ViTs. The improved inductive
bias of RPE over absolute position encoding can offer improved performance and, sometimes, the ability to extrapolate to larger images at test-time, i.e., directly applying a model trained on one image size to another without further training. This extrapolation ability would add considerable value to remote sensing foundation models since image sizes vary widely—images are often cropped from large scenes down to smaller sizes chosen idiosyncratically. Positional Encoding Generator (PEG, [90]) is a SoTA RPE method that uses a convolution between ViT layers; PEG was demonstrated to significantly outperform other RPE methods when tested on larger images. iRPE [133] is another, more complex, RPE method for ViTs; however, it demonstrates no extrapolation ability. We found no prior work that leverages RPE in cross-attention.
We propose a novel framework for learning unimodal and multimodal representations for Earth Observation by jointly leveraging contrastive and reconstruction self-supervised objectives. We extend a SoTA position encoding method for 1D sequences to 2D inputs and cross-attention; to the best of our knowledge, this is the first time explicit position encoding has been leveraged in cross-attention. These strategies allow our models to extrapolate to larger images at test-time and improve performance on both unimodal and multimodal data. We extensively evaluate our pretrained models on diverse tasks and methods, outperforming the previous SoTA. Although our method is designed for multimodal satellite imagery, it can be leveraged on other applications that offer spatially aligned multimodal data, for instance, medical imaging or autonomous vehicles.
The main limitation of our work is our focus on static-in-time Sentinel-1 & 2 data; in the future, we will explore other sensors that offer higher spatial or spectral resolutions and time-series data. Despite this limitation, Sentinel-1 & 2 are the most widely used sources for satellite imagery in research—making our models an incredible resource for the remote sensing community and users of remote sensing-derived products, for instance, geographers, economists, or environmental scientists.
This work was made possible with compute provided by Cyxtera Technologies and the NVIDIA Academic Hardware Grant Program. Anthony thanks the Vector Institute for their financial support during his master's program.
[1] Jihyeon Lee, Nina R. Brooks, Fahim Tajwar, Marshall Burke, Stefano Ermon, David B. Lobell, Debashish Biswas, and Stephen P. Luby. Scalable deep learning to identify brick kilns and aid regulatory capacity. Proceedings of the National Academy of Sciences, 2021.
[2] Arati Paul, Soumya Bandyopadhyay, and Uday Raj. Brick kiln detection in remote sensing imagery using deep neural network and change analysis. *Spatial Information Research*, 2021.
[3] Fen Chen, Ruilong Ren, Tim Van de Voorde, Wenbo Xu, Guiyun Zhou, and Yan Zhou. Fast Automatic Airport Detection in Remote Sensing Images Using Convolutional Neural Networks. Remote Sensing, 2018.
[4] Raian V. Maretto, Leila M. G. Fonseca, Nathan Jacobs, Thales S. Korting, Hugo N. Bendini, and Leandro L. Parente. Spatio-Temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters, 2021.
[5] Seong-Hyeok Lee, Kuk-Jin Han, Kwon Lee, Kwang-Jae Lee, Kwan-Young Oh, and Moung-Jin Lee. Classification of Landscape Affected by Deforestation Using High-Resolution Remote Sensing Data and Deep-Learning Techniques. Remote Sensing, 2020.
[6] Pablo Pozzobon de Bem, Osmar Abilio de Carvalho Junior, Renato Fontes Guimarães, and Roberto Arnaldo Trancoso Gomes. Change Detection of Deforestation in the Brazilian Amazon Using Landsat Data and Convolutional Neural Networks. Remote Sensing, 2020.