SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anthony Fuller $^{1,*}$ , Koren Millard $^{2}$ , James R. Green $^{1}$
$^{1}$ Department of Systems and Computer Engineering
$^{2}$ Department of Geography and Environmental Studies
Carleton University, Ottawa, Canada
A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples—aligned in space and time—and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to  $17.6 \times$  larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks—finetuning (avg.  $\uparrow 1.8\%$ ), linear (avg.  $\uparrow 2.4\%$ ) and nonlinear (avg.  $\uparrow 1.4\%$ ) probing, kNN classification (avg.  $\uparrow 3.5\%$ ), and  $K$ -means clustering (avg.  $\uparrow 8.4\%$ ); and three segmentation benchmarks (avg.  $\uparrow 6.4\%$ ). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.
Deep learning has led to rapid advances in remote sensing, augmenting our ability to understand and monitor our planet. The remote sensing community has developed many application-specific deep learning models, specifically for satellite imagery: identifying heavily polluting brick kilns [1, 2] or illegal airstrips [3]; monitoring deforestation [4, 5, 6, 7] or crops [8, 9, 10]; detecting floods [11, 12] or wildfires [13, 14, 15]; even estimating household income [16, 17] or poverty [18, 19, 20]. Deep learning-based remote sensing is playing a growing role in tackling our climate crisis [21, 22, 23]. Recently, researchers leveraged self-supervised learning to pretrain remote sensing models that can be employed on these tasks, and more [24, 25, 26, 27]. Self-supervised methods are invaluable for remote sensing, as there are petabytes of publicly available raw data from which to learn general representations, while only limited annotated data exists for downstream applications.
Self-supervised representations are often learned via contrastive approaches [28, 29, 30] or reconstruction approaches [31, 32, 33]. Contrastive approaches encourage the representations of positive pairs of samples—built by producing another view of a sample, for instance, from another sensor [34] or time [35], or by augmentations [30]—to be similar, and the representations of negative pairs to be dissimilar; this process can learn descriptive, object-focused representations. Models trained with a
*All correspondence should be addressed to Anthony Fuller: anthony.fuller@carleton.ca
contrastive objective learn to discard information not shared between views [36, 37], regardless of the information's usefulness on downstream tasks; this makes the representations they learn sensitive to how positive pairs are built [30, 38, 39, 40]. Conversely, models trained with reconstruction or autoencoding objectives—for instance, predicting hidden pixels [31, 41]—learn to capture as much information as possible [42, 43, 44]. Reconstruction approaches do not rely on multiple views and scale incredibly well [31, 45, 46], but they learn representations that require significant finetuning to be useful on downstream tasks [43, 45]. Park et al. [47] show vision transformers (ViTs [48]) trained with contrastive learning focus more on shapes and low-frequency information than ViTs trained with reconstruction approaches, which focus more on textures and high-frequency information. They show that combining both objectives may achieve a sweet spot that learns better representations than either objective alone. Several other frameworks have been developed that leverage both objectives to learn SoTA representations [49, 50, 51, 52]; however, none are designed for spatially aligned multimodal data.
Researchers developing foundation models for remote sensing have yet to take advantage of the multimodal data ubiquitous to remote sensing. For instance, the Sentinel missions—imaging the Earth's landmass multiple times per month since 2015—consist of multispectral optical imagery acquired by Sentinel-2 and Synthetic Aperture Radar (SAR) data acquired by Sentinel-1. By exploiting differences in how electromagnetic radiation interacts with Earth surface materials and measuring the radiation at many wavelengths (ranging from 440 to  $2,200\mathrm{nm}$ ), Sentinel-2 multispectral optical imagery can be used to characterize the material composition of objects [53]. By actively transmitting and receiving longer wavelength ( $5.5\mathrm{cm}$ ) electromagnetic pulses, Sentinel-1 SAR can be used to characterize the geometry, roughness, and electrical properties of objects [54]. These modalities have proven complementary across remote sensing applications [55, 56, 57]. Importantly for our work, they are spatially aligned, allowing multiple views of the same feature on the ground. Moreover, because data from the Sentinel missions are freely available, they have become the most widely used source of satellite imagery in research; thus, models with useful representations of Sentinel-1 & 2 imagery can be immediately leveraged in scientific research, improving our ability to understand our planet.
These observations motivate Contrastive Radar-Optical Masked Autoencoders (CROMA): a framework for learning rich representations of multimodal, spatially aligned, 2D data; which we leverage to pretrain ViT models on Sentinel-1 & 2 data, providing the most valuable foundation models for Earth Observation, to date. We highlight three contributions: ① CROMA significantly outperforms the current SoTA multispectral model, SatMAE [26], under an extensive evaluation. ② CROMA learns representations that are optionally multimodal (i.e., they can be effectively leveraged when either or both modalities are available) and extrapolate to larger images at test-time (i.e., they can be effectively leveraged on images larger than those on which the model was trained). ③ CROMA's effectiveness is driven by two innovations: our complementary pretraining objectives and our novel relative position encoding (RPE) strategies that contribute to the quality of learned representations.
In this section, "optical" refers to 12-channel multispectral optical imagery acquired by Sentinel-2, and "radar" refers to 2-channel SAR backscatter data acquired by Sentinel-1.
Architecture Background. Most work combining contrastive and reconstruction objectives learning joint multimodal representations is in the image-text domain [51, 58, 59, 60, 61, 62, 63, 64, 65]; heavily inspiring our architecture, Contrastive Captioning (CoCa [49]) learns SoTA image-text representations that are optionally multimodal. The CoCa framework consists of two unimodal encoders—one for images, one for text—and a multimodal decoder that receives text encodings at the bottom of the network and cross-attends to image encodings. CoCa is trained with two objectives: an image  $\leftrightarrow$  text contrastive objective between unimodal encoders and an image-captioning objective at the output of the multimodal decoder. Our framework significantly adapts CoCa to aligned multimodal 2D data by masking both modalities, introducing a multimodal encoder, a lightweight decoder (only used during pretraining, inspired by masked autoencoders [31]), and novel cross-attention and self-attention positional biases.
Model Architecture. CROMA consists of three encoders (Fig. 1). A unimodal radar ViT  $f_{\mathrm{R}}$  that encodes radar inputs  $I_{\mathrm{R}} \in \mathbb{R}^{2 \times H \times W}$  into  $L$  patch encodings  $\mathcal{E}_{\mathrm{R}} \in \mathbb{R}^{L \times D}$ , i.e.,  $\mathcal{E}_{\mathrm{R}} = f_{\mathrm{R}}(I_{\mathrm{R}})$ . A unimodal optical ViT  $f_{\mathrm{O}}$  that encodes optical inputs  $I_{\mathrm{O}} \in \mathbb{R}^{12 \times H \times W}$  into  $L$  patch encodings
Figure 1: (Left) Our CROMA framework jointly leverages radar  $\leftrightarrow$  optical contrastive learning and masked autoencoding to learn rich, self-supervised representations. (Right) Leverage representations on: unimodal or multimodal data, larger images at test-time, and diverse tasks and methods.
Figure 2: 2D-ALiBi matrix for an image with 9 patches  $(3^{2}$  before flattening);  $q$  is for query,  $k$  is for key, and  $m$  is a fixed scalar that biases attention heads at different rates.
$\mathcal{E}_0 \in \mathbb{R}^{L \times D}$ , i.e.,  $\mathcal{E}_0 = f_0(I_0)$ . A multimodal radar-optical transformer  $f_{\mathrm{RO}}$  that encodes  $L$  radar-optical patches,  $\mathcal{E}_{\mathrm{RO}} \in \mathbb{R}^{L \times D}$ , i.e.,  $\mathcal{E}_{\mathrm{RO}} = f_{\mathrm{RO}}(\mathcal{E}_{\mathrm{R}}, \mathcal{E}_{\mathrm{O}})$ . For each set of unimodal encodings, we build full-image representations  $\mathcal{R} \in \mathbb{R}^D$  by processing the mean pooled patch encodings through a feedforward network (FFN), i.e.,  $\mathcal{R}_{\mathrm{R}} = \mathrm{FFN}_{\mathrm{R}}(\mathrm{MeanPool}(\mathcal{E}_{\mathrm{R}}))$  and  $\mathcal{R}_{\mathrm{O}} = \mathrm{FFN}_{\mathrm{O}}(\mathrm{MeanPool}(\mathcal{E}_{\mathrm{O}}))$ . Our patch size is  $8 \times 8$  pixels for both modalities, and our default image size is  $120 \times 120$  pixels. Our radar encoder has  $N/2$  transformer layers, and our optical encoder has  $N$  transformer layers ( $N$  is 12 for ViT-B and 24 for ViT-L backbones). All unimodal encoder layers are composed of self-attention and FFN sublayers. Our multimodal  $N/2$ -layer encoder—composed of self-attention, cross-attention, and FFN sublayers—encodes both modalities into a single sequence of  $L$  patch encodings; this encoder receives radar patch encodings at the bottom of the network and learns multimodal representations by cross-attending to optical patch encodings. Multimodal representations can be built via pooling multimodal patch encodings, i.e.,  $\mathcal{R}_{\mathrm{RO}} = \mathrm{MeanPool}(\mathcal{E}_{\mathrm{RO}})$ . Our ViT backbones do not use sinusoidal position embeddings; instead, we bias the self-attention and cross-attention matrices with the distances between patches.
ALiBi Background. ALiBi [66] is a simple and intuitive RPE method for transformers that biases the self-attention matrix based on the distance between tokens in a 1D sequence. Each self-attention head receives positional biases with different strengths, called slopes  $m$ . With 16 attention heads, the geometric sequence defines these scalar slopes starting at  $\frac{1}{\sqrt{2}}$ , i.e.,  $\frac{1}{2^{0.5}}, \frac{1}{2^1}, \frac{1}{2^{1.5}}, \ldots, \frac{1}{2^8}$ . Biases are subtracted from the attention matrix before the softmax is calculated. Specifically, the pre-softmax attention matrix  $A \in \mathbb{R}^{h \times L \times L}$  ( $h$  is the number of heads,  $L$  is the sequence length) is populated with attention scores  $a_{hij}$  for the  $i$ th query  $q_{hi} \in \mathbb{R}^d$  and  $j$ th key  $k_{hj} \in \mathbb{R}^d$  ( $d$  is the head dimension):  $a_{hij} = \sqrt{d} \cdot q_{hi} \cdot k_{hj}$ , without ALiBi; and  $a_{hij} = \sqrt{d} \cdot q_{hi} \cdot k_{hj} - \text{distance}(i,j) \cdot m(h)$ , with ALiBi. Crucially, ALiBi does not add position embeddings at the bottom of the network; instead, the relative positions between tokens are encoded in the attention matrix itself. To date, ALiBi is the only position encoding method for transformers that has been demonstrated to extrapolate at test-time to sequences
far longer than those on which it was trained.
2D-ALiBi and X-ALiBi. We extend ALiBi to 2D inputs by biasing the self-attention matrix based on the Euclidean distance between query-key pairs in a ViT—we call this 2D-ALiBi (Fig. 2). We also extend ALiBi to cross-attention by biasing the cross-attention matrix based on the Euclidean distance between cross-modal query-key pairs—we call this X-ALiBi. (In cross-attention [67], queries are built from the previous layer, whereas keys and values are built from optical encodings.) For both 2D and X-ALiBi, we calculate our attention-head slopes with the same geometric sequence as ALiBi. Since our two modalities
$k_{1}k_{2}k_{3}k_{4}k_{5}k_{6}k_{7}k_{8}k_{9}$
$q_{1}\left|0.01.02.01.01.42.22.02.22.8\right|$
$q_{2}$  1.00.01.01.41.01.42.22.02.2
$q_{3}$  2.01.00.02.21.41.02.82.22.0
$q_{4}$  1.0 1.4 2.2 0.0 1.0 2.0 1.0 1.4 2.2
$q_{5}$  1.41.01.41.00.01.01.41.01.4·m
$q_{6}$  2.21.41.0 2.01.00.0 2.21.41.0
q7 2.0 2.2 2.8 1.0 1.4 2.2 0.0 1.0 2.0
$q_{8}$  2.2 2.0 2.2 1.4 1.0 1.4 1.0 0.0 1.0
$q_{9}$  2.8 2.2 2.0 2.2 1.4 1.0 2.0 1.0 0.0]
are aligned 2D sensor data, our 2D-ALiBi and X-ALiBi matrices are identical. The primary motivation of 2D-ALiBi is to learn representations that can generalize across image sizes; this is particularly useful in remote sensing and is likely useful in other domains. The primary motivation of X-ALiBi is to improve sensor fusion by inserting positional information in the cross-attention sublayer of our multimodal encoder,  $f_{\mathrm{RO}}$ . These position encoding techniques are rotation and translation invariant, which are desirable properties for the overhead imagery in Earth Observation.
MAE Background. A masked autoencoder (MAE [31]) rearranges an image into a sequence of non-overlapping patches, then randomly samples a large portion of patches to be held-out. The "visible" patches, that are not held-out, are encoded by a ViT. This MAE-style masking is ingenious: it leverages sparse computation while only requiring dense operations that run efficiently on modern hardware. MAE introduces a lightweight decoder that receives visible patch encodings and hidden mask embeddings, which are both added to 2D-sinusoidal position embeddings. The decoder outputs predictions of the pixel values of the held-out patches. Both the encoder and decoder are pretrained end-to-end to minimize the mean squared error between patch predictions and the originally held-out patches. The pretrained encoder can then be leveraged on downstream applications.
Reconstruction Objective. We independently mask  $75\%$  of radar and optical patches and encode the unmasked patches with our three encoders, i.e.,  $\mathcal{E}_{\mathrm{R}}^{um} = f_{\mathrm{R}}(\mathbf{I}_{\mathrm{R}}^{um})$ ,  $\mathcal{E}_{\mathrm{O}}^{um} = f_{\mathrm{O}}(\mathbf{I}_{\mathrm{O}}^{um})$  and  $\mathcal{E}_{\mathrm{RO}}^{um} = f_{\mathrm{RO}}(\mathcal{E}_{\mathrm{R}}^{um}, \mathcal{E}_{\mathrm{O}}^{um})$ ; where  $um$  means unmasked. We introduce a lightweight 1-layer transformer decoder  $f_{\mathrm{DEC}}$  that receives multimodal patch encodings and mask embeddings after adding 2D-sinusoidal position embeddings and predicts a target image, i.e.,  $\hat{\mathrm{I}} = f_{\mathrm{DEC}}(\mathrm{CONCAT}[\mathcal{E}_{\mathrm{RO}}^{um}, \mathrm{Emb}^{\mathrm{mask}}] + \mathrm{Emb}^{\mathrm{positions}})$ . We split the channels of  $\hat{\mathrm{I}}$  to form predictions for each sensor,  $\hat{\mathrm{I}}_{\mathrm{O}}$  and  $\hat{\mathrm{I}}_{\mathrm{R}}$ ; the loss is only applied at the locations of the masked-out patches:
$$
\mathcal {L} _ {\mathrm {M A E}} = \frac {1}{N} \sum_ {i} ^ {N} \left(\underbrace {\frac {\sum_ {j} ^ {M} \left(\hat {\mathbf {I}} _ {\mathrm {O}} ^ {i j} - \operatorname {N o r m} (\mathbf {I} ^ {i j}) _ {\mathrm {O}}\right) ^ {2}}{M}} _ {\text {o p t i c a l r e c o n s t r u c t i o n}} + \underbrace {\frac {\sum_ {j} ^ {M} \left(\hat {\mathbf {I}} _ {\mathrm {R}} ^ {i j} - \operatorname {N o r m} (\mathbf {I} ^ {i j}) _ {\mathrm {R}}\right) ^ {2}}{M}} _ {\text {r a d a r r e c o n s t r u c t i o n}}\right)
$$
where  $N$  is the batch size,  $M$  is the number of masked patches, and Norm sets the mean to 0 and standard deviation to 1 for target patches (following MAE). Along with learning unimodal representations, this objective spatially fuses our two sensors, i.e., it builds multimodal patch encodings that represent information from both sensors in the patch, corresponding to an  $80\mathrm{m}\times 80\mathrm{m}$  square on the ground ( $8\times 8$  patches at  $10\mathrm{m}$  resolution). Finally, 2D and X-ALiBi can be easily adapted to MAE-style masking by removing the masked-out columns and rows from the bias matrix.
Contrastive Learning Background. Contrastive learning aims to classify the correct pairings of samples derived from a batch. Logits are formed by measuring the similarity between the projected representations of samples. As a result, the representations of positive pairs are pulled together, and the representations of negative pairs are pushed apart. Very recently, FLIP [68] performs contrastive learning with masked-out representations via MAE-style masking. This speeds up pretraining and enables larger batches due to the reduced memory per sample. FLIP performs on par with CLIP [69]—the foundational work that learns rich representations via image  $\leftrightarrow$  text contrastive learning—but can be pretrained at half the cost.
Contrastive Objective. We perform radar  $\leftrightarrow$  optical contrastive learning across the unimodal representations of our masked-out sensor data using the InfoNCE loss [28]. For an optical anchor image, the positive sample is the geographically and temporally matched radar sample, and the negative samples are all other radar samples from the batch; likewise, our radar representations are pulled towards (positives) or pushed apart (negatives) from optical representations:
$$
\mathcal {L} _ {\mathrm {C o n}} = - \frac {1}{2 N} \underbrace {\sum_ {i} ^ {N} \log \frac {\exp \left(z _ {\mathrm {R}} ^ {i \top} z _ {\mathrm {O}} ^ {i} / \sigma\right)}{\sum_ {j} ^ {N} \exp \left(z _ {\mathrm {R}} ^ {i \top} z _ {\mathrm {O}} ^ {j} / \sigma\right)}} _ {\text {r a d a r - t o - o p t i c a l}} + \underbrace {\sum_ {i} ^ {N} \log \frac {\exp \left(z _ {\mathrm {O}} ^ {i \top} z _ {\mathrm {R}} ^ {i} / \sigma\right)}{\sum_ {j} ^ {N} \exp \left(z _ {\mathrm {O}} ^ {i \top} z _ {\mathrm {R}} ^ {j} / \sigma\right)}} _ {\text {o p t i c a l - t o - r a d a r}}
$$
where  $z_{\mathrm{R}}$  and  $z_{\mathrm{O}}$  are  $\ell_2$  normalized linear projections of radar and optical representations, respectively, i.e.,  $z_{\mathrm{R}} = \mathrm{Norm}(\mathrm{Linear}_{\mathrm{R}}(\mathcal{R}_{\mathrm{R}}))$  and  $z_{\mathrm{O}} = \mathrm{Norm}(\mathrm{Linear}_{\mathrm{O}}(\mathcal{R}_{\mathrm{O}}))$ .  $\sigma$  is the softmax temperature, and
$N$  is the batch size. Crucially, we only encode a small portion of input patches, which form our representations. This masking provides advantages: it enables larger batches, speeds up pretraining, and enables our multimodal reconstruction objective with the same computational graph. This radar  $\leftrightarrow$  optical contrastive objective encourages representations to be sensor-invariant, i.e., to capture information shared between sensors.
Combined Pretraining Objective. We combine contrastive learning and masked sensor modeling pretraining objectives:  $\mathcal{L} = \lambda_{\mathrm{Con}}\mathcal{L}_{\mathrm{Con}} + \lambda_{\mathrm{MAE}}\mathcal{L}_{\mathrm{MAE}}$ . We set both task weights (i.e.,  $\lambda_{\mathrm{Con}}$  and  $\lambda_{\mathrm{MAE}}$ ) to 1 and ablate them in Appendix §A.1.
Pretraining. We pretrain CROMA models on the SSL4EO dataset [70]—a large geographically and seasonally diverse unlabeled dataset. SSL4EO consists of 1 million paired Sentinel-1 GRD & Sentinel-2 L2A samples of  $264 \times 264$  pixels. Sentinel-1 channels consist of VV and VH backscatter. Sentinel-2 channels consist of 12 surface reflectance multispectral bands (the cirrus band is removed). We pretrain CROMA-B (ViT-B backbone) for 300 epochs and CROMA-L (ViT-L backbone) for 600 epochs. Crucially, a single pretraining run trains all three encoders (optical, radar, and joint radar-optical) end-to-end; users can then finetune one or multiple encoders, depending on their task and data availability. We perform all pretraining experiments on an NVIDIA DGX server ( $8 \times$  A100–80 GB), including ablations. Please see Appendix §A.3 for more details.
Comparisons. We compare CROMA to all available multispectral optical foundation models, which include two models pretrained by [71] using radar  $\leftrightarrow$  optical contrastive learning; two models pretrained by [70] using the MAE [31] and DINO [72] frameworks; and two models pretrained by [26] using their multispectral representation learning framework, SatMAE. We also compare CROMA to a SoTA method for learning visual representations of natural images—image joint embedding predictive architecture (I-JEPA, [73])—that we leverage to pretrain a ViT-B model on SSL4EO's optical imagery for 300 epochs. To enable a fair comparison between models, we evaluate all models under identical conditions and hyper-parameter budgets (please see Appendix §A.4.2 for details). This is necessary because the originally reported results of these models occurred under inconsistent evaluation conditions—for instance, data splits or training data amounts. We use the latest publicly available models for all evaluations and preprocess data according to official repositories. For radar and radar-optical datasets, we compare CROMA to SatViT-V2 [74], a model pretrained using MAE [31] on stacked Sentinel-1 & 2 imagery; and DeCUR, a model pretrained—concurrently with this work—by [75] using their multimodal representation learning framework.
3.1 Multispectral Optical Experiments
Classification Setup. We evaluate CROMA by finetuning, frozen linear and nonlinear probing, kNN classifying, and  $K$ -means clustering pretrained representations across four Sentinel-2 classification benchmarks. 1 The multi-label BigEarthNet dataset [76] (35,420 train samples and 118,065 validation samples); this is  $10\%$  of the complete BigEarthNet training set that is now used by default [25, 26] to reduce the costs of finetuning and is better suited for a remote sensing benchmark [22]. 2 The fMoW-Sentinel dataset [26] (71,287 train samples and 84,939 validation samples); this is also  $10\%$  of the complete training set. Following BigEarthNet's use, we believe this smaller training set is a more appropriate benchmark for model evaluation, but we show results on the complete training set in Appendix §A.4.1. 3 The EuroSAT dataset [77] (16,200 train samples and 5,400 validation samples). 4 The Canadian Cropland dataset [78] (53,884 train samples and 23,088 validation samples); this is a new benchmark inspired by EuroSAT but is more challenging, as the crop types (barley, canola, corn, etc.) can be visually similar. For finetuning and linear probing, we add a linear layer for these tasks atop the full-image representations, i.e.,  $\mathrm{Linear}(\mathcal{R}_0)$ . For nonlinear probing, we use an MLP with one hidden 2048-d layer, i.e., Linear(ReLU(Linear(\mathcal{R}_0))). Additionally, we perform non-parametric  $k\mathrm{NN}$  classification ( $k = 20$ ) and  $K$ -means clustering for single-label benchmarks to evaluate frozen representations. [79] shows that no single method of evaluating representations is the best; they recommend including  $k\mathrm{NN}$  and  $K$ -means alongside linear probing. Other studies [80, 72, 81] show a rank mismatch between  $k\mathrm{NN}$  and linear probing evaluations—indicating they offer complementary estimates of representation quality. Please see Appendix §A.4.1 and A.4.2 for implementation, data splits, and hyper-parameter details.