SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finetuning. We select reasonable hyper-parameters that we use for all models and datasets unless otherwise stated and sweep across learning rates. This learning rate sweep is essential to creating fair evaluation conditions across models since each model is given the same search budget (in terms of finetuning runs, not compute hours), and different models have different optimal learning rates. Models pretrained with reconstruction approaches tend to require higher base learning rates during finetuning than models pretrained with contrastive learning. For instance, MAE [31] lists a base learning rate of 1e-3, FLIP [68] lists a base learning rate of 5e-5, CoCa [49] lists base learning rates from 1e-5 to 5e-4, depending on the downstream dataset.
No single learning rate would enable a fair comparison across all models and datasets. Therefore, we sweep learning rates across an extensive range  $\{3\mathrm{e - 5},5\mathrm{e - 5},8\mathrm{e - 5},1\mathrm{e - 4},3\mathrm{e - 4},5\mathrm{e - 4},8\mathrm{e - 4},1\mathrm{e - 3}\}$  and report the best single evaluation result obtained for each dataset; this sweep is performed for CROMA models and all other models. We convert these base learning rates to actual learning rates via the widely used linear scaling rule:  $lr =$  base  $lr\times$  batch_size/256. We use the largest batch size that can fit on an A100-40GB GPU (using bfloat16 precision), the AdamW optimizer with  $\beta_{1} = 0.9$ $\beta_{2} = 0.999$  , and a weight decay of 0.01. We warmup for 5 epochs and cooldown for 30 epochs using a cosine decay schedule (other than EuroSAT, which we cooldown for 150 epochs); this follows SatMAE [26]. For classification tasks, we use mixup  $= 0.8$  ,cutmix  $= 1.0$  , switch probability  $= 0.5$  , label smoothing  $= 0.1$  , vertical and horizontal flipping, and 90-degree rotations. We enlarge images to the default image size of the model we are finetuning (i.e., the image size on which the model was pretrained), with one exception. The default image size of SatMAE is  $96\times 96$  ; however, BigEarthNet images are  $120\times 120$  [76], requiring that we either crop BigEarthNet samples (losing information) or adapt SatMAE to larger images. We achieve better performance by adapting SatMAE to  $120\times 120$  images via the widely used position embedding interpolation algorithm than cropping BigEarthNet samples down to  $96\times 96$  . This allowed us to achieve an mAP of 86.18 for SatMAE, a significant improvement over the 82.62 reported in the SatMAE paper. All other datasets use images of  $96\times 96$  or smaller—thus, there is no reason to use this technique for other datasets.
Linear and Nonlinear Probing. We encode each image without data augmentation then train linear and nonlinear probes on the frozen representations. Since each model only encodes each image once, we can sweep through a large range of learning rates ( $\{1,2,3,4,5,6,7,8,9\} \in \{-4, -3, -2\}$ ) very quickly. Unlike finetuning, we do not evaluate probes after every epoch; instead, we evaluate trained probes after all epochs are complete. We use a batch size of 1024, bfloat16 precision, the AdamW optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and a weight decay of 0.01. We warmup for 5 epochs and cooldown for 100 epochs using a cosine decay schedule.
Non-parametric  $k$  NN and  $K$ -means. For  $k$  NN, we use the implementation from [27]. This consists of encoding all training and validation samples and then using the representations of validation samples as queries and training samples as keys to fetch training labels. These fetched training labels are used to classify validation samples. We use  $k = 20$ , other values for  $k$  (i.e., 10, 50) ranked
Table 14: CROMA vs SatMAE training and inference throughput on an A100-40GB GPU.
Model	Backbone	Image Size	Train Images/s	Inference Images/s
SatMAE	ViT-B	96×96	249.3	692.5
CROMA	ViT-B	96×96	1,079.3	2,957.7
CROMA	ViT-B	120×120	555.0	1,532.1
SatMAE	ViT-L	96×96	84.2	263.2
CROMA	ViT-L	96×96	389.1	1,168.2
CROMA	ViT-L	120×120	209.6	640.3
models in the same order as  $k = 20$ . For  $K$ -means, we use the implementation from [79]. This consists of encoding all training and validation samples, then clustering training samples with  $K$ -means ( $K$ -means++ [134] initialization run 10 times). Then, we assign validation samples to clusters and assign clusters to classes via the Hungarian matching algorithm [135].
Sparse Probing. For each model and dataset, we rank the dimensions of representations  $\mathcal{R}$  using the mean difference between classes—[86] shows that the mean difference performs on-par with more complex ranking methods and is simple to implement. Specifically, this is our procedure to sparsely probe the representations of CROMA for BigEarthNet's "beaches, dunes, sands" class: (i) compute the average representation,  $\mathcal{R}_{\mathrm{CROMA}}^{\mathrm{Beaches}} \in \mathbb{R}^{768}$ , of all samples in the BigEarthNet training set that contain the class "beaches, dunes, sands"; (ii) compute the average representation,  $\mathcal{R}_{\mathrm{CROMA}}^{\mathrm{NoBeaches}} \in \mathbb{R}^{768}$ , of all samples in BigEarthNet that do not contain the class "beaches, dunes, sands"; (iii) compute the difference between these averaged representations,  $\mathcal{R}_{\mathrm{diff}} \in \mathbb{R}^{768}$ ; (iv) rank all 768 dimensions in  $\mathcal{R}_{\mathrm{diff}}$  by the absolute value, i.e., the dimension with the greatest absolute difference between classes is ranked first; (v) train a separate linear probe to perform binary classification on the top  $k$  dimensions, we sweep many values of  $k$  between 1 and 768; (vi) using these trained probes, evaluate them on the BigEarthNet validation set. Again, this procedure is performed for all three ViT-B models (CROMA, SatMAE, and I-JEPA), all classification datasets (BigEarthNet, fMoW-Sentinel, Canadian Cropland, and EuroSAT), and all classes—these plots are displayed at the end of this Appendix §A.6. We also sparsely probe radar-only,  $\mathcal{R}_{\mathrm{R}}$ , and radar-optical,  $\mathcal{R}_{\mathrm{RO}}$ , representations for BigEarthNet, since we have access to Sentinel-1 samples.
SatMAE Specifics. SatMAE [26] divides spectral bands into three groups and outputs patch encodings for every group; thus, SatMAE outputs three patch encodings per patch location. To be as fair as possible to SatMAE, we explore four ways of merging these co-located patch encodings to perform segmentation: unnormalized spatial concatenation, normalized spatial concatenation, unnormalized spatial pooling, and normalized spatial pooling. We find unnormalized spatial concatenation (i.e., concatenating the patch encodings of co-located patches before the LayerNorm) performed best. Thus, we use the unnormalized spatially concatenated patch encodings for all segmentation datasets. Conversely, CROMA does not divide spectral bands into groups—resulting in  $3 \times$  shorter sequence lengths. The computation required to process a sequence of tokens with a transformer increases with increasing sequence lengths. This makes CROMA much more computationally efficient than SatMAE for a given ViT backbone and image size (Table 14). We also pretrain a SatMAE-B model for 300 epochs on the SSL4EO dataset. However, this model performs poorly, so we do not report these results; this experiment indicates that SatMAE's hyper-parameters may not transfer well to different pretraining datasets.
A.5 Societal Impact
Since we pretrain our models on the SSL4EO dataset [70], our models may be biased towards the distribution from which SSL4EO data were sampled. Although SSL4EO samples are geographically diverse (please see Fig. 2 from the SSL4EO paper [70]), locations are sampled from areas surrounding human settlements. As a result, large geographic areas that are sparsely populated—for instance, the Amazon rainforest, the Sahara desert, and the Australian outback—are underrepresented. This could negatively impact the quality of representations in these locations and any decisions made on their basis.
Another distribution shift—this time, between finetuning and inference—is our primary concern. For example, finetuning a model on the imagery of one geography and then making predictions on the imagery of another geography creates a distribution shift. As a result, biases from the finetuning geography may be realized in the predictions made by the finetuned model. This is particularly
problematic when these predictions are used in decision-making, for instance, allocating poverty assistance. However, it is well-demonstrated that pretrained models are more robust to distribution shifts than models trained from scratch. Additionally, as we develop better foundation models for Earth Observation, we reduce the need for annotated data; this may allow practitioners to be more selective of the data they wish to leverage during finetuning.
We do not expect our pretrained models to be particularly valuable for military applications, as militaries likely have access to higher resolutions (spatially, spectrally, and temporally) than Sentinel-1 & 2 provide. However, our framework may be leveraged to pretrain models on higher-resolution imagery, which could be useful for military applications, although this is a risk of all novel learning algorithms.
A.5.1 Compute
We approximate the computational resources we use for pretraining and finetuning (frozen representation evaluations are negligible in comparison). For pretraining, estimates are in A100-80GB GPU hours; for finetuning, estimates are in A100-40GB GPU hours. Please see Table 15.
Table 15: Estimated GPU hours used for developing and validating CROMA.
Method	Backbone	Task	GPU Hours
radar↔optical [71]	ResNet50	Classification Finetuning	10
radar↔optical [71]	Swin-T	Classification Finetuning	25
MAE [31, 70]	ViT-S	Classification Finetuning	20
DINO [72, 70]	ViT-S	Classification Finetuning	20
SatMAE [26]	ViT-B	Classification Finetuning	75
CROMA	ViT-B	Classification Finetuning	35
SatMAE [26]	ViT-L	Classification Finetuning	215
CROMA	ViT-L	Classification Finetuning	90
CROMA	ViT-B	Pretraining 300 epochs	80
CROMA	ViT-L	Pretraining 600 epochs	380
CROMA	ViT-B	Pretraining Ablations	1,100
A.6 Visualizations
We visualize representations and patch encodings using UMAP and t-SNE. For both segmentation datasets (DFC2020 [87] and DW-Expert [88]), we visualize patch encodings of 50,000 randomly sampled patches and use the most dominant class in a patch as its label. We also plot sparse probing results for binary classifiers trained on the top  $k$  representation dimensions.
Figure 5: t-SNE plots of EuroSAT [77] representations.
Figure 6: UMAP plots of Canadian Cropland [78] representations.
Figure 7: t-SNE plots of Canadian Cropland [78] representations.
Figure 8: UMAP and t-SNE plots of DW-Expert [88] patch encodings.
Figure 9: UMAP and t-SNE plots on DFC2020 [87] patch encodings.
Figure 10: Sparse Probing all classes in BigEarthNet
Figure 11: Sparse Probing all classes in Canadian Cropland
Figure 12: Sparse Probing all classes in EuroSAT
Figure 13: Sparse Probing all classes in fMoW-Sentinel (part i)
Figure 14: Sparse Probing all classes in fMoW-Sentinel (part ii)
Figure 15: Sparse Probing all classes in fMoW-Sentinel (part iii)
Figure 16: Sparse Probing all classes in fMoW-Sentinel (part iv)