SlowGuess/NeurIPS_2023_OCR huggingface dataset:
KNN Performance. Similar to the original MAE configuration, the encoder of Cross-Scale MAE uses a ViT-Large model, and the decoder is a light ViT model with the 8-layer attention blocks. The datasets we use include RESISC45 [10], WHU-RS19 [47], UC Merced [47], EuroSAT [22], as shown in Table 1. To evaluate the representation learning capacity of the proposed method with
Table 2: Average KNN accuracy with different scale ratios (12.5%, 25%, 50%, 100%)
RESISC45	WHU-RS19	UC Merced	EuroSAT
SatMAE	66.3	69.9	69.7	81.9
Scale-MAE	70.0	79.5	75.0	86.7
Cross-Scale MAE	75.6	79.8	74.5	87.8
different GSD images, we evaluate the network performance using images with different scale ratios,  $\{12.5\%, 25\%, 50\%, 100\}$ , to the raw image to simulate different GSD.
The results are presented in Fig. 2. This figure shows the performance of Cross-Scale MAE compared to both SatMAE and Scale-MAE for different scale ratios. From Fig. 2, we observe that, in general, the Cross-Scale MAE performs better than SatMAE or Scale-MAE in all different scale ratios on all datasets. And, for scale ratio in  $\{25\%, 50\%, 100\}$ , the Cross-Scale MAE can obtain more stable performance compared with SatMAE or Scale-MAE. When the scale ratio equals  $12.5\%$ , all three models perform relatively worse. This is because the scale ratio set during pre-training is from 0.2 to 0.8, and 0.125 is out of this range. Nonetheless, Cross-Scale MAE still presents overwhelmingly better performance than the other two. Table 2 shows the average accuracy at all scale ratios and compares Cross-Scale MAE with SatMAE and Scale-MAE. We can find that Cross-Scale MAE presents overwhelmingly better performance than SatMAE in all datasets and we observe similar trend as in Fig. 2. Cross-Scale MAE generally performs better than Scale-MAE except in the UC Merced dataset. It may be because RESISC45 covers an extensive range of GSD, from  $0.2m$  to  $30m$ , but the other three datasets have fixed GSD. This comparison effectively demonstrates the robustness of representations generated from Cross-Scale MAE, especially on multi-scale datasets.
Figure 2: The KNN classification of Cross-Scale MAE for different datasets.
Downstream Tasks. We further evaluate the performance of Cross-Scale MAE on different downstream tasks to assess the effectiveness of the learned representations in various practical scenarios, including classification and segmentation. We conduct the classification task on the fMoW-RGB dataset. The segmentation downstream tasks are performed on two datasets, Potsdam and Vaihungen. We fine-tune the model for 50 epochs for all downstream tasks, following the same hyper-parameter settings as Scale-MAE [36].
To test the capability of handling multi-scale inputs, we compare the performance of different methods on low-resolution images with different scaling ratios applied on full-resolution images. In Table 3, we report the classification performance regarding Top-1 and Top-5 accuracy. In Table 4, we report the segmentation performance regarding mIoU. The proposed Cross-Scale MAE performs superior to state-of-the-art methods in both classification and semantic segmentation downstream tasks.
The ablation study is four-fold. First, we investigate the critical role of each of the three loss functions to multi-scale inputs. Second, we investigate the effect of negative samples in different levels of the representation. This includes two parts, with the first evaluating the whole Cross-Scale MAE framework and the second under the pure contrastive learning framework (leaving out the masking patch reconstruction part). Third, we compare the effect of multi-scale input and the GSD positional encoding used in Scale-MAE [36]. Although we have shown performance improvement with scale augmentation, we extend the investigation to see whether it has better generalization capacity than
Table 3: Linear classification fine-tuning performance on fMoW-RGB
Scale=50%	Scale=100%
Backbone	Top-1	Top-5	Top-1	Top-5
GASSL	ResNet50	0.494	0.540	0.683	0.715
SeCo	ResNet50	0.508	0.602	0.614	0.796
SatMAE	ViT-Base	0.551	0.776	0.651	0.892
SatMAE	ViT-Large	0.591	0.793	0.678	0.923
Scale-MAE	ViT-Base	0.623	0.850	0.673	0.930
Cross-Scale MAE	ViT-Base	0.632	0.914	0.692	0.954
Table 4: Semantic segmentation performance on Potsdam & Vaihingen (mIoU)
Potsdam	Vaihingen
Backbone	25%	50%	100%	25%	50%	100%
Supervised Baseline	ResNet50	0.4518	0.5212	0.7411	0.4603	0.5237	0.7438
SimCLR	ResNet50	0.5327	0.5472	0.7233	0.5286	0.5904	0.7075
BYOL	ResNet50	0.5509	0.5715	0.7264	0.5618	0.5903	0.7298
SimSiam	ResNet50	0.5439	0.5903	0.7188	0.5522	0.6005	0.7205
SeCo	ResNet50	0.5571	0.5935	0.7298	0.5741	0.6213	0.7250
mCL-LC	ResNet50	0.5589	0.5973	0.7316	0.5795	0.6277	0.7262
SatMAE	ViT-Base	0.6325	0.6879	0.7355	0.6239	0.6980	0.7364
Scale-MAE	ViT-Base	0.6853	0.7333	0.7507	0.6910	0.7280	0.7512
Cross-Scale MAE	ViT-Base	0.7082	0.7225	0.7617	0.7163	0.7354	0.7603
the GSD positional encoding. Finally, we explore the effects of some hyper-parameters, including, for example, the number of epochs and the type of backbone used. All experiments in this section are conducted on the RESISC45 dataset [10] and use the KNN classification as a metric.
The weight update of the encoder and decoder networks is mainly controlled by the loss function in Eq. 6 that consists of three modules, the cross consistency loss in the encoder,  $\mathcal{L}_{cc}$ , the cross prediction loss,  $\mathcal{L}_{cp}$ , and the reconstruction loss,  $\mathcal{L}_{re}$ . Table 5 thoroughly compares the classification accuracy using different loss combinations, from which we make some interesting observations. This experiment is conducted on RESISC45 at two scaling ratios, 0.5, 1, on the raw image with ViT-Base and pre-trained 300 epochs on fMoW-RGB dataset with input size of  $128 \times 128$ .
The first row is essentially SatMAE, with single scale input, and thus serves as the baseline for the subsequent comparisons. The second row adds the multi-scale input and solely uses reconstruction loss. The improved accuracy displays the effectiveness of multi-scale inputs, especially for lower-scale samples. From the second to the last row, all experiments use multi-scale inputs with different loss components. From the 3rd to the 4th row, where only one additional loss module is applied, we observe that both the consistency in the encoder representation and decoder representation are adequate, with the consistency in the decoder representation playing a more critical role. Finally, in the last row, we observe that the combination of consistency in the encoder and decoder has a performance increment of  $3\% - 4\%$  on both scaling rates, again demonstrating the effectiveness of the proposed Cross-Scale MAE.
Second, we look into the effectiveness of negative samples in formulating the contrastive loss. Because Cross-Scale MAE leverages contrastive learning at two levels to handle multi-level representations, we must investigate the necessity of involving negative samples at each stage. Specifically, two kinds of losses are evaluated: InfoNCE relating to positive and negative samples and prediction distance relating to only positive samples. First, we assess it under the complete Cross-Scale MAE framework with two scaling ratios, and the results are reported in Table 6. From Table 6, we can find the negative samples in the encoder can improve the classification performance but decrease the performance when involved in the decoder. It may be because the representation from the encoder contains more
Table 5: Ablation study of the effect of each of the three losses in Eq. 6 (%)
Multi-Scale	Cross-Consis	Cross-Pred	KNN 50%	KNN 100%
○	○	○	52.1	58.9
✓	○	○	68.3	69.2
✓	✓	○	72.4	74.4
✓	○	✓	74.9	76.5
✓	✓	✓	78.7	79.3
Table 6: The effect of negative samples (NS) in Cross-Scale MAE (%)
NS in Encoder	NS in Decoder	KNN 50%	KNN 100%
○	○	75.9	77.9
✓	✓	76.8	77.7
○	✓	75.1	77.1
✓	○	78.7	79.3
"structural" information, but the decoder contains more "semantic" information [36]. Hence, the involvement of negative samples in the encoder can help boost the discriminative capacity between multi-scale inputs. On the other hand, with prediction loss between positive samples at the decoder, the consistency in "semantic" information can be better preserved. Having observed this interesting phenomenon, we are intrigued to study if the same pattern would be found in a pure contrastive learning (CL) framework, i.e., Cross-Scale MAE without the reconstruction loss. And we did, the results are shown in Table 7.
Third, we investigate the differences between scale augments and the GSD positional encoding proposed in Scale-MAE [36]. The results are presented in Fig. 3. Two sets of comparisons have been conducted – the left figure shows the performance of Cross-Scale MAE with or without the GSD positional encoding, and the right one shows the SatMAE performance with GSD position or multi-scale augments. The scaling ratios used in the comparison are  $\{25\%, 50\%, 75\%, 100\}$ . From Fig. 3(a), we find that Cross-Scale MAE adding GSD positional encoding decreases the performance. The reason may be that Cross-Scale MAE has already used multi-scale augmentation, rendering the GSD positional encoding redundant. On the other hand, from Fig. 3(b), we can find both GSD positional encoding and multi-scale augmentation can improve SatMAE's performance since SatMAE does not exploit cross-scale consistency in the images.
Finally, we show the performance of Cross-Scale MAE with different backbones at different training epochs and the effect of different masking ratios. The results are posted in supplementary materials.
4.3 Efficient Backbone
Large-scale models are slow to train and have high memory consumption. Thus, they are usually trained on large clusters of GPUs. We aim to make the model more accessible and optimize the implementation in various ways to minimize the training time and memory footprint for training and inference, such that the model can be feasibly trained end-to-end and used for inference on a single GPU while maintaining competitive results. In this work, we showed using the xFormers library to build an efficient foundation for our models, allowing us to train and conduct experiments more quickly.
Table 7: The effect of negative samples in the pure CL framework (%)
NS in Encoder	NS in Decoder	KNN 50%	KNN 100%
○	○	57.2	58.7
✓	✓	57.3	59.5
○	✓	57.1	58.2
✓	○	58.3	60.7
(b)
Figure 3: The comparison of multi-scale augmentation and GSD positional encoding.
Most of the optimizations offered by the xFormers library were made with Tensor-architecture GPUs in mind. We show that we can still gain noticeable improvements in training time even when using an older and less powerful GPU. Thus, for this comparison, we stress-test the xFormers and Timm versions of the baseline implementations by training them on individual Nvidia RTX A6000s. The details of the implementation and ablation studies on attention and loss types have been placed in the supplementary.
5 Conclusion & Future Work
This paper introduced Cross-Scale MAE, a novel multi-scale representation learning framework specifically designed for remote sensing image understanding. By incorporating scale augmentation and enforcing cross-scale consistency at both structural and semantic levels, our approach enhances the representational capabilities of the Masked Autoencoder in the context of multi-scale datasets. Remote sensing image analysis, which often relies on diverse multi-source data, greatly benefits from this framework. To achieve more robust and meaningful representations, we utilize both contrastive and generative losses, which encourage consistency and representation quality across different image scales. Furthermore, we employed the xFormers library to expedite network pre-training without sacrificing performance. Through extensive evaluation of various benchmarks, Cross-Scale MAE demonstrated superior performance compared to other state-of-the-art frameworks for satellite imagery.
Nonetheless, Cross-Scale MAE still has its limitations, which provide directions for our future work. (1) In scale augmentation, we currently focus on spatial augmentation (scaling) while maintaining consistent channel content across two augmentations. However, the complexity of remote sensing scenes extends beyond just scale differences. Variations in source images include both scale and channel disparities. We currently retain shared channels, like RGB, to address this challenge while discarding differing channels. This approach, although maintaining consistency, could lead to the loss of valuable information from dropped channels. For future work, we aim to investigate more into the multi-spectral perspective of representation learning. (2) Our multi-level contrastive learning employs diverse strategies across levels—leveraging both positive and negative samples at the encoder level and exclusively positive samples at the decoder level. This strategy currently yields optimal performance, although the underlying mechanism remains unexplored. In future research, we intend to investigate more into this strategy to gain deeper insights.
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2021-20111000006. The views and conclusions contained herein are those of the authors and should not be interpreted
as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.