Cross-Scale MAE: A Tale of Multiscale Exploitation in Remote Sensing
Authors: Maofeng Tang
Andrei Liviu Cozma
Konstantinos Georgiou
Hairong Qi
Venue: NeurIPS 2023 poster (2023)
Keywords: Remote Sensting
Self-Supervised Learning

Abstract: Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning prob- lem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned represen- tations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing
Maofeng Tang', Andrei Cozma', Konstantinos Georgiou', / Hairong Qi' Min H. Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville
(mtang4, acozma, Xveorgio)@vols uuk. edu, hqi@utk.
Abstract
Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning prob- lem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on single GPU while maintaining the quality of learned represen- tations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of of the-art remote sensing MAE methods.
Introduction
Remote sensing image understanding has been reaping the benefits recent advances in computer vision while at the same time posing unique challenges. First, remote sensing imagery usually covers the Earth's 's surface extensively for Earth observation purposes [52] 43 31]. Compared to the vast area the images cover, even large amount of training samples would become sparse. Furthermore, generating set of representative training samples is challenging, especially in less-explored regions. Hence, self-supervised learning (SSL) becomes viable solution since training data is required for representation learning purposes. A recently published paper 46 provides a comprehensive review of SSL remote sensing.
The second challenge arises from the inherent hardware limitations of remote sensing devices, where one can only expect to acquire images of high resolution in the spatial, spectral, or temporal domain, but not all. These devices usually sacrifice spatial resolution to gain spectral and/or temporal resolution for different functional and/or material analyses of the Earth's 's surface. Ground Sample Distance (GSD) measures the spatial resolution, denoting the physical distance between two adjacent pixels. For example, Landsat-8 (45) has spectral bands at 30 GSD; Sentinel-2 [17] has 4 bands 10 m. 6 bands at 20 m. and bands at 60 m GSD; and WorldView-3 [29] has multi-spectral bands 1.24 GSD. These multi-scale images, although imaging the same site, might not be aligned at all, and the alignment process can be both time-consuming and expensive.
It is worth noting that in remote sensing imagery, due to the rich and unique data provided along the temporal and spectral domains, algorithms tend to exploit more on the spectral and temporal organization but neglect the spatial characteristics, pointed out in [4]. So, extracting effective representations from these misaligned multi-scale images poses a significant challenge. In this
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

paper, we delve into multi-scale analysis and develop novel ways to fully explore the wealth of information offered by images at multiple scales. We present Cross-Scale MAE, a sulf-supervised model based on the Masked Autoencoder (MAE) [2]] that explicitly learns relationships between data at different scales throughout the pre-training process. By leveraging this information, Cross-Scale MAE produces a robust pre-trained model with superior performance across various GSDs and tasks.
Like MAE, Cross-Scale MAE also masks most of the transformed image and then tasks a Vision Transformer (ViT)-based Autoencoder with embeddings of the unmasked components. A decoding light ViT then decodes the entire image from these learned embeddings, where the decoder is later discarded, and the encoder is used to produce representations for an unmasked input image. The main differences can be addressed from two aspects. First, Cross-Scale MAE introduces the "scale augmentation" to synthesize inputs of different scales of one input image to engage the transformer to learn features with varying scales in the encoder. The scale augmentation eliminates the need for aligned multi-scale images of the same ground site as input. The pre-trained model is generic enough to extract arbitrary scale input image representations. The second key difference stems from the term Ccros-SScal"" where features extracted from the multi-scale images are not simply concatenated or combined in any ad hoc fashion as many existing fusion frameworks do; instead, Cross-Scale MAE exploits information consistency across the multi-scale images and uses it constraints to train more robust Autoencoder model. The ccoss-scale information consistency is examined at both the encoder and decoder end to ensure consistency is reinforced at both the structural level (i.e., the encoder end) and the semantic level (i.e., the decoder end).
The experimental results overwhelmingly show that Cross-Scale MAE yields more robust multi-scale representations and better performance for downstream tasks as compared to other state-of-the art methods, across couple of remote sensing datasets with a variety of scale and resolution characteristics. Our contributions can be summarized as follows:
(1) We develop Cross Scale MAE, flexible SSL framework that yields robust representations by enforcing cross-scale information consistency at both structural and semantic levels without the need for aligned multi-scale remote sensing imagery.
(2) We investigate the combination of contrastive learning and masked imagery modeling, specifically, the effect of negative samples on representation at different levels.
(3) We deploy xFormers to realize Cross-Scale MAE, where both the pre-training time and memory usage are improved without performance degradation, making large language model training affordable on single GPU.
The remainder of this paper organized as follows. Sec. 2 reviews recent developments in SSL and multi-scale representation learning. Sec. elabboraas on the proposed Cross-Scale MAE model design. Sec. 4 presents details about experiments and results. Sec. Scc.clucsss the paper and provides general directions for further improvements.
2 Related Work
We mainly discuss recent works related to SSL and multi-scale representation learning.
Self-Supervised Learning (SSL). Self-supervised learning is a branch of unsupervised learning aiming to learn effective representations for downstream tasks without using training labels. SSL can be divided into two groups, namely, discriminative models and generative models. The discriminative approach, e.g., contrastive learning (CL), requires the representation to be distinctive in information for different inputs. The gist of CL. is to make the representations of positive samples close and those of negative samples far from each other, where positive samples are obtained by applying various augmentation schemes on the same image. CL has emerged as a dominant visual SSL method, especially after the advent of MoCo 20 and SimCLR 77. Negative-free, i.e., non-contrastive, joint- embedding methods have been developed [5] 50 9. demonstrating comparable performance as CL methods. In the remote sensing community, the Ca-based representation learning methods have been developed, such as SeCo (28). mCL-LC 42), where the SeCo, short for Seasonal Contrast, offers a novel approach to the challenge of insufficient labeled data in remote sensing by capitalizing on self-supervised learning mechanisms. The mCL-LC method jointly learns global-local representations to improve the segmentation performance for satellite images. On the other hand, the generative approach, e.g., MAE, requires the representation to preserve much of the input information as

---

possible. MAE is an autoencoder with masked prediction, i.e., predicting a property of masked input from unmasked input. Benefiting from the success of masked language models [34] B5 [ the similar mechanism was investigated in vision tasks and has dramatically advanced the field of SSL Following the trend in computer vision, the masked image model has also shown strength in
[6 21.
the remote sensing field. For example, the GFM model [0) excels in representation learning through continual learning, aiming to enhance the applicability of large language models to satellite imagery via knowledge distillation.
In Cross-Scale MAE, both the discriminative and generative mechanisms are used to guarantee the representation from images of different scales to be consistent and meaningful.
Multi-Scale Representation Learning. As mentioned in Sec. I the multi-scall phenomenon is common in vision tasks. To leverage the information in multi-scale sources, the vision community has proposed to extract multi-scale features using traditional approaches, including, for example, spatial pyramids 39 2 26), dense sampling of windows (23 48 49), and the combination of them [I9]. In the last decade, deep convolution neural networks (CNNs) have emerged the de facto architecture for a wide range of vision tasks. Because of the pooling and multi-onnvoltionn kernel operation, the CNN constructs the feature pyramids inherently so that it has been used to build deep multi- scale representations 244. CNN-based multi-scale representation learning is commonly approached in two ways. The first approach involves utilizing external or preset factors, which can include multi-scale kernel architecture [4] [2 37 53 and multi-scale inputs architecture [14] 18 27].
The second approach involves designing internal layers of the network, such as skip and dense connections (40 33 5. Over the past three years, there has been notable surge of interest in applying transformer-based architectures to computer vision tasks. Among these architectures, the Vision Transformer (VIT) [16] stands out particularly successful example, where, compared to CNN, ViT balances the global and local features much better. Recently, other efforts have been made for the multi-scale feature learning, including the multí-scale Deformable Attention and Multi- level Features Aggregation (MSDAM and MLFAM) network [15) and the Shunted Self-Attention (SSA) network (38). These methods have shown strength in multi-level feature extraction for the natural image containing multiple objects of different sizes. However, in remote sensing images, the multi-scale phenomenon is even more prevalent and dynamic. As a result, there has been growing recognition of the need for customized approaches to address this. The recently published Scale-MAE 36 is one such example.
Essentially, compared with the most recent two state-of-thh-arr MAE-based satellite imagery repre- sentation learning methods, i.e., SatMAE [12], and Scale-MAE [36.. SatMAE is the first paper that applies MAE to extract representations from satellite images with single and fixed scale. Scale-MAE and the proposed SSale MAE are based on MAE but focus on multi-scale characteristics. Specifically, Scale-MAE develops the GSD position encoding and applies multi-de-convolutinn after the decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD, and the de-convolution can only result in specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, Cross-Scale MAE can learn informative and consistent representation across different scales without the need of known GSD. Additionally, we leverage xFormers on a single GPU for pre-training efficiency.
3 Methodology
This section elaborates on the Cross-Scale MAE pre-training framework as illustrated in Fig. D Cross- Scale MAE is a self-supervised pre-training framework built upon the MAE (21). Its objective is to learn consistent representations from multi-scale images captured the same site by harnessing the strengths of both discriminative and generative learning approaches. Cross-Scale MAE incorporates two novel loss functions at the encoder and decoder stages to achieve this objective. Firstly, the representations obtained from images of different scales are enforced to be consistent. This ensures that the learned representations capture relevant information across scales. Secondly, decoder is employed to reconstruct the corresponding input, promoting the learned representation to be more representative. Additionally, the decoder utilizes cross cross-prediction. where the embeddings generated by the decoder from an image with a lower GSD are used to predict the embeddings from an image
3

---

Decedler
tmage
Contusiive
Fine
Figure I: The architecture of Cross-Scale MAE comprises encoder and decoder (Left). The encoder (iop-Right) employs a vision transformer (ViT) backbone, specifically a ViT-base with 12 self-attention blocks. The decoder (Bottom (Bothom-Right) uses lightweight ViT backbone with 8 self- attention blocks. A single satellite image undergoes scale augmentation through random cropping and resizing. The contrastive loss is computed using encoder outputs for the scale inputs. The cross- prediction loss (in MSE) is applied to the last self-attention block's s decoder output. Reconstruction loss compares the predicted masked patch and the actual input.
with higher GSD. This cross-prediction mechanism helps capturing and leveraging the multi-scale information present in the images.
3.1
Basic Setting
yet'p E Hx represent an input image of height H and width W with C channels. First, multi- scale augmentation (Sec. 3.2) is applied to p generate two images, Ph of relatively higher GSD and Pt of relatively lower GSD. For each the augmented images, a sequence of |S]] independent patches of height and width of pixels are created, where each of the patches, E S, has dimension E RR fraction, m, of the patches removed, while the remaining patches are passed through a projection function (c.g., linear layer) to project the patches s into D dimensions, fema I2"AC + RD, to obtain embedded patches Semd (S) A standard positional encoding vector is added the embedded patches and fed into the encoder. After the encoder, the removed m patches are placed back into their original location in the sequence of patches where learned mask token represents the masked patches that were not encoded, Another positional encoding vector is added to all patches, and sequence of transformer blocks decodes these patches to form the original input image; these steps are illustrated in Fig.
3.2 Multi-Scale Augmentation
A specialized augmentation method is applied to the input image before patchifying it in the Cross- Scale MAE framework. This augmentation process generates an additional image of lower GSD for the same site, and the two images are then used as inputs through two independent instances of the MAE network. More specifically, given an input image p with a certain GSD, the framework randomly selects two scale ratios, Th and rı, from the range [O, 1]. where Ti Th- The image is then down-sampled using these scale ratios to generate two images, Ps and pt. In our experiments, we set T4 1, so Pk- p. Both images, P/, and pl, are subsequently resized to fixed size of 128px². and the GSD values of pı. and pl are denoted as Ih and gl, respectively.
Note that this augmentation approach differs from traditional augmentation, where the rescaled images are traditionally only used as separate samples. Instead, the multi-scale inputs are generated dynamically at runtime and used as inputs through multiple network branches, where the branches have shared weights. The constraints enforced upon the learned representations of these network branches are detailed in the following sections.

---

3.3 Cross-Scale Information Consistency in the Encoder
Typically, the standard MAE trains network to reconstruct an image by masking out a significant portion of its pixels. However, the standard MAE encoder does not explicitly address the issue of representation consistency for multi-scale input sources. In contrast, the proposed Cross-Scale MAE aims learn meaningful and consistent scale-invariant representations. To achieve this, Cross- Scale MAE focuses on two fundamental properties in the learned representations: (1) consistency of information across different-scale images and (2) representativeness of the image. In the Cross-Scale MAE framework, contrastive learning strategy is employed within the encoder to ensure consistency among representations derived from different-scale images.
The intuitive idea behind Cross-Scas Pross-Scaae MAE is that different-scale images captured at the same ground site contain highly relevant information despite potential visual differences. Cross-Scaa MAE leverages this notion by maximizing the shared information between representations derived from different-scale images of the same site while minimizing the shared information between images from different sites. This approach aligns with the principles contrastive learning. In contrastive learning, the InfoNCE loss is commonly used estimate the lower bound of mutual information between data samples. By minimizing the InfoNCE loss, we effectively maximize the mutual information between representations from different scales [4 32].
Specifically, the encoder E, uses the ViT-base [66] as the backbone. Upon feeding an input image p to encoder E, high-level representation, (p)), can be obtained. Before computing the contrastive loss, a nonlinear projection head g-(+) is needed, which has been proven to effective [8]. So, after the encoder and projection head, obtain the representation, gj (E (p)). The cross-scale consistency loss is thus defined as:
N Lec
(Cinoo (pi ph) + linfo (phopf)
2N
k=
where pf and of are the different-scale images from the same input image pk. N denotes the number of samples in a mini-batch, and the InfoNCE contrastive loss function into is the same as in [6) which is defined follows:
exp (sim (z² =/h) /T)
Canfo (pi, log
sim( # gy (E(p)))
Epen exp
where 2/ Of (E (pt)), 2 = =9s ( (rh)) and the sample batch. The sim operator is the cosine distance.
3.4 Cross-Scale Prediction in the Decoder
The InfoNCE loss in the encoder provides consistency for multi-scale input but cannot guarantee that the learned representation adequately represents the input. In other words, the representation learned by the network from multi-scale images should be not only consistent among different scales, but also representative of the semantic information. According to K the self-supervised ViT automatically learns class-specific features leading to unsupervised object segmentation; specifically, block's last layer includes abundant semantic information. Motivated by [ the decoder of the Cross-Scale MAE has two tasks: (1) reconstruction of the multi-scale input as close as possible, (2) cross-scale prediction between attentions from different scales, shown in the decoder part of Fig and detailed below.
attention
Similar to the standard MAE, Cross-Scale MAE also adopts a light decoder. Decoding follows the standard MAE decoder where, following the encoder, the removed m patches are placed back into their original location in the sequence of patches where a learned mask token represents the masked patches that were not encoded, and positional encoding is added. Then, a series transformer layers decode all patches. Given the two representations fe, and fe,,, where fe, E (px). fe,t E (pi). after the decoder attention block, we will obtain two new representations, Sa, and fa.-- Then, the cross-scale prediction is applied between them get the cross-scale prediction loss, defined as:
5

---

Table 1:
The information of datasets used in Sec.
Dataset
GSD(m)
Number of Images
Number of Categories
RESISC45
0.2-30
31500
45
WHU-19
0.5
1050
19
UC Merced
0.3
2100
21
EuroSAT
10
27000
10
Cop =
Σ
(Epred (fā, fa,,))
(3)
N
k=1
where is the batch size and the prediction loss ppee is defined as:
Epred (Sá., f(.,)) I1,2,, 9p (12
(4)
follows the Mean Squared Error (MSE) between the prediction and target representation, where the predictor, 9p(-), is multi-layer perceptron (MLP). Besides the cross-scale prediction loss, the decoder reconstructs images of different seales. The reconstruction loss is thus defined as:
Cre 1 E
(IPP: p:ll2 + l|p
(5)
where Dt D(E(pt)) and pa (p()))) Via the cross-scale prediction loss and the reconstruction loss, the consistency and effectiveness of the semantic information of the representation can be better preserved.
The total loss the Cross-Scale MAE is the sum of cross-scale consistency loss at the encoder (Coc), cross-scale prediction loss at the decoder (Cep). and the reconstruction loss (Cre), defined as:
L = Lec + Lep + Lre
(6)
4 Experiments & Results
investigate the quality representations learned from Cross-Scale MAE pre-training through comparisons to state-of-the-art approaches (Sec. 4.1) and comprehensive ablation studies (Sec. 4.2). Sec. 4.3, we briefly explore their robustness to scale variation and transfer performance to additional tasks. We SatMAE [12] the baseline, the state-of stuee-ofthhe of-the-arttee MAE-based approach for remote sensing image analysis. We pre-train Cross-Scall MAE with a ViT-Large model (unless mentioned otherwise) using the Functional Map of the World (fMoW) 363.6k 63k6 images of varying image resolution and GSD. In Sec. 4. we present experiments and results validate the effectiveness of the proposed method, compared with baseline and other state-of-art methods, such as Scale-MAE [6). Section 4.2 presents set of ablation experiments to show how different losses and syper-parameee affect the performance of Cross-Scale MAE. Section 4.3 briefly describes the efficient backbone built through xFoomers.
[T] RGB training set, which consists of
4.1 Comparison with State-of-the-Art
Similar Scale-MAE, we both the K-nearest neighbors (KNN) classification accuracy and performance in downstream tasks as metrics to evaluate the quality of the representation learned by Cross-Scale MAE. Specifically, we pre-train the Cross-Scale MAE on the IMoW-RGB dataset and freeze the encoder as a representation generator; then, we test its effectiveness on different datasets via KNN classification.
KNN Performance. Similar to the original MAE configuration, the encoder Cross-Scale MAE uses a ViT-Large model, and the decoder is light ViT model with the 8-layer attention blocks.
The datasets we use include RESISC45 110, WHU-RS19 47), UC Merced [47], EuroSAT [22] as shown in Table I To evaluate the representation learning capacity of the proposed method with

---

whether it has better generalization capacity than Table 2: Average KNN accuracy with different scale ratios (12.5%, 25% 50% 100%)
RESISC45
WHU-RS19
UC Merced
EuroSAT
SatMAE
66.3
69.9
69.7
81.9
Scale-MAE
70.0
79.5
75.0
86.7
Cross-Scale MAE
75.6
79.8
74.5
87.8
different GSD images, we evaluate the network performance using images with different scale ratios, (12.5%, 25%, 50%, 100%), ). to the raw image to simulate different GSD.
The results are presented in Fig. 2 This figure shows the performance of Cross-Scale MAE compared to both SatMAE and Scale-MAE for different scale ratios. From Fig. 2 we observe that, in general, the Cross-Scale MAE performs better than SatMAE or Scale-MAB in all different scale ratios on all datasets. And, for scale ratio in (25% 50%, 100%). 3. the Cross-Scale MAE can obtain more stable performance compared with SatMAE or Scale-MAE. When the scale ratio equals 12.5%, all three models perform relatively worse. This is because the scale ratio set during pre-training is from 0.2 to 0.8, and 0.125 is out of this range. Nonetheless, Cross-Scale MAE still presents overwhelmingly better performance than the other two. Table shows the average accuracy at all scale ratios and compares Cross-Scale MAE with SatMAE and Scale-MAE. We can find that Cross- Scale MAE presents overwhelmingly better performance than SatMAE all datasets and we observe similar trend in Fig. Cross-Scale MAE generally performs better than Scale-MAE except in the UC Merced dataset. It may be because RESISC45 covers an extensive range of GSD, from 0.2m to 30m, but the other three datasets have fixed GSD. This comparison effectively demonstrates the robustness of representations generated from Cross-Scale MAE, especially on multi-scale datasets.
430044
UcMecod
199-29
6.55
0.50
5.0
500
Figure 2: The KNN classification of Cross-Scale MAE for different datasets.
Downstream Tasks. We further evaluate the performance of Scals-Scale MAE different down- stream tasks to assess the effectiveness of the learned representations in various practical scenarios, including classification and segmentation. We conduct the classification task the fMoW-RGB dataset. The segmentation downstream tasks are performed on two datasets, Potsdam and Vaihungen. We fine-tune the model for 50 epochs for all downstream tasks, following the same hyper-parameter settings as Scale-MAE (6).
To test the capability of handling multi-scale inputs, compare the performance of different methods on low-resolution images with different scaling ratios applied on full-resolution images. Table 3 report the classification performance regarding Top-1 and Top-5 accuracy. In Table 4 we report the segmentation performance regarding mloU. The proposed Cross-Scan Scale MAE performs superior to ofie--o--he-arr art methods in both classification and semantic segmentation downstream tasks.
4.2 Ablation Study
The ablation study is four-fold, First, we investigate the critical role of each of the three loss functions to multi-scale inputs. Second, we investigate the effect of negative samples in different levels of the representation. This includes two parts, with the first evaluating the whole Cross-Scall MAE framework and the second under the pure contrastive learning framework (leaving out the masking patch reconstruction part). Third, we compare the effect of multi-scale input and the GSD positional encoding used in Scale-MAR (36). Although we have shown performance improvement with scale augmentation, extend the investigation
7

---

Table 3: Linear classification fine-tuning performance on fMoW-GBB
50%
Scale= stall=10055
100%
Backbone
Top-1
Top-5
Top-1
Top-5
GASSL
ResNet50
0.494
0.540
0.683
0.715
SeCo
ResNet50
0.508
0.602
0.614
0.796
SatMAE
ViT-Base
0.551
0.776
0.651
0.892
SatMAE
ViT-Large
0.591
0.793
0.678
0,923
Scale-MAE
ViT-Base
0.623
0.850
0.673
0,930
Cross-Scale MAE
ViT-Base
0.632
0.914
0.692
0.954
Table 4: Semantic segmentation performance on Potsdam & Vaihingen (mloU)
Potsdam
Vaihingen
Backbone
25%
50%
100%
25%
50%
100%
Supervised Baseline
ResNet50
0.4518
0.5212
0.7411
0.4603
0.5237
0.7438
SimCLR
ResNer50
0.5327
0.5472
0.7233
0.5286
0.5904
0.7075
BYOL
ResNet50
0.5509
0.5715
0.7264
0,5618
0.5903
0.7298
SimSiam
ResNet50
0.5439
0.5903
0.7188
0,5522
0.6005
0.7205
SeCo
ResNe150
0.5571
0.5935
0.7298
0.5741
0.6213
0.7250
mCL-LC
ResNet50
0.5589
0.5973
0.7316
0.5795
0.6277
0.7262
SatMAE
ViT-Base
0.6325
0.6879
0.7355
0.6239
0.6980
0.7364
Scale-MAE
ViT-Base
0.6853
0.7333
0.7507
0.6910
0.7280
0.7512
Cross-Scale MAE
ViT-Base
0.7082
0.7225
0.7617
0.7163
0.7354 0.7603
the GSD positional encoding. Finally, explore the effects of some syper-parameters including, for example, the number epochs and the type of backbone used. All experiments this section are conducted on the RESISC45 dataset [10] and use the KNN classification as metric.
The weight update of the encoder and decoder networks is mainly controlled by the loss function in Eq. 6 that consists of three modules, the cross consistency loss in the encoder, C... the cross prediction loss, Lep. and reconstruction loss, Lre. Table thoroughly compares the classification accuracy using different loss combinations, from which we make some interesting observations. This experiment conducted on RESISC45 at two scaling ratios, 0.5, 1, on the raw image with ViT-Base and pre-trained 300 epochs fMoW-RGB dataset with input size of 128 x 128.
The first row is essentially SatMAE, with single scale input, and thus serves as the baseline for the subsequent comparisons. The second row adds the multi-scale input and solely uses reconstruction loss. The improved accuracy displays the effectiveness of multi-scale inputs, especially for lower- scale samples. From the second to the last row, all experiments use multi-scale inputs with different loss components. From the 3rd to the 4th row, where only one additional loss module is applied, we observe that both the consistency in the encoder representation and decoder representation are adequate, with the consistency in the decoder representation playing more critical role. Finally, in the last row, we observe that the combination of consistency in the encoder and decoder has performance increment of 3% 4% on both scaling rates, again demonstrating the effectiveness of the proposed Cross-Scale MAE.
Second, we look into the effectiveness of negative samples in formulating the contrastive loss. Because Cross-Scale MAE leverages contrastive learning at two levels to handle multi-level representations, we must investigate the necessity involving negative samples at each stage. Specifically, two kinds of losses are evaluated: InfoNCE relating to positive and negative samples and prediction distance relating to only positive samples. First, we assess under the complete Cross-Scale MAE framework with two scaling ratios, and the results are reported in Table From Table 6 we can find the negative samples in the encoder can improve the classification performance but decrease the performance when involved in the decoder. It may be because the representation from the encoder contains more

---

Table 5; Ablation study of the effect of each of the three losses in Eq. 6 (%)
Multi-Scale
Cross-Consis
Cross-Pred
KNN 50%
KNN 100%
52.1
58.9
68.3
69.2
72.4
74.4
74.9
76.5
78.7
79.3
Table 6: The effect of negative samples (NS) in Cross-Scale MAE (%)
NS in Encoder
NS in Decoder
KNN 50%
KNN 100%
75.9
77.9
76.8
77.7
75.1
77.1
78.7
79.3
"structural" information, but the decoder contains more "semantic" information [36 Hence, the involvement of negative samples in the encoder can help boost the discriminative capacity between multi-scale inputs. On the other hand, with prediction loss between positive samples at the decoder, the consistency in "semantic" information can be better preserved. Having observed this interesting phenomenon, we are intrigued to study if the same pattern would be found in pure contrastive learning (CL) framework, i.e., Cross-Scale MAE without the reconstruction loss. And we did, the results are shown in Table 7
Third, investigate the differences between scale augments and the GSD positional encoding proposed in Scale-MAE [36]. The results are presented in Fig. 3 Two sets comparisons have been conducted the left figure shows the performance of Cross-Scale MAE with without the GSD positional encoding, and the right one shows the SatMAE performance with GSD position or multi-scale augments. The scaling ratios used in the comparison
(25%, 50%, 75%, 100%). From Fig. Ba), we find that Cross-Scale MAE adding GSD positional encoding decreases the performance. The reason may be that Cross-Scale MAE has already used multi-scale augmentation, rendering the GSD positional encoding redundant. On the other hand, from Fig. b), we can find both GSD positional encoding and multi-scale augmentation can improve SatMAE's performance since SatMAE does not exploit scase consistency in the images.
Finally, show the performance of Cross-Scale MAE with different backbones at different training epochs and the effect of different masking ratios. The results are posted supplementary materials.
4.3 Efficient Backbone
Large-scale models are slow train and have high memory consumption. Thus, they are usually trained on large clusters of GPUs. We aim to make the model more accessible and optimize the implementation various ways to minimize the training time and memory footprint for training and inference, such that the model can be feasibly trained end-to-end and used for inference on a single GPU while maintaining competitive results. In this work, we showed using the xFormers library to build an efficient foundation for our models, allowing us to train and conduct experiments more quickly.
Table 7: The effect of negative samples in the pure CL framework (%)
NS in Encoder
NS in Decoder
KNN 50%
KNN 100%
57.2
58.7
57.3
59.5
57.1
58.2
58.3
60.7

---

0.62
Sant MAA-WW Scale
0.60
0.76
1301
0.70
0.54
0.65
Cross MAR GSDP
0.52
Cross-MAL GSDP
0.5
0.6
6.7
0.8
0.9
1.0
0.3
0.4
0.6
0.7
0.5
0.9
1.0
Scale matio to naw image
Scale maaloo to faw image
(b)
Figure 3: The comparison of multi-scale augmentation and GSD positional encoding.
Most of the optimizations offered by the xFormers library were made with Tensor-architectus GPUs mind. We show that we can still gain noticeable improvements in training time even when using an older and less powerful GPU. Thus, for this comparison, we stress-test the xFormers and Timm versions the baseline implementations by training them on individual Nvidia RTX A6000s. The details of the implementation and ablation studies on attention and loss types have been placed in the supplementary.
5 Conclusion & Future Work
This paper introduced Cross-Scale MAE, a novel multi-scale representation learning framework specifically designed for remote sensing image understanding. By incorporating scale augmentation and enforcing cross-scale consistency at both structural and semantic levels, our approach enhances the representational capabilities of the Masked Autoencoder in the context of multi-scale datasets. Remote sensing image analysis, which often relies on diverse multi-source data, greatly benefits from this framework. To achieve more robust and meaningful representations, and generative losses, which encourage consistency and representation quality across different image scales. Furthermore, employed the xFormers library expedite network pre-training without sacrificing performance. Through extensive evaluation of various benchmarks, Cross-Scaa MAE demonstrated superior performance compared to other state-of-the- frameworks for satellite imagery.
utilize both contrastive
Nonetheless, Cross-Scale MAE still has its limitations, which provide directions for future work. (1) In scale augmentation, we currently focus on spatial augmentation (scaling) while maintaining consistent channel content across augmentations. However, the complexity of remote sensing scenes extends beyond just scale differences. Variations in source images include both scale and channel disparities. We currently retain shared channels, like RGB, to address this challenge while discarding differing channels. This approach, although maintaining consistency, could lead to the loss of valuable information from dropped channels. For future work, we aim investigate more into the multi-spectral perspective of representation learning. (2) Our multi-level contrastive learning employs diverse strategies across levels-leveraging both positive and negative samples at the encoder level and exclusively positive samples at the decoder level. This strategy currently yields optimal performance, although the underlying mechanism remains unexplored. In future research, we intend to investigate more into this strategy to gain deeper insights.
Acknowledgements
This research is based upon work supported in part by the Office of the Director of National Intelli- gence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2021-2011 1000006. The views and conclusions contained herein are those of the authors and should not be interpreted
10

---

as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances Neural Information Processing Systems, 33:1877-1900 1901, 2020.
[2] P. Burt and E. H. Adelson. The laplacian pyramid as a compact image code. In Readings in Computer Vision, pages 671 671-679. Elsevier, 1987.
[3] M. Caron, H. Touvron, L. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self--upe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650 (00009660,2200 2021.
[4] E. Catanzariti and A. Mackworth. Forests and pyramids: Using image hicrarchies to understand landsat images. Technical Report 78-2, The University of British Columbia, Vancouver, British Columbia, 1978.
[5] L.-C. Chen, G. Papandreou. I. Kokkinos, K. Murphy, and L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-843 2017.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. Pmlr. 13-18 Jul 2020, doi: 00.48550/axxvx20002.555 chen200 .html
URL ltttss:// mlr press/v119/
[7] T. Chen, S. Korbliihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[8] T. Chen, S. Komblith, K. Swersky, M. Norouzi, and G. E. Hinton, Big self-supervised models are strong semi sume-supervised learners. Advances in Neural Information Processing Sysiems. 01-22233-222555 2020.
191 X. Chen and K. Hc. Exploring simple siamese representation lcarning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 157588 2021.
[10] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification; Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-188) 1883, 2017.
[11] G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In Proceedings of the JEEE Conference on Computer Vision and Pottern Recognition, pages 6172-6180, 2018.
[12] Y. Cong, S. Khanna, C. Meng, Liu, F. Rozi, Y. He, M. Burke, D. B. Lobell, and Ermon. Satmae: Pre- training transformers for temporal and multi-spectral satellite imagery. arXiv preprint arXiv:2207.08N) 2022.
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
[14] P. Dollár and C. I.. Zitnick. Fast edge detection using structured forests. IEEE Transactions on Pattern 2014.
Analysis and Machine Intelligence,
[15] S. Dong, G. Hangel, W. Bogner, G. Widhalm, K. Rössler, S. Trattnig, C. You, R. de Graaf, J. A. Onofrey, and S. Duncan. Multi-scale super-resolution magnetic resonance spectroscopic imaging with adjustable sharpness. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 410 +10-420. Springer, 2022.
[16] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini arXiv:2010.1 19299, 2020.
[17] T. E. S. A. (ESA). Sentinel 2. https:: 2015.
esa. http:://wwwweee...aaalplllllloos httpaa//wwwpppplcaaa.ooo///eeee/ http//////////////////////ooo//r the t
Coperni www.:eunntteneneeee.emm

---

[18] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1955-1229, 2012.
[19] P. Felzenszwalb, D. McAllester, and D. Ramanan. A discriminatively trained, multiscale, deformable part model. In 2008 IEFE Conference on Computer Vision and Pattern Recognition, pages 1 1-8. Icce, 2008.
[20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[21] K. Hc, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision Iearners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19000-1009, 2022.
[22] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-22) 2222, 2019.
[23] P. Ji, S. Yan, and Q. Liu. Region-based spatial sampling for image classification. In 2013 Seventh International Conference on Image and Gruphics, pages 874-879. IEEE, 2013.
[24] L. Jiao, J. Gao, X. Liu, Liu, S. Yang, and Hou. Multi-scale representation learning for image classification; A survey. IEEF Transactions on Artificial Intelligence, 2021.
[25] A. Kovashka and K. Grauman. Learning hicrarchy of discriminative space-time time neighborhood features for human action recognition. In 2010 IEEE International Conference on Computer Vision and Pattern Recognition, pages 2046-2053. IEEE, 2010.
[26] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In 2006 IEEE International Conference on Computer Vision and Patter Recognition (CVPR 06), volume 2, pages 2169-2178. IEEE, 2006.
[27] G. Lin, C. Shen, A. Den Hengel, and Reid, Efficient piecewise training deep structured models for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3194-3203, 2016.
[28] O. Manas, Lacoste, X. Giró-i Nieto, D. Vazquez, and Rodriguez. Seasonal contrast: Unsupervised pre- training from uncurated remote sensing data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9414-9423, 2021.
[29] A. Mapping- Worldview 3. 2014.
[30] M. Mendicta, B. Han, Shi, Y. Zhu, C. Chen, and M. Li. Gfm: Building geospatial foundation models via continual pretraining. arXiv preprint 24X.022302.04476, 2023.
[31] M. Moran, K. Woputz, D. Hee, M. Girotto, P. D'Odorico, R. Gupta, D. Feldman, P. Vahabi, A. Todeschini, and C. J. Reed. Snowpack estimation key mountainous water basins from openly-available, multimodal data sources. arXiv preprint arxiv:2208.0422 2022.
[32] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 180700374, 2018
[33] T. Pohlen, A. Hermans, M. Mathias, and B. Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4151-4166 2017.
[34] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, al. Improving language understanding by generative pre-training. 2018.
[35] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 1(8);9 2019.
[36] C. J. Reed, R. Cupta, S. Li, S. Brockman, C. Funk, B. Clipp. S. Candido, M. Uyttendaele, and T. Darrell. Scalc-mae: A scale-aware masked autocncoder for multiscale geospatial representation learning. arXiv preprint 2212. 14532, 2022.
[37] J. Reininghaus, S. Huber, U. Bauer, and R. Kwitt. A stable multi-scale kernel for topological machine learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4741-4744 2015.
12

---

[38] S. Ren, D. Zhou, S. He, J. Feng, and X. Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10853- 10862, 2022.
[39] A. Rosenfeld and M. Thurston. Edge and curve detection for visual scene analysis. IEEE Transactions on Computers, 100(5):562-33 1971.
[40] B. Singh and L. S. Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3578 3587, 2018.
[41] C. Szegedy, W. Liu, Y. Jia, Sermanet. S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke. and A. Rabinovich. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogmition, pages 1-9, 2015.
[42] Tang. K. Georgiou, H. Qi. Champion, and M. Bosch. Semantic segmentation acrial imagery using multi-level contrastive learning with local consistency. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages -809, 2023.
[43] B. Tellman, J. A. Sullivan, and C. S. Doyle. Global flood observation with multiple satellites: Applications in rio salado (argentina) and the castemn nile basin. Global Drought and Flood: Observation, Modeling, and Prediction, pages 99 121, 2021.
[44] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Asion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[45] USGS. Lansat missions. http:://ww.ugg..go/laadddss.
2013.
[46] Y. Wang, C. M. Albrecht, N. Braham, L. Mou, and X. X. Zhu. supe-supe learning in remote sensing. IEEE Geoscience and Remote Sensing Magazine, December:213 247, 2022.
[47] G.-S. Xia. J. Hu, F. Hu. B. Shi, X. Bai, Y. Zhong. L. Zhang, and X. Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions Geoscience and Remote Sensing, 55(7):3965-399 -9811, 2017.
[48] S. Yan. X. Xu, D. Xu, Lin, and X. Li. Beyond spatial pyramids: A new feature extraction framework with dense spatial sampling for image classification. In Computer ECco-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings. Part IV 12, pages 473-487. Springer, 2012.
[49] S. Yan, X. Xu, D. Xu, S. Lin, and X. Li. Image classification with densely sampled image windows and generalized adaptive multiple kernel learning. IEEE Transactions on Cybernetics, 45(3):381 -390,, 2014.
[50] Zbontar, L. Jing, Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310-12320, PMLR, 2021.
[51] C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, and S. Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self supervied contrastive learning. arXiv preprint arXiv:2203.16262 2022.
[52] D. Zhou, R. Duan, L. Zhao, and X. Chai. Single image super-resolution reconstruction based on multi-scalc feature mapping adversarial network. Signal Processing, 166: 107:107,,,, 2020.
[53] Y. Zou and L. Cheng. A transfer learning model for gesture recognition based on the deep features extracted by cnn. IEEE Transactions Artificial Intelligence, 2(5):447-455 2021.
13


Reviews:
[{"review_id": "GnGQoHcPv8", "reviewer": "Reviewer_CVyA", "summary": "The authors propose a Masked-Auto-Encoder approach for training remote sensing data at different scales.  The model is pretrained on the FMoW-RGB and then the representations assessed through KNN classification performance on four other remote sensing datasets.  Comparisons are made to Sat-MAE and Scale-MAE.  Ablation studies are performed to analyze the impact of the different parts of the loss function.  Performance is seen to be higher than Sat-MAE across different amounts of data and on different datasets and generally higher than Scale-MAE on different datasets (performance with different amounts of data not shown).  ", "questions": "The caption for Figure 2 could be more descriptive explaining the different components.\n\nIt might be more helpful to make Table 5 a chart so the saturation of the performance can be visualized.  It's difficult to tell from the raw numbers if the model has finished training.\n\nThe authors reference the varying number of channels used in remote sensing imagery in the introduction, but then do not explore it here.  How can the method be used to handle imagery from sources with a different number of channels.", "limitations": "Limitations are not explicitly discussed although possible next steps are briefly mentioned.  No negative societal impacts are expected.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The multi-scale problem in remote sensing is a key challenge that rightly deserves research focus.\n\nThe writing is mostly clear, although some sentences or pronoun references could be made clearer in places.\n\nThe ablation studies indicate the contribution of the various architectural components.\n\nKNN performance as compared to SAT-MAE is very promising.\n\nA range of datasets at different scales are analyzed.  ", "weaknesses": "Unlike natural images, the scale in most remote sensing tasks is known because of the geoinformation (this is similar to the situation in microscopy where the zoom/scale is known).  One question is whether features across scales is actually the best approach in this regard because the scale from a given sensor is fixed.  A very relevant approach is given by https://github.com/mahmoodlab/HIPT.  Although originally developed for microscopy images, the same notion around handling large image with a known scale size is explored.\n\nIn the related works, there is no reference to remote-sensing specific work in this field including approaches like SeCo.  A broad review article is mentioned on, but these other methods are not evaluated for comparison.  \n\nGiven the comparison to Sat-MAE and Scale-MAE, it would be beneficial to discuss these explicitly in the related works and how the current method differs from them. \n\nIt would be beneficial to perform the analysis over multiple runs so the confidence internals and significance of performance difference established.  \n\nFigure 3 (performance as a function of data) is only shown in comparison to Sat-MAE which is shown in FIgure 2 to substantially underperform Scale-MAE.  Why not show results from Scale-MAE here as well?\n\nWhile using KNN classification to compare the representations learned, ultimately this does not measure the main goal of SSL models which is to learn a representation which performs better on the downstream task.  KNN evaluation is one measure, but it is also cirtical to analyze with a (full) frozen decoder as well as a fine-tuned decoder. \n\nDetails around training protocol (i.e. learning rate) are missing.  Training on a single GPU is mentioned in 4.3, but it is uncler if this setup is used throughout the work, or only done after the fact as a demonstration that it can be done.\n\nEven though multiple datasets are explored, pretraining is done on a single dataset and then transferred to others.  How does changing which dataset is used impact results?  Furthermore, how does training on multiple datasets with different resolutions work?  In theory this could be the greatest advantage especially if pretraining on low-resolution data (which is often publicly available) could transfer to very high-resolution datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your meaningful comments. Regarding to your question and comments, our answers are listed as following.\n\nTo the weaknesses:\n\nW1. On scale being fixed: We completely agree that remote sensing images usually come with known scale which is fixed for a specific sensing modality. This is why existing practices usually have to pre-train different models to handle images with different scales/resolutions. By introducing scale augmentation as input and cross-scale losses, the Cross-Scale MAE model is able to handle images of various scales using just ONE model. This has made the model \"scale-invariant\" and largely increased the model generalization capacity. Additionally, even though the paper is motivated by remote sensing applications, the same network can be used to process natural images, where scale information usually is not known apriori. The experimental results shown in Supplement D, where the network is pre-trained on CoCo, validate the effectiveness of the model to scale changes.\nFurthermore, from the ablation study summarized in Tab3 of the main text, we also showed the significant performance gain between using scale augmentation (2nd row) and without it (1st row).\n\nW2. On related works like SeCo: We will include references to more related works like SeCo in addition to the broad review article. We also added comparisons to more methods to Tab3 in Supplement C. The new table (Tab2 of rebuttal) shows Cross-Scale MAE outperforms SeCo and Scale-MAE in the linear classification task.\n\nW3. On more discussions to SatMAE and Scale-MAE: We will add the following details. SatMAE is the first paper that applies MAE to extract representations from satellite images but with single and fixed scale. Scale-MAE and the proposed Cross-Scale MAE are also based on MAE, but focus on the multi-scale problem. Specifically, Scale-MAE develops the Ground Sample Distance (GSD) position encoding and applies multi de-convolution after decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD. And, the de-convolution can only result in a specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, it can learn informative and consistent representation across different scales without the need of known GSD. \n\nW4. On the necessity of multiple runs: When employing the ViT backbone, we noticed that displaying discrepancies across multiple training runs isn't customary, echoing practices in related works like SatMAE and Scale-MAE. This stability is influenced by consistent loss convergence and the considerable compute resources needed for training, discouraging multiple runs of the same model setup. Our development process encompassed training multiple models, revealing a consistent convergence pattern across them. Hence, we opted against training multiple runs for the same final model configuration. However, testing runs underwent scrutiny; the results are available in the Supplement (Figs. 3, 5, 8). These graphs stem from 25 runs per testing set sample, with each run involving varied cropping and scaling. In every run, uniform image scale and masking are employed for fair comparison.\n\nW5. On adding Scale-MAE performance to Fig. 3: The Scale-MAE model was not open source until recently. We are thus able to add its performance curve to Fig. 3. Please check Fig.2 of the rebuttal.\n\nW6. On more downstream tasks: We completely agree that having kNN evaluation is insufficient. Due to the usage of cross-scale contrastive loss at the encoder and a cross-scale predictive loss at the decoder, the extracted representation would possess both discriminative and representative characteristics. Hence, downstream tasks should include both classification and segmentation performance. We have since conducted additional experiments on segmentation that further demonstrate consistent improvements over SOTA self-supervised segmentations. The results are shown in Tab1 of the rebuttal. \n\nW7. On details of training protocol. The training details, including hyperparameter choices, can be found in Supplement C (2nd paragraph). We should have referred to it in the main text. It's important to note that all other hyperparameters remained consistent with those utilized for the final models presented in the MAE and SatMAE papers we built upon. Regarding the training GPU setup, we confirm that the models were indeed trained on a single GPU throughout the entirety of our work. \n\nW8. On using multiple datasets with different resolutions to train: The fMoW dataset is indeed a multi-scale dataset with the Ground Sample Distance (GSD) varying from 0.3m-3m. Also, during training, we assume the actual GSD is unknown and apply scale augmentation to synthesize image pairs with different scales. So the trained model can be used to handle input images with a wide range of scale variation - it can handle both single-scale and multi-scale dataset without the need of the ground truth GSD. Besides fMoW, we also pre-trained the model on CoCo to evaluate its effectiveness on natural images. The results were provided in Supplement D.\n\nQ1. On Fig2. We redrew Fig2 with more descriptive caption. Please see Fig3 in rebuttal.\n\nQ2. On Tab5: We converted Tab5 to a line chart. Please see Fig1 in rebuttal.\n\nQ3. On imagery with different channels: This is indeed a challenging problem and has been dealt with in areas like domain adaptation and channel harmonization, where different channels are preprocessed to map to a reference set of channels. Cross-Scale MAE cannot fundamentally solve this problem. For now, we pick the subset of bands shared by different sensing modalities, i.e., RGB and near-infrared.\n\n\n"}, {"review_id": "pH9PLM6F6d", "reviewer": "Reviewer_UGSM", "summary": "This paper presents Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE), which tackles the challenges in remote sensing image understanding (such as extensive coverage, hardware limitations, and misaligned multiscale images) by learning relationships between data at different scales during pretraining. By introducing scale augmentation and enforcing information consistency across multiscale images at both structural and semantic levels, the proposed model demonstrates improved performance in downstream tasks compared to standard MAE and other existing remote sensing MAE methods.", "questions": "1.\tIntroduction should be simplified. The main contributions and motivation should be highlighted as clearly and simply as possible.\n2.\tFrom the current version, I just learn that the authors combine some existing techniques to handle their tasks, limiting the novelty of the proposed model.\n3.\tThe caption of Fig. 2 is incomplete and meaningless. Also, the information conveyed by Fig. 2 does not align well with the description in the text, resulting in limited useful information for readers when interpreting the figure.\n4.\tPlease explain Eq. 1 in detail. If it is proposed by yourselves, please clarify the rationality and significance. Or, please cite the original literature at least.\n5.\tThe multiscale augmentation scheme described in Section 3.2 is just a simple down-sampling. I fail to comprehend the reason behind its inclusion as a section.\n6.\tThe details of the encoder and decoder are missing.\n7.\tThe reasons for the three terms in Eq. 7 are the same should be explained and confirmed.\n8.\tThe compared models are insufficient to confirm the usefulness of cross-scale MAE.\n", "limitations": "See “Questions.”", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Cross-Scale MAE combines discriminative and generative learning approaches, benefiting from self-supervised and multiscale representation learning advances. The novel approach of scale augmentations and multi-level cross-scale consistency constraints ensures consistent and meaningful representations, enabling enhanced remote sensing image understanding.", "weaknesses": "Some points are confusing, which prevents the readers from understanding the main ideas. Also, the compared methods are not enough to illustrate the advantages of the proposed cross-scale MAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1&2) \n\nOn simplifying the Introduction and clarifying contribution and novelty: As suggested, we will remove Fig. 1 and integrate the information there to Fig. 2. We have also redrawn Fig. 2 with more detailed captions to clarify the contribution. Please refer to Fig. 3 of the rebuttal. \n\nEssentially, compared to SatMAE which is the first paper that applies MAE to extract representations from satellite images with single and fixed scale, Scale-MAE and the proposed Cross-Scale MAE are also based on MAE, but focus on the multi-scale problem. Specifically, Scale-MAE develops the Ground Sample Distance (GSD) position encoding and applies multi de-convolution after decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD. And, the de-convolution can only result in a specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, Cross-Scale MAE can learn informative and consistent representation across different scales without the need of known GSD. Additionally, we leverage xFormer on a single GPU for pre-training efficiency.\n\nWhile our pipeline may seem straightforward at first glance, it solves three challenging problems. First, given that masking in MAE can be interpreted as a type of augmentation, are there other types of augmentations that can enhance the robustness of representation learning? This paper figures out an intuitive scale augmentation approach that enables scale-invariant representation. Our ablation study (Table 3 of the main text) showed the significant performance gain between using scale augmentation (2nd row) and without it (1st row). Second, given the multiple-scale inputs, most existing pipelines would extract features independently from these inputs without exploiting the cross-scale correlation. In the proposed pipeline, we designed three losses that exploit this cross-scale correlation at two levels, the cross-scale consistency loss at the encoder embed and the cross-scale predictive loss at the decoder embed. In fact, how to use the contrastive loss to calculate the cross-scale consistency loss is a contribution of its own. The ablation study in Table 3 of the main text also showed the significant contribution of each of the cross-scale loss function (Table 3, rows 3-5). Third, the incorporation of xFormers into our pipeline enables the pre-training process to be performed on single GPU, providing a practical pipeline for ViT-based model training.\n\nQ3. On Fig. 2: Thanks for the suggestion. We will remove Fig. 1. We redrew Fig. 2 with more descriptive caption. Please see Fig. 3 in rebuttal.\n\nQ4. On Eq. 1: This is standard positional encoding. We will make sure to add reference and clarify this is part of the background of basic setting, not our proposed work. \n\nQ5.  On the necessity of having Sec. 3.2 as a section: We've dedicated a section to this process due to its pivotal role in our framework. Different from Scale-MAE's complex position encoding, we use a more intuitive approach—scale augmentation with random ratios—to exploit scale information. This enables the network to learn from data. Leveraging scale pairs, we devise multi-stage contrastive learning. Moreover, our framework transforms into a multi-modality fusion tool, accommodating inputs from diverse sensing modalities, like WV and Sentinel 2, or Sentinel 2 and Landsat, without the need for synthesis.\n\nQ6. Our encoder and decoder architecture follows MAE and SatMAE's method, adopting ViT as the backbone. For the encoder, a ViT processes visible patches, embedding them with positional embeddings. Transformer blocks are utilized, excluding masked patches and negating mask tokens. ViT Base employs an embedding dimension of 768 with 12 transformer blocks, each having 12 attention heads. In ViT Large, the embedding dimension is 1024, and the encoder comprises 24 transformer blocks with 16 attention heads.\nThe decoder incorporates all tokens, including encoded visible patches and mask tokens, representing missing patches. Mask tokens are shared, learned vectors. Although also using Transformer blocks, the decoder is smaller and shallower than the encoder, enhancing efficiency without compromising meaningful representation learning. During pre-training, the decoder guides the encoder, fostering image reconstruction. Both ViT Base and ViT Large maintain an identical decoder. It boasts an embedding dimension of 512, housing 8 layers with 16 attention heads. Furthermore, Fig. 2 has been updated for enhanced encoder/decoder depiction.Please see Fig. 3 of rebuttal.\n\nQ7. Regarding the terms in Eq. 7: These combined losses comprehensively optimize the proposed Cross-Scale MAE model. The cross-scale consistency loss ($L_{cc}$) ensures uniform and dependable features across diverse scales. The decoder's cross-scale prediction loss ($L_{cp}$) highlights the model's versatile scaling prediction, promoting robustness. Lastly, the reconstruction loss ($L_{re}$) guarantees faithful input-output representation, preserving vital details. Together, these losses offer a holistic optimization strategy, each addressing distinct challenges in multi-scale feature alignment and extraction.\n\nQ8. On compared models being insufficient to showcase the proposed: In the Supplement, we posted more results, including the downstream classification task (Supplement C) and zero-shot transfer learning performance on CoCo (Supplement D). We have since added comparisons with more models and on more downstream tasks. In the rebuttal file, please refer to Table 1 (for comparing segmentation performance with SOTA self-supervised segmentation approaches), and Table 2 (for adding linear classification comparisons to three more models)."}, {"review_id": "ufcrct2x2t", "reviewer": "Reviewer_TztZ", "summary": "This paper proposes a Cross-Scale MAE to tackle the multiscale problem in remote sensing images. The triplet loss is designed including cross-scale consistency loss at the encoder, cross-scale prediction loss at the decode, and reconstruction loss. Comparative experiments show competitive performances.", "questions": "Fig.1 can not convey effective information. It just shows different sizes of aircraft, common sense, and very little useful information. I think this picture can be integrated into your research motivation or method innovation, not just three remote-sensing images.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces the cross-scale consistency for image representation.\n2. The scales are aligned in both the encoder and decoder.\n3. Compared experiments show its superiority.\n", "weaknesses": "1. The authors claim the semantics have been aligned in this method. However, the object semantics described in Fig.1 are vague. Because, there are no semantic inputs or guidance (object labels, etc.). Therefore, it is inaccurate to claim that the feature output by the decoder is the semantics. Normally, decode is used for image reconstruction tasks (Fig.2), and the obtained features should be the low-level structure information.\n2. Multiscale is an old research topic. In this paper, methods such as MAE and contrastive learning are applied to align the multi-scale enhanced inputs, most of these techniques exist in contrastive learning. I am not optimistic about the novelty of the paper.\n3. In the experiment, it is only compared with the MAE method in remote sensing. Other contrastive learning methods and the MAE method in machine learning have not been discussed yet. \n4. The experimental accuracies in table 2 should align with their original papers (Sat-MAE, Scale-MAE). Only K-NN testing seems not enough.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your meaningful comments. Regarding to your question and comments, our answers are listed as following.\n\nW1. On “semantic” vs. “structural”: We agree with the reviewer about the meaning of low-level features. We would like to explain our viewpoint by addressing them at two levels of representations used in the loss. In the paper, we used two contrastive losses at different levels of representation; one is the output of the encoder (feature 1), and the other is the output after layernorm of the last self-attention block in the decoder (feature 2), which is used to map to the reconstructed image. Comparing the two representations, we could say that feature 1 is a relatively 'lower' feature than feature 2 and thus contains more structural information. Feature 2 is relatively 'closer' to the reconstructed image, which includes more semantic information than feature 1. We will clarify this by emphasizing that this is a “relative” definition.\n\nW2. On the novelty of the paper: To clarify, we are not 'aligning' multi-scale inputs, but extracting shared representations. Scale augmentation guarantees alignment. \n\nWhile our pipeline may seem straightforward at first glance, it solves three challenging problems. First, given that masking in MAE can be interpreted as a type of augmentation, are there other types of augmentations that can enhance the robustness of representation learning? This paper figures out an intuitive scale augmentation approach that enables scale-invariant representation. Our ablation study (Table 3 of the main text) showed the significant performance gain between using scale augmentation (2nd row) and without it (1st row). \n\nSecond, given the multiple-scale inputs, most existing pipelines would extract features independently from these inputs without exploiting the cross-scale correlation. In the proposed pipeline, we designed three losses that exploit this cross-scale correlation at two levels, the cross-scale consistency loss at the encoder embed and the cross-scale predictive loss at the decoder embed. In fact, how to use the contrastive loss to calculate the cross-scale consistency loss is a contribution of its own. The ablation study in Table 3 of the main text also showed the significant contribution of each of the cross-scale loss function (Table 3, rows 3-5). \n\nThird, the incorporation of xFormers into our pipeline enables the pre-training process to be performed on single GPU, providing a practical pipeline for ViT-based model training. For additional xFormer insights, please refer to Supplement E.\n\nEssentially, compared to SatMAE which is the first paper that applies MAE to extract representations from satellite images with single and fixed scale, Scale-MAE and the proposed Cross-Scale MAE are also based on MAE, but focus on the multi-scale problem. Specifically, Scale-MAE develops the Ground Sample Distance (GSD) position encoding and applies multi de-convolution after decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD. And, the de-convolution can only result in a specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, Cross-Scale MAE can learn informative and consistent representation across different scales without the need of known GSD. \n\n\nW3. On comparing with other contrastive learning and MAE methods: We have added more comparisons in the rebuttal file, including more contrastive learning frameworks and MAE-based framework. Please check Tables 1 and 2 of the rebuttal file.\n\nW4. On KNN testing being insufficient. We completely agree that having kNN evaluation is insufficient. Due to the usage of cross-scale contrastive loss at the encoder and a cross-scale predictive loss at the decoder, the extracted representation would possess both discriminative and representative characteristics. Hence, downstream tasks should include both classification and segmentation performance. We have since conducted additional experiments on segmentation that further demonstrate consistent improvements over SOTA self-supervised segmentations. The results are shown in Table 1 of the rebuttal. \n\nIn addition, the Supplement includes more results, such as the downstream classification task (Supplement C) and zero-shot transfer learning performance on CoCo (Supplement D).\n\nQ1.On Figs. 1 and 2: Thanks for the suggestion. We have redrawn Fig. 2 and combined information from Fig. 1. Please see Fig. 3 in the rebuttal file for a revised illustration of the system architecture. "}, {"review_id": "yzMn6t44x5", "reviewer": "Reviewer_LSpN", "summary": "This paper proposed a flexible self-supervised learning (SSL) framework that yields robust representations named Cross-Scale MAE by enforcing cross-scale information consistency at structural and semantic levels without needing aligned multiscale remote sensing imagery. And this paper deploys xFormers to realize the Cross-Scale MAE.", "questions": "1. What exactly is the semantic understanding of remote sensing image understanding, and are there any specific applications？ such as remote sensing image classification, and remote sensing image change detection,...\n\n2. In Section 4, are there some visualization results and the related analysis to show the advantages of the Cross-Scale MAE?\n\n3. In Section 4, lines 235-238, “In Section 5.1” and “Section 5.2” have errors.\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper designed Cross-Scale MAE to enhance the consistency of the representations obtained from different scales for the multi-scale objects in remote sensing.\n2. This paper deploys xFormers to realize the Cross-Scale MAE, which provides a viable reference for other models to be trained in a single GPU.\n", "weaknesses": "Lack of clarity in expressing the specific problem to be solved by the designed model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your meaningful comments. Regarding to your question and comments, our answers are listed as following.\n\nTo the questions:\n\nQ1. On semantic understanding: We would like to explain our viewpoint by addressing them at two levels of representations used in the loss. In the paper, we used two contrastive losses at different levels of representation; one is the output of the encoder (feature 1), and the other is the output after layernorm of the last self-attention block in the decoder (feature 2), which is used to map to the reconstructed image. Comparing the two representations, we could say that feature 1 is a relatively 'lower' feature than feature 2 and thus contains more structural information. Feature 2 is relatively 'closer' to the reconstructed image, which includes more semantic information than feature 1. We will clarify this by emphasizing that this is a “relative” definition.\n\nDue to the usage of cross-scale contrastive loss at the encoder and a cross-scale predictive loss at the decoder, the extracted representation would possess both discriminative and representative characteristics. Hence, the learned representation can serve well for downstream tasks like classification and segmentation. In the Supplement, we posted more results, including the downstream classification task (Supplement C) and zero-shot transfer learning performance on CoCo (Supplement D). We have since added comparisons with more models and on more downstream tasks. In the rebuttal file, please refer to Table 1 (for comparing segmentation performance with SOTA self-supervised segmentation approaches).\n\n\nQ2. On visualization results: We have included visualization results and related analyses in the Supplement. Supplement A (Figures 1 & 2) provided visualizations of the advantages of our method pre-trained on the fMoW dataset. We also showcased results from a model pre-trained on the CoCo dataset in Figure 4 of Supplement D.1. Finally In Supplement D.2., Figure 6, we provided a visualization of the zero-shot performance of our method where a model pre-trained on the CoCo dataset is tested against samples from the fMoW test set, without any additional fine-tuning on CoCo.\n\nQ3. Thanks for pointing out the typos made in Sec. 4. We will make sure to correct it in the revision."}, {"review_id": "tzrpqR0mHS", "reviewer": "Reviewer_14Vd", "summary": "The authors proposed a self-supervised method based on ViT masked auto encoders (MAE), namely Cross-Scale MAE, to improve the representations learnt in remote sensing based models. It includes a scale augmentation of each input image to learn features at different scales in the encoder while overcoming the need of multiscale aligned data. A constraint is also imposed by exploiting information consistency across multiscale images and improving structural and semantic level representation. \nThe proposed Cross-Scale MAE methods includes both discriminative and generative approaches.\nExperiments have shown better performances on downstream tasks compared to remote sensing self-supervised MAE methods, namely SatMAE and Scale-MAE. The model has been optimized using xFormers to improve its computational efficiency without performance degradation.", "questions": "Questions\n\n1/ Figure 3: why performance curves of Scale-MAE are not included?\n\n2/ The experiments have been conducted considering several fixed scale ratios. Have the performances been quantified using a random scale ratio at each batch during training?\n\n\nComments\n\n1/ Figure 2: AT blocks are not defined, it is not clear in (b) what $f_{e,n}$ is.\n\n2/ Typo: lack of consistency between \"multiscale\" and \"multi-scale\"\n\n3/ Typo: lack of consistency between \"SatMAE\" and \"Sat-MAE\"\n\n4/ Typo: $\\tau$ in Eq. 3 is not defined.\n", "limitations": "The authors have neither detailed the limits, nor the potential negative societal impacts (military or surveillance applications) of the proposed method. The publication of the code and the pre-trained model was not mentioned eithe and will condition the final rating of the submission for reproducibility reasons and the accessibility to the public.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1/ The Cross-Scale MAE method has been clearly explained. It combines both discriminative and generative approaches to learn scale-invariant representations. The cross-scale consistency loss aims to maximize information at different scales while minimizing the sharing information between images at different locations. The cross-scale prediction and reconstruction losses improve the consistency and effectiveness of the decoder to retrieve the semantic information. This original combination is agnostic to the backbone used. It may tackle the actual multiscale problems related to various resolutions in satellite data.\n\n2/  The presented results have shown that Cross-Scale MAE outperforms competing methods on several datasets according the KNN classification accuracy with representation as a metric highlighting the relevance of the learnt presentations.\nThe ablation studies have been rigorously conducted showing the impact of each proposed loss term and the global combination, the effect of the contrastive loss used to constrain the encoder and the influence of the scale augmentation methods.\n\n3/ Experiments on the computational time and memory usage conducted in the Appendices are appreciated.\n\n4/ This work could have a significant impact since large volume of remote sensing data are publicly available without labels. There is an urgent need of efficient and powerful pretrained models for many remote sensing applications, including climate change and global warming understanding and monitoring.\n", "weaknesses": "1/ Additional experiments presented in the appendices have led to similar conclusions than presented in the main document. However, one may notice that the proposed method outperforms SatMAE in average but not consistently considering the presented standard deviations in Appendices Figure 3, Figure 5 and Figure 8. Standard deviations in the main document, including competing methods, would have been appreciated to highlight the consistency of the results.\n\n2/ L231 \"SatMAE is a current state-of-the-art Masked Auto-Encoder (MAE) for remote sensing imagery\" The GFM model [1] could also be considered as a strong MAE competing method outperforming SatMAE. This method has not been compared to Scale-MAE which could be interesting for the community and could potentially highlight the relevance of the proposed method.\n\n\n[1] M. Mendieta et al., GFM: Building Geospatial Foundation Models via Continual Pretraining. In Arxiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your meaningful comments. Regarding to your question and comments, our answers are listed as following.\n\nW1. On moving figures from Supplement to the main text. Thanks for the suggestion. We will move one of these figures to the main text and highlight the following: The graphs in the supplementary material are generated by conducting 25 runs for each sample in the testing dataset. In each of these 25 runs for a specific sample, the sample is cropped to a different random scale before being input into the models. For an individual run, the same image scale and masking configuration are used as inputs to the models being compared to ensure a fair comparison.\n\nW2.  On comparison to GFM. The GFM model excels in representation learning through continual learning, aiming to enhance large language model applicability to satellite images via knowledge distillation. Although GFM's source code is not yet available, we recognize its reported segmentation mIoU of 0.753 on the Vaihingen dataset. In a rough comparison, Cross-Scale MAE achieved an accuracy of 0.7603 on the same task. We will include the GFM paper in our related work for comprehensive context.\n\nQ1. On missing Scale-MAE in Fig. 3. Thanks for pointing this out. Scale-MAE was not open source until recently. We are thus able to add its performance to Fig. 3. Please check Fig. 2 of the rebuttal.\n\nQ2.  On using random scale ratio. Yes, indeed! The model randomly selects a scale ratio from 0.2-0.8 at each batch during the pre-training. \n\n\nTo the comments:\n1. We have improve the model structure figure(figure.2 of the paper) and posted in the rebuttal file as figure.3, please check.\n\n2-4. Thanks for pointing out the typos, we will correct them in the revision."}]

Author rebuttal:
We are appreciate all reviewers' significant comments. 

Initially, we wish to elucidate that our supplementary material, accompanying the main paper, encompasses a wealth of substantial experimental outcomes. This includes the visualization of multiscale representation benefits, downstream task analyses, zero-shot transfer learning demonstrations, and an evaluation of xFormers. We believe that several concerns raised by reviewers can be addressed effectively through the supplementary material.

Furthermore, to comprehensively address reviewer queries, we've curated a rebuttal document. This document integrates additional experiment results as requested by reviewers and incorporates refined figures in accordance with their suggestions.

We trust that our responses and explanations address your inquiries and uncertainties. Feel free to inquire further about our explanations.




Meta-review: The reviewers strongly disagree about this paper. Having read the paper, reviews and rebuttal, the AC find themself agreeing with the most positive reviewers that - while building on existing idea around SSL, MAE and scale augmentation - the paper reports strong results, of interest to the community, with a simple well explained and well presented method, and therefore is happy to recommend the paper for acceptance.

The authors should incorporate additional results from the sup. mat. and the rebuttal in the final version of the paper.
Decision: Accept (poster)