Title: Cross-Scale MAE: A Tale of Multiscale Exploitation in Remote Sensing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Remote Sensting

Self-Supervised Learning
Abstract: Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning prob- lem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned represen- tations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing

Abstract

Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning problem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned representations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

1 Introduction

Remote sensing image understanding has been reaping the benefits of recent advances in computer vision while at the same time posing unique challenges. First, remote sensing imagery usually covers the Earth's surface extensively for Earth observation purposes [52, 43, 31]. Compared to the vast area the images cover, even a large amount of training samples would become sparse. Furthermore, generating a set of representative training samples is challenging, especially in less-explored regions. Hence, self-supervised learning (SSL) becomes a viable solution since no training data is required for representation learning purposes. A recently published paper [46] provides a comprehensive review of SSL in remote sensing.

The second challenge arises from the inherent hardware limitations of remote sensing devices, where one can only expect to acquire images of high resolution in the spatial, spectral, or temporal domain, but not all. These devices usually sacrifice spatial resolution to gain spectral and/or temporal resolution for different functional and/or material analyses of the Earth's surface. Ground Sample Distance (GSD) measures the spatial resolution, denoting the physical distance between two adjacent pixels. For example, Landsat-8 [45] has 8 spectral bands at 30 m GSD; Sentinel-2 [17] has 4 bands at 10 m, 6 bands at 20 m, and 3 bands at 60 m GSD; and WorldView-3 [29] has 8 multi-spectral bands at 1.24 m GSD. These multi-scale images, although imaging the same site, might not be aligned at all, and the alignment process can be both time-consuming and expensive.

It is worth noting that in remote sensing imagery, due to the rich and unique data provided along the temporal and spectral domains, algorithms tend to exploit more on the spectral and temporal organization but neglect the spatial characteristics, as pointed out in [4]. So, extracting effective representations from these misaligned multi-scale images poses a significant challenge. In this

paper, we delve into multi-scale analysis and develop novel ways to fully explore the wealth of information offered by images at multiple scales. We present Cross-Scale MAE, a self-supervised model based on the Masked Autoencoder (MAE) [21] that explicitly learns relationships between data at different scales throughout the pre-training process. By leveraging this information, Cross-Scale MAE produces a robust pre-trained model with superior performance across various GSDs and tasks.

Like MAE, Cross-Scale MAE also masks most of the transformed image and then tasks a Vision Transformer (ViT)-based Autoencoder with embeddings of the unmasked components. A decoding light ViT then decodes the entire image from these learned embeddings, where the decoder is later discarded, and the encoder is used to produce representations for an unmasked input image. The main differences can be addressed from two aspects. First, Cross-Scale MAE introduces the 'scale augmentation' to synthesize inputs of different scales of one input image to engage the transformer to learn features with varying scales in the encoder. The scale augmentation eliminates the need for aligned multi-scale images of the same ground site as input. The pre-trained model is generic enough to extract arbitrary scale input image representations. The second key difference stems from the term 'Cross-Scale', where features extracted from the multi-scale images are not simply concatenated or combined in any ad hoc fashion as many existing fusion frameworks do; instead, Cross-Scale MAE exploits information consistency across the multi-scale images and uses it as constraints to train a more robust Autoencoder model. The cross-scale information consistency is examined at both the encoder and decoder end to ensure consistency is reinforced at both the structural level (i.e., the encoder end) and the semantic level (i.e., the decoder end).

The experimental results overwhelmingly show that Cross-Scale MAE yields more robust multi-scale representations and better performance for downstream tasks as compared to other state-of-theart methods, across a couple of remote sensing datasets with a variety of scale and resolution characteristics. Our contributions can be summarized as follows:

We develop Cross-Scale MAE, a flexible SSL framework that yields robust representations by enforcing cross-scale information consistency at both structural and semantic levels without the need for aligned multi-scale remote sensing imagery.

We investigate the combination of contrastive learning and masked imagery modeling, specifically, the effect of negative samples on representation at different levels.

We deploy xFormers to realize Cross-Scale MAE, where both the pre-training time and memory usage are improved without performance degradation, making large language model training affordable on a single GPU.

The remainder of this paper is organized as follows. Sec. 2 reviews recent developments in SSL and multi-scale representation learning. Sec. 3 elaborates on the proposed Cross-Scale MAE model design. Sec. 4 presents details about experiments and results. Sec. 5 concludes the paper and provides general directions for further improvements.

2 Related Work

We mainly discuss recent works related to SSL and multi-scale representation learning.

Self-Supervised Learning (SSL). Self-supervised learning is a branch of unsupervised learning aiming to learn effective representations for downstream tasks without using training labels. SSL can be divided into two groups, namely, discriminative models and generative models. The discriminative approach, e.g., contrastive learning (CL), requires the representation to be distinctive in information for different inputs. The gist of CL is to make the representations of positive samples close and those of negative samples far from each other, where positive samples are obtained by applying various augmentation schemes on the same image. CL has emerged as a dominant visual SSL method, especially after the advent of MoCo [20] and SimCLR [7]. Negative-free, i.e., non-contrastive, jointembedding methods have been developed [51, 50, 9], demonstrating comparable performance as CL methods. In the remote sensing community, the CL-based representation learning methods have been developed, such as SeCo [28], mCL-LC [42], where the SeCo, short for Seasonal Contrast, offers a novel approach to the challenge of insufficient labeled data in remote sensing by capitalizing on self-supervised learning mechanisms. The mCL-LC method jointly learns global-local representations to improve the segmentation performance for satellite images. On the other hand, the generative approach, e.g., MAE, requires the representation to preserve as much of the input information as

possible. MAE is an autoencoder with masked prediction, i.e., predicting a property of masked input from unmasked input. Benefiting from the success of masked language models [34, 35, 1, 13], the similar mechanism was investigated in vision tasks and has dramatically advanced the field of SSL [16, 21]. Following the trend in computer vision, the masked image model has also shown strength in the remote sensing field. For example, the GFM model [30] excels in representation learning through continual learning, aiming to enhance the applicability of large language models to satellite imagery via knowledge distillation.

In Cross-Scale MAE, both the discriminative and generative mechanisms are used to guarantee the representation from images of different scales to be consistent and meaningful.

Multi-Scale Representation Learning. As mentioned in Sec. 1, the multi-scale phenomenon is common in vision tasks. To leverage the information in multi-scale sources, the vision community has proposed to extract multi-scale features using traditional approaches, including, for example, spatial pyramids [39, 2, 26], dense sampling of windows [23, 48, 49], and the combination of them [19]. In the last decade, deep convolution neural networks (CNNs) have emerged as the de facto architecture for a wide range of vision tasks. Because of the pooling and multi-convolution kernel operation, the CNN constructs the feature pyramids inherently so that it has been used to build deep multiscale representations [24]. CNN-based multi-scale representation learning is commonly approached in two ways. The first approach involves utilizing external or preset factors, which can include multi-scale kernel architecture [41, 25, 37, 53] and multi-scale inputs architecture [14, 18, 27]. The second approach involves designing internal layers of the network, such as skip and dense connections [40, 33, 5]. Over the past three years, there has been a notable surge of interest in applying transformer-based architectures to computer vision tasks. Among these architectures, the Vision Transformer (ViT) [16] stands out as a particularly successful example, where, compared to CNN, ViT balances the global and local features much better. Recently, other efforts have been made for the multi-scale feature learning, including the multi-scale Deformable Attention and Multilevel Features Aggregation (MSDAM and MLFAM) network [15], and the Shunted Self-Attention (SSA) network [38]. These methods have shown strength in multi-level feature extraction for the natural image containing multiple objects of different sizes. However, in remote sensing images, the multi-scale phenomenon is even more prevalent and dynamic. As a result, there has been a growing recognition of the need for customized approaches to address this. The recently published Scale-MAE [36] is one such example.

Essentially, compared with the most recent two state-of-the-art MAE-based satellite imagery representation learning methods, i.e., SatMAE [12], and Scale-MAE [36], SatMAE is the first paper that applies MAE to extract representations from satellite images with single and fixed scale. Scale-MAE and the proposed Cross-Scale MAE are based on MAE but focus on multi-scale characteristics. Specifically, Scale-MAE develops the GSD position encoding and applies multi-de-convolution after the decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD, and the de-convolution can only result in a specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, Cross-Scale MAE can learn informative and consistent representation across different scales without the need of known GSD. Additionally, we leverage xFormers on a single GPU for pre-training efficiency.

3 Methodology

This section elaborates on the Cross-Scale MAE pre-training framework as illustrated in Fig. 1. CrossScale MAE is a self-supervised pre-training framework built upon the MAE [21]. Its objective is to learn consistent representations from multi-scale images captured at the same site by harnessing the strengths of both discriminative and generative learning approaches. Cross-Scale MAE incorporates two novel loss functions at the encoder and decoder stages to achieve this objective. Firstly, the representations obtained from images of different scales are enforced to be consistent. This ensures that the learned representations capture relevant information across scales. Secondly, a decoder is employed to reconstruct the corresponding input, promoting the learned representation to be more representative. Additionally, the decoder utilizes cross-prediction, where the embeddings generated by the decoder from an image with a lower GSD are used to predict the embeddings from an image

Encoder

Patchity

Reconstruction

SS07

Encoder

Embed

Encoder

Decoder

Decoder

Pred

Image

Input

Positional Enc.

Random Masking

Transformer Block

aer

Encoder

Embed

Decoder

Embed

RandomResizedCrop

Contrastive

Loss

MSE Loss

Mean

Final

Loss

Decoder

Encoder

Encoder

Embed

Decoder

Decoder

Embed

Predictor

Linear Layer

nsformer Block

Decoder

Embed

Decoder

Reconstruction

Encoder

Pred

SS07

Embed

Projection

Decoder

Pred

Pachily

with a higher GSD. This cross-prediction mechanism helps capturing and leveraging the multi-scale information present in the images.

3.1 Basic Setting

Let p ∈ R H × W × C represent an input image of height H and width W with C channels. First, multiscale augmentation (Sec. 3.2) is applied to p to generate two images, p h of relatively higher GSD and p l of relatively lower GSD. For each of the augmented images, a sequence of | S | independent patches of height and width of n pixels are created, where each of the patches, s ∈ S , has dimension s ∈ R n 2 C . A fraction, m , of the patches is removed, while the remaining patches are passed through a projection function (e.g., a linear layer) to project the patches s into D dimensions, f emd : R n 2 C → R D , to obtain embedded patches S emd = f emd ( S ) . A standard positional encoding vector is added to the embedded patches and fed into the encoder. After the encoder, the removed m patches are placed back into their original location in the sequence of patches where a learned mask token represents the masked patches that were not encoded. Another positional encoding vector is added to all patches, and a sequence of transformer blocks decodes these patches to form the original input image; these steps are illustrated in Fig. 1.

3.2 Multi-Scale Augmentation

A specialized augmentation method is applied to the input image before patchifying it in the CrossScale MAE framework. This augmentation process generates an additional image of lower GSD for the same site, and the two images are then used as inputs through two independent instances of the MAE network. More specifically, given an input image p with a certain GSD, the framework randomly selects two scale ratios, r h and r l , from the range [0, 1], where r l < r h . The image p is then down-sampled using these scale ratios to generate two images, p h and p l . In our experiments, we set r h = 1 , so p h = p . Both images, p h and p l , are subsequently resized to a fixed size of 128px 2 , and the GSD values of p h and p l are denoted as g h and g l , respectively.

Note that this augmentation approach differs from traditional augmentation, where the rescaled images are traditionally only used as separate samples. Instead, the multi-scale inputs are generated dynamically at runtime and used as inputs through multiple network branches, where the branches have shared weights. The constraints enforced upon the learned representations of these network branches are detailed in the following sections.

3.3 Cross-Scale Information Consistency in the Encoder

Typically, the standard MAE trains a network to reconstruct an image by masking out a significant portion of its pixels. However, the standard MAE encoder does not explicitly address the issue of representation consistency for multi-scale input sources. In contrast, the proposed Cross-Scale MAE aims to learn meaningful and consistent scale-invariant representations. To achieve this, CrossScale MAE focuses on two fundamental properties in the learned representations: (1) consistency of information across different-scale images and (2) representativeness of the raw image. In the Cross-Scale MAE framework, a contrastive learning strategy is employed within the encoder to ensure consistency among representations derived from different-scale images.

The intuitive idea behind Cross-Scale MAE is that different-scale images captured at the same ground site contain highly relevant information despite potential visual differences. Cross-Scale MAE leverages this notion by maximizing the shared information between representations derived from different-scale images of the same site while minimizing the shared information between images from different sites. This approach aligns with the principles of contrastive learning. In contrastive learning, the InfoNCE loss is commonly used to estimate the lower bound of mutual information between data samples. By minimizing the InfoNCE loss, we effectively maximize the mutual information between representations from different scales [44, 32].

Specifically, the encoder E , uses the ViT-base [16] as the backbone. Upon feeding an input image p to encoder E , a high-level representation, E ( p ) , can be obtained. Before computing the contrastive loss, a nonlinear projection head g f ( · ) is needed, which has been proven to be effective [8]. So, after the encoder and projection head, we obtain the representation, z = g f ( E ( p )) . The cross-scale consistency loss is thus defined as:

where p k h and p k l are the different-scale images from the same input image p k , N denotes the number of samples in a mini-batch, and the InfoNCE contrastive loss function ℓ info is the same as in [6], which is defined as follows:

where z k l = g f ( E ( p k l )) , z k h = g f ( E ( p k h )) and Λ is the sample batch. The sim operator is the cosine distance.

3.4 Cross-Scale Prediction in the Decoder

The InfoNCE loss in the encoder provides consistency for multi-scale input but cannot guarantee that the learned representation adequately represents the input. In other words, the representation learned by the network from multi-scale images should be not only consistent among different scales, but also representative of the semantic information. According to [3], the self-supervised ViT automatically learns class-specific features leading to unsupervised object segmentation; specifically, the attention block's last layer includes abundant semantic information. Motivated by [3], the decoder of the Cross-Scale MAE has two tasks: (1) reconstruction of the multi-scale input as close as possible, (2) cross-scale prediction between attentions from different scales, as shown in the decoder part of Fig. 1 and detailed below.

Similar to the standard MAE, Cross-Scale MAE also adopts a light decoder. Decoding follows the standard MAE decoder where, following the encoder, the removed m patches are placed back into their original location in the sequence of patches where a learned mask token represents the masked patches that were not encoded, and a positional encoding is added. Then, a series of transformer layers decode all patches. Given the two representations f e,h and f e,l , where f e,h = E ( p h ) , f e,l = E ( p l ) , after the decoder attention block, we will obtain two new representations, f d,h and f d,l . Then, the cross-scale prediction is applied between them to get the cross-scale prediction loss, defined as:

where N is the batch size and the prediction loss ℓ pred is defined as:

It follows the Mean Squared Error (MSE) between the prediction and target representation, where the predictor, g p ( · ) , is a multi-layer perceptron (MLP). Besides the cross-scale prediction loss, the decoder reconstructs images of different scales. The reconstruction loss is thus defined as:

where ˜ p l = D ( E ( p l )) and ˜ p h = D ( E ( p h )) . Via the cross-scale prediction loss and the reconstruction loss, the consistency and effectiveness of the semantic information of the representation can be better preserved.

The total loss of the Cross-Scale MAE is the sum of cross-scale consistency loss at the encoder ( L cc ), cross-scale prediction loss at the decoder ( L cp ), and the reconstruction loss ( L re ), defined as:

4 Experiments & Results

We investigate the quality of representations learned from Cross-Scale MAE pre-training through comparisons to state-of-the-art approaches (Sec. 4.1) and comprehensive ablation studies (Sec. 4.2). Sec. 4.3, we briefly explore their robustness to scale variation and transfer performance to additional tasks. We use SatMAE [12] as the baseline, the state-of-the-art MAE-based approach for remote sensing image analysis. We pre-train Cross-Scale MAE with a ViT-Large model (unless mentioned otherwise) using the Functional Map of the World (fMoW) [11] RGB training set, which consists of 363 . 6k images of varying image resolution and GSD. In Sec. 4.1, we present experiments and results to validate the effectiveness of the proposed method, compared with baseline and other state-of-art methods, such as Scale-MAE [36]. Section 4.2 presents a set of ablation experiments to show how different losses and hyper-parameters affect the performance of Cross-Scale MAE. Section 4.3 briefly describes the efficient backbone built through xFormers.

4.1 Comparison with State-of-the-Art

Similar to Scale-MAE, we use both the K-nearest neighbors (KNN) classification accuracy and performance in downstream tasks as metrics to evaluate the quality of the representation learned by Cross-Scale MAE. Specifically, we pre-train the Cross-Scale MAE on the fMoW-RGB dataset and freeze the encoder as a representation generator; then, we test its effectiveness on different datasets via KNN classification.

KNN Performance. Similar to the original MAE configuration, the encoder of Cross-Scale MAE uses a ViT-Large model, and the decoder is a light ViT model with the 8-layer attention blocks. The datasets we use include RESISC45 [10], WHU-RS19 [47], UC Merced [47], EuroSAT [22], as shown in Table 1. To evaluate the representation learning capacity of the proposed method with

different GSD images, we evaluate the network performance using images with different scale ratios, { 12 . 5% , 25% , 50% , 100% } , to the raw image to simulate different GSD.

The results are presented in Fig. 2. This figure shows the performance of Cross-Scale MAE compared to both SatMAE and Scale-MAE for different scale ratios. From Fig. 2, we observe that, in general, the Cross-Scale MAE performs better than SatMAE or Scale-MAE in all different scale ratios on all datasets. And, for scale ratio in { 25% , 50% , 100% } , the Cross-Scale MAE can obtain more stable performance compared with SatMAE or Scale-MAE. When the scale ratio equals 12 . 5% , all three models perform relatively worse. This is because the scale ratio set during pre-training is from 0.2 to 0.8, and 0.125 is out of this range. Nonetheless, Cross-Scale MAE still presents overwhelmingly better performance than the other two. Table 2 shows the average accuracy at all scale ratios and compares Cross-Scale MAE with SatMAE and Scale-MAE. We can find that Cross-Scale MAE presents overwhelmingly better performance than SatMAE in all datasets and we observe similar trend as in Fig. 2. Cross-Scale MAE generally performs better than Scale-MAE except in the UC Merced dataset. It may be because RESISC45 covers an extensive range of GSD, from 0 . 2 m to 30 m , but the other three datasets have fixed GSD. This comparison effectively demonstrates the robustness of representations generated from Cross-Scale MAE, especially on multi-scale datasets.

0.9

RESISC45

0.90

UCMerced

060

WHU-19

0.90

EuroSAT

0.8

0.85

0.85

0.85

0.80

080

0.80

KNN accuracy

0.7

accuracy

0.75

accuracy

0.75

accuracy

0.75

0.6

0.70

0.70

0.70

0.5

NNX

0.65

NNX

0.65

NNX

0.65

0.4

0.3

0.2

0.4

0.6

0.8

Cross-MAE

Scale-MAE

10

0.55

0.60

0.50

0.2

0.4

0.6

80

Cross-MAE

Scale-MAE

10

0.55

0.60

0.50

0.2

0.4

0.6

80

Cross-MAE

Scale-MAE

1.0

0.55

0.50

0.60

0.2

0.4

0.6

8'0

Cross-MAE

Scale-MAE

10

Downstream Tasks. We further evaluate the performance of Cross-Scale MAE on different downstream tasks to assess the effectiveness of the learned representations in various practical scenarios, including classification and segmentation. We conduct the classification task on the fMoW-RGB dataset. The segmentation downstream tasks are performed on two datasets, Potsdam and Vaihungen. We fine-tune the model for 50 epochs for all downstream tasks, following the same hyper-parameter settings as Scale-MAE [36].

To test the capability of handling multi-scale inputs, we compare the performance of different methods on low-resolution images with different scaling ratios applied on full-resolution images. In Table 3, we report the classification performance regarding Top-1 and Top-5 accuracy. In Table 4, we report the segmentation performance regarding mIoU. The proposed Cross-Scale MAE performs superior to state-of-the-art methods in both classification and semantic segmentation downstream tasks.

4.2 Ablation Study

The ablation study is four-fold. First, we investigate the critical role of each of the three loss functions to multi-scale inputs. Second, we investigate the effect of negative samples in different levels of the representation. This includes two parts, with the first evaluating the whole Cross-Scale MAE framework and the second under the pure contrastive learning framework (leaving out the masking patch reconstruction part). Third, we compare the effect of multi-scale input and the GSD positional encoding used in Scale-MAE [36]. Although we have shown performance improvement with scale augmentation, we extend the investigation to see whether it has better generalization capacity than

the GSD positional encoding. Finally, we explore the effects of some hyper-parameters, including, for example, the number of epochs and the type of backbone used. All experiments in this section are conducted on the RESISC45 dataset [10] and use the KNN classification as a metric.

The weight update of the encoder and decoder networks is mainly controlled by the loss function in Eq. 6 that consists of three modules, the cross consistency loss in the encoder, L cc , the cross prediction loss, L cp , and the reconstruction loss, L re . Table 5 thoroughly compares the classification accuracy using different loss combinations, from which we make some interesting observations. This experiment is conducted on RESISC45 at two scaling ratios, 0 . 5 , 1 , on the raw image with ViT-Base and pre-trained 300 epochs on fMoW-RGB dataset with input size of 128 × 128 .

The first row is essentially SatMAE, with single scale input, and thus serves as the baseline for the subsequent comparisons. The second row adds the multi-scale input and solely uses reconstruction loss. The improved accuracy displays the effectiveness of multi-scale inputs, especially for lowerscale samples. From the second to the last row, all experiments use multi-scale inputs with different loss components. From the 3rd to the 4th row, where only one additional loss module is applied, we observe that both the consistency in the encoder representation and decoder representation are adequate, with the consistency in the decoder representation playing a more critical role. Finally, in the last row, we observe that the combination of consistency in the encoder and decoder has a performance increment of 3% -4% on both scaling rates, again demonstrating the effectiveness of the proposed Cross-Scale MAE.

Second, we look into the effectiveness of negative samples in formulating the contrastive loss. Because Cross-Scale MAE leverages contrastive learning at two levels to handle multi-level representations, we must investigate the necessity of involving negative samples at each stage. Specifically, two kinds of losses are evaluated: InfoNCE relating to positive and negative samples and prediction distance relating to only positive samples. First, we assess it under the complete Cross-Scale MAE framework with two scaling ratios, and the results are reported in Table 6. From Table 6, we can find the negative samples in the encoder can improve the classification performance but decrease the performance when involved in the decoder. It may be because the representation from the encoder contains more

'structural' information, but the decoder contains more 'semantic' information [36]. Hence, the involvement of negative samples in the encoder can help boost the discriminative capacity between multi-scale inputs. On the other hand, with prediction loss between positive samples at the decoder, the consistency in 'semantic' information can be better preserved. Having observed this interesting phenomenon, we are intrigued to study if the same pattern would be found in a pure contrastive learning (CL) framework, i.e., Cross-Scale MAE without the reconstruction loss. And we did, the results are shown in Table 7.

Third, we investigate the differences between scale augments and the GSD positional encoding proposed in Scale-MAE [36]. The results are presented in Fig. 3. Two sets of comparisons have been conducted - the left figure shows the performance of Cross-Scale MAE with or without the GSD positional encoding, and the right one shows the SatMAE performance with GSD position or multi-scale augments. The scaling ratios used in the comparison are { 25% , 50% , 75% , 100% } . From Fig. 3(a), we find that Cross-Scale MAE adding GSD positional encoding decreases the performance. The reason may be that Cross-Scale MAE has already used multi-scale augmentation, rendering the GSD positional encoding redundant. On the other hand, from Fig. 3(b), we can find both GSD positional encoding and multi-scale augmentation can improve SatMAE's performance since SatMAE does not exploit cross-scale consistency in the images.

Finally, we show the performance of Cross-Scale MAE with different backbones at different training epochs and the effect of different masking ratios. The results are posted in supplementary materials.

4.3 Efficient Backbone

Large-scale models are slow to train and have high memory consumption. Thus, they are usually trained on large clusters of GPUs. We aim to make the model more accessible and optimize the implementation in various ways to minimize the training time and memory footprint for training and inference, such that the model can be feasibly trained end-to-end and used for inference on a single GPU while maintaining competitive results. In this work, we showed using the xFormers library to build an efficient foundation for our models, allowing us to train and conduct experiments more quickly.

0.62

Sat-MAE

0.78

Sat-MAE-w/ GSDP

?

Sat-MAE-w/ Multi-Scale

0.60

0.76

accuracy

0.74

KNN accuracy

0.58

KNN

0.72

0.56

0.70

0.54

0.68

Cross-MAE-w/o GSDP

0.52

0.66

?

CrosS-MAE-w/ GSDP

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

a

(b)

Most of the optimizations offered by the xFormers library were made with Tensor-architecture GPUs in mind. We show that we can still gain noticeable improvements in training time even when using an older and less powerful GPU. Thus, for this comparison, we stress-test the xFormers and Timm versions of the baseline implementations by training them on individual Nvidia RTX A6000s. The details of the implementation and ablation studies on attention and loss types have been placed in the supplementary.

5 Conclusion & Future Work

This paper introduced Cross-Scale MAE, a novel multi-scale representation learning framework specifically designed for remote sensing image understanding. By incorporating scale augmentation and enforcing cross-scale consistency at both structural and semantic levels, our approach enhances the representational capabilities of the Masked Autoencoder in the context of multi-scale datasets. Remote sensing image analysis, which often relies on diverse multi-source data, greatly benefits from this framework. To achieve more robust and meaningful representations, we utilize both contrastive and generative losses, which encourage consistency and representation quality across different image scales. Furthermore, we employed the xFormers library to expedite network pre-training without sacrificing performance. Through extensive evaluation of various benchmarks, Cross-Scale MAE demonstrated superior performance compared to other state-of-the-art frameworks for satellite imagery.

Nonetheless, Cross-Scale MAE still has its limitations, which provide directions for our future work. (1) In scale augmentation, we currently focus on spatial augmentation (scaling) while maintaining consistent channel content across two augmentations. However, the complexity of remote sensing scenes extends beyond just scale differences. Variations in source images include both scale and channel disparities. We currently retain shared channels, like RGB, to address this challenge while discarding differing channels. This approach, although maintaining consistency, could lead to the loss of valuable information from dropped channels. For future work, we aim to investigate more into the multi-spectral perspective of representation learning. (2) Our multi-level contrastive learning employs diverse strategies across levels-leveraging both positive and negative samples at the encoder level and exclusively positive samples at the decoder level. This strategy currently yields optimal performance, although the underlying mechanism remains unexplored. In future research, we intend to investigate more into this strategy to gain deeper insights.

References

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

P. J. Burt and E. H. Adelson. The laplacian pyramid as a compact image code. In Readings in Computer Vision , pages 671-679. Elsevier, 1987.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

E. Catanzariti and A. Mackworth. Forests and pyramids: Using image hierarchies to understand landsat images. Technical Report 78-2, The University of British Columbia, Vancouver, British Columbia, 1978. https://www.cs.ubc.ca/sites/default/files/tr/1978/TR-78-02.pdf.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):834-848, 2017.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1597-1607. Pmlr, 13-18 Jul 2020. doi: 10.48550/arxiv.2002.05709. URL https://proceedings.mlr.press/v119/ chen20j.html .

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607. PMLR, 2020.

T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems , 33:22243-22255, 2020.

X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE , 105(10):1865-1883, 2017.

G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6172-6180, 2018.

Y. Cong, S. Khanna, C. Meng, P. Liu, E. Rozi, Y. He, M. Burke, D. B. Lobell, and S. Ermon. Satmae: Pretraining transformers for temporal and multi-spectral satellite imagery. arXiv preprint arXiv:2207.08051 , 2022.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

P. Dollár and C. L. Zitnick. Fast edge detection using structured forests. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(8):1558-1570, 2014.

S. Dong, G. Hangel, W. Bogner, G. Widhalm, K. Rössler, S. Trattnig, C. You, R. de Graaf, J. A. Onofrey, and J. S. Duncan. Multi-scale super-resolution magnetic resonance spectroscopic imaging with adjustable sharpness. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 410-420. Springer, 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

T. E. S. A. (ESA). Sentinel 2. https://www.esa.int/Applications/Observing_the_Earth/ Copernicus/Sentinel-2_overview , 2015.

C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1915-1929, 2012.

P. Felzenszwalb, D. McAllester, and D. Ramanan. A discriminatively trained, multiscale, deformable part model. In 2008 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8. Ieee, 2008.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

P. Ji, S. Yan, and Q. Liu. Region-based spatial sampling for image classification. In 2013 Seventh International Conference on Image and Graphics , pages 874-879. IEEE, 2013.

L. Jiao, J. Gao, X. Liu, F. Liu, S. Yang, and B. Hou. Multi-scale representation learning for image classification: A survey. IEEE Transactions on Artificial Intelligence , 2021.

A. Kovashka and K. Grauman. Learning a hierarchy of discriminative space-time neighborhood features for human action recognition. In 2010 IEEE International Conference on Computer Vision and Pattern Recognition , pages 2046-2053. IEEE, 2010.

S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In 2006 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 2169-2178. IEEE, 2006.

G. Lin, C. Shen, A. Van Den Hengel, and I. Reid. Efficient piecewise training of deep structured models for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3194-3203, 2016.

O. Manas, A. Lacoste, X. Giró-i Nieto, D. Vazquez, and P. Rodriguez. Seasonal contrast: Unsupervised pretraining from uncurated remote sensing data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9414-9423, 2021.

A. Mapping. Worldview 3. https://apollomapping.com/worldview-3-satellite-imagery , 2014.

M. Mendieta, B. Han, X. Shi, Y. Zhu, C. Chen, and M. Li. Gfm: Building geospatial foundation models via continual pretraining. arXiv preprint arXiv:2302.04476 , 2023.

M. Moran, K. Woputz, D. Hee, M. Girotto, P. D'Odorico, R. Gupta, D. Feldman, P. Vahabi, A. Todeschini, and C. J. Reed. Snowpack estimation in key mountainous water basins from openly-available, multimodal data sources. arXiv preprint arXiv:2208.04246 , 2022.

A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

T. Pohlen, A. Hermans, M. Mathias, and B. Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4151-4160, 2017.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.

C. J. Reed, R. Gupta, S. Li, S. Brockman, C. Funk, B. Clipp, S. Candido, M. Uyttendaele, and T. Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. arXiv preprint arXiv:2212.14532 , 2022.

J. Reininghaus, S. Huber, U. Bauer, and R. Kwitt. A stable multi-scale kernel for topological machine learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4741-4748, 2015.

S. Ren, D. Zhou, S. He, J. Feng, and X. Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1085310862, 2022.

A. Rosenfeld and M. Thurston. Edge and curve detection for visual scene analysis. IEEE Transactions on Computers , 100(5):562-569, 1971.

B. Singh and L. S. Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3578-3587, 2018.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1-9, 2015.

M. Tang, K. Georgiou, H. Qi, C. Champion, and M. Bosch. Semantic segmentation in aerial imagery using multi-level contrastive learning with local consistency. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3798-3807, 2023.

B. Tellman, J. A. Sullivan, and C. S. Doyle. Global flood observation with multiple satellites: Applications in rio salado (argentina) and the eastern nile basin. Global Drought and Flood: Observation, Modeling, and Prediction , pages 99-121, 2021.

Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020.

USGS. Lansat missions. https://www.usgs.gov/landsat-missions , 2013.

Y. Wang, C. M. Albrecht, N. Braham, L. Mou, and X. X. Zhu. Self-supervised learning in remote sensing. IEEE Geoscience and Remote Sensing Magazine , December:213-247, 2022.

G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, and X. Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing , 55(7):3965-3981, 2017.

S. Yan, X. Xu, D. Xu, S. Lin, and X. Li. Beyond spatial pyramids: A new feature extraction framework with dense spatial sampling for image classification. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV 12 , pages 473-487. Springer, 2012.

S. Yan, X. Xu, D. Xu, S. Lin, and X. Li. Image classification with densely sampled image windows and generalized adaptive multiple kernel learning. IEEE Transactions on Cybernetics , 45(3):381-390, 2014.

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, and I. S. Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. arXiv preprint arXiv:2203.16262 , 2022.

D. Zhou, R. Duan, L. Zhao, and X. Chai. Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network. Signal Processing , 166:107251, 2020.

Y. Zou and L. Cheng. A transfer learning model for gesture recognition based on the deep features extracted by cnn. IEEE Transactions on Artificial Intelligence , 2(5):447-458, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a self-supervised method based on ViT masked auto encoders (MAE), namely Cross-Scale MAE, to improve the representations learnt in remote sensing based models. It includes a scale augmentation of each input image to learn features at different scales in the encoder while overcoming the need of multiscale aligned data. A constraint is also imposed by exploiting information consistency across multiscale images and improving structural and semantic level representation. \nThe proposed Cross-Scale MAE methods includes both discriminative and generative approaches.\nExperiments have shown better performances on downstream tasks compared to remote sensing self-supervised MAE methods, namely SatMAE and Scale-MAE. The model has been optimized using xFormers to improve its computational efficiency without performance degradation.\n\nSTRENGTHS:\n1/ The Cross-Scale MAE method has been clearly explained. It combines both discriminative and generative approaches to learn scale-invariant representations. The cross-scale consistency loss aims to maximize information at different scales while minimizing the sharing information between images at different locations. The cross-scale prediction and reconstruction losses improve the consistency and effectiveness of the decoder to retrieve the semantic information. This original combination is agnostic to the backbone used. It may tackle the actual multiscale problems related to various resolutions in satellite data.\n\n2/  The presented results have shown that Cross-Scale MAE outperforms competing methods on several datasets according the KNN classification accuracy with representation as a metric highlighting the relevance of the learnt presentations.\nThe ablation studies have been rigorously conducted showing the impact of each proposed loss term and the global combination, the effect of the contrastive loss used to constrain the encoder and the influence of the scale augmentation methods.\n\n3/ Experiments on the computational time and memory usage conducted in the Appendices are appreciated.\n\n4/ This work could have a significant impact since large volume of remote sensing data are publicly available without labels. There is an urgent need of efficient and powerful pretrained models for many remote sensing applications, including climate change and global warming understanding and monitoring.\n\nWEAKNESSES:\n1/ Additional experiments presented in the appendices have led to similar conclusions than presented in the main document. However, one may notice that the proposed method outperforms SatMAE in average but not consistently considering the presented standard deviations in Appendices Figure 3, Figure 5 and Figure 8. Standard deviations in the main document, including competing methods, would have been appreciated to highlight the consistency of the results.\n\n2/ L231 \"SatMAE is a current state-of-the-art Masked Auto-Encoder (MAE) for remote sensing imagery\" The GFM model [1] could also be considered as a strong MAE competing method outperforming SatMAE. This method has not been compared to Scale-MAE which could be interesting for the community and could potentially highlight the relevance of the proposed method.\n\n\n[1] M. Mendieta et al., GFM: Building Geospatial Foundation Models via Continual Pretraining. In Arxiv 2023.\n\nQUESTIONS:\nQuestions\n\n1/ Figure 3: why performance curves of Scale-MAE are not included?\n\n2/ The experiments have been conducted considering several fixed scale ratios. Have the performances been quantified using a random scale ratio at each batch during training?\n\n\nComments\n\n1/ Figure 2: AT blocks are not defined, it is not clear in (b) what $f_{e,n}$ is.\n\n2/ Typo: lack of consistency between \"multiscale\" and \"multi-scale\"\n\n3/ Typo: lack of consistency between \"SatMAE\" and \"Sat-MAE\"\n\n4/ Typo: $\\tau$ in Eq. 3 is not defined.\n\nLIMITATIONS:\nThe authors have neither detailed the limits, nor the potential negative societal impacts (military or surveillance applications) of the proposed method. The publication of the code and the pre-trained model was not mentioned eithe and will condition the final rating of the submission for reproducibility reasons and the accessibility to the public.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper proposed a flexible self-supervised learning (SSL) framework that yields robust representations named Cross-Scale MAE by enforcing cross-scale information consistency at structural and semantic levels without needing aligned multiscale remote sensing imagery. And this paper deploys xFormers to realize the Cross-Scale MAE.

STRENGTHS:
1. This paper designed Cross-Scale MAE to enhance the consistency of the representations obtained from different scales for the multi-scale objects in remote sensing.
2. This paper deploys xFormers to realize the Cross-Scale MAE, which provides a viable reference for other models to be trained in a single GPU.

WEAKNESSES:
Lack of clarity in expressing the specific problem to be solved by the designed model

QUESTIONS:
1. What exactly is the semantic understanding of remote sensing image understanding, and are there any specific applications？ such as remote sensing image classification, and remote sensing image change detection,...

2. In Section 4, are there some visualization results and the related analysis to show the advantages of the Cross-Scale MAE?

3. In Section 4, lines 235-238, “In Section 5.1” and “Section 5.2” have errors.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Cross-Scale MAE to tackle the multiscale problem in remote sensing images. The triplet loss is designed including cross-scale consistency loss at the encoder, cross-scale prediction loss at the decode, and reconstruction loss. Comparative experiments show competitive performances.\n\nSTRENGTHS:\n1. This paper introduces the cross-scale consistency for image representation.\n2. The scales are aligned in both the encoder and decoder.\n3. Compared experiments show its superiority.\n\nWEAKNESSES:\n1. The authors claim the semantics have been aligned in this method. However, the object semantics described in Fig.1 are vague. Because, there are no semantic inputs or guidance (object labels, etc.). Therefore, it is inaccurate to claim that the feature output by the decoder is the semantics. Normally, decode is used for image reconstruction tasks (Fig.2), and the obtained features should be the low-level structure information.\n2. Multiscale is an old research topic. In this paper, methods such as MAE and contrastive learning are applied to align the multi-scale enhanced inputs, most of these techniques exist in contrastive learning. I am not optimistic about the novelty of the paper.\n3. In the experiment, it is only compared with the MAE method in remote sensing. Other contrastive learning methods and the MAE method in machine learning have not been discussed yet. \n4. The experimental accuracies in table 2 should align with their original papers (Sat-MAE, Scale-MAE). Only K-NN testing seems not enough.\n\nQUESTIONS:\nFig.1 can not convey effective information. It just shows different sizes of aircraft, common sense, and very little useful information. I think this picture can be integrated into your research motivation or method innovation, not just three remote-sensing images.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE), which tackles the challenges in remote sensing image understanding (such as extensive coverage, hardware limitations, and misaligned multiscale images) by learning relationships between data at different scales during pretraining. By introducing scale augmentation and enforcing information consistency across multiscale images at both structural and semantic levels, the proposed model demonstrates improved performance in downstream tasks compared to standard MAE and other existing remote sensing MAE methods.\n\nSTRENGTHS:\nCross-Scale MAE combines discriminative and generative learning approaches, benefiting from self-supervised and multiscale representation learning advances. The novel approach of scale augmentations and multi-level cross-scale consistency constraints ensures consistent and meaningful representations, enabling enhanced remote sensing image understanding.\n\nWEAKNESSES:\nSome points are confusing, which prevents the readers from understanding the main ideas. Also, the compared methods are not enough to illustrate the advantages of the proposed cross-scale MAE.\n\nQUESTIONS:\n1.\tIntroduction should be simplified. The main contributions and motivation should be highlighted as clearly and simply as possible.\n2.\tFrom the current version, I just learn that the authors combine some existing techniques to handle their tasks, limiting the novelty of the proposed model.\n3.\tThe caption of Fig. 2 is incomplete and meaningless. Also, the information conveyed by Fig. 2 does not align well with the description in the text, resulting in limited useful information for readers when interpreting the figure.\n4.\tPlease explain Eq. 1 in detail. If it is proposed by yourselves, please clarify the rationality and significance. Or, please cite the original literature at least.\n5.\tThe multiscale augmentation scheme described in Section 3.2 is just a simple down-sampling. I fail to comprehend the reason behind its inclusion as a section.\n6.\tThe details of the encoder and decoder are missing.\n7.\tThe reasons for the three terms in Eq. 7 are the same should be explained and confirmed.\n8.\tThe compared models are insufficient to confirm the usefulness of cross-scale MAE.\n\nLIMITATIONS:\nSee “Questions.”", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Masked-Auto-Encoder approach for training remote sensing data at different scales.  The model is pretrained on the FMoW-RGB and then the representations assessed through KNN classification performance on four other remote sensing datasets.  Comparisons are made to Sat-MAE and Scale-MAE.  Ablation studies are performed to analyze the impact of the different parts of the loss function.  Performance is seen to be higher than Sat-MAE across different amounts of data and on different datasets and generally higher than Scale-MAE on different datasets (performance with different amounts of data not shown).\n\nSTRENGTHS:\nThe multi-scale problem in remote sensing is a key challenge that rightly deserves research focus.\n\nThe writing is mostly clear, although some sentences or pronoun references could be made clearer in places.\n\nThe ablation studies indicate the contribution of the various architectural components.\n\nKNN performance as compared to SAT-MAE is very promising.\n\nA range of datasets at different scales are analyzed.\n\nWEAKNESSES:\nUnlike natural images, the scale in most remote sensing tasks is known because of the geoinformation (this is similar to the situation in microscopy where the zoom/scale is known).  One question is whether features across scales is actually the best approach in this regard because the scale from a given sensor is fixed.  A very relevant approach is given by https://github.com/mahmoodlab/HIPT.  Although originally developed for microscopy images, the same notion around handling large image with a known scale size is explored.\n\nIn the related works, there is no reference to remote-sensing specific work in this field including approaches like SeCo.  A broad review article is mentioned on, but these other methods are not evaluated for comparison.  \n\nGiven the comparison to Sat-MAE and Scale-MAE, it would be beneficial to discuss these explicitly in the related works and how the current method differs from them. \n\nIt would be beneficial to perform the analysis over multiple runs so the confidence internals and significance of performance difference established.  \n\nFigure 3 (performance as a function of data) is only shown in comparison to Sat-MAE which is shown in FIgure 2 to substantially underperform Scale-MAE.  Why not show results from Scale-MAE here as well?\n\nWhile using KNN classification to compare the representations learned, ultimately this does not measure the main goal of SSL models which is to learn a representation which performs better on the downstream task.  KNN evaluation is one measure, but it is also cirtical to analyze with a (full) frozen decoder as well as a fine-tuned decoder. \n\nDetails around training protocol (i.e. learning rate) are missing.  Training on a single GPU is mentioned in 4.3, but it is uncler if this setup is used throughout the work, or only done after the fact as a demonstration that it can be done.\n\nEven though multiple datasets are explored, pretraining is done on a single dataset and then transferred to others.  How does changing which dataset is used impact results?  Furthermore, how does training on multiple datasets with different resolutions work?  In theory this could be the greatest advantage especially if pretraining on low-resolution data (which is often publicly available) could transfer to very high-resolution datasets.\n\nQUESTIONS:\nThe caption for Figure 2 could be more descriptive explaining the different components.\n\nIt might be more helpful to make Table 5 a chart so the saturation of the performance can be visualized.  It's difficult to tell from the raw numbers if the model has finished training.\n\nThe authors reference the varying number of channels used in remote sensing imagery in the introduction, but then do not explore it here.  How can the method be used to handle imagery from sources with a different number of channels.\n\nLIMITATIONS:\nLimitations are not explicitly discussed although possible next steps are briefly mentioned.  No negative societal impacts are expected.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}