SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maofeng Tang<sup>1</sup>, Andrei Cozma<sup>1</sup>, Konstantinos Georgiou<sup>1</sup>, Hairong Qi<sup>1</sup>
<sup>1</sup>Min H. Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville
{mtang4, acozma, kgeorgio}@vols.utk.edu, hqi@utk.edu,
Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning problem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned representations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.
Remote sensing image understanding has been reaping the benefits of recent advances in computer vision while at the same time posing unique challenges. First, remote sensing imagery usually covers the Earth's surface extensively for Earth observation purposes [52, 43, 31]. Compared to the vast area the images cover, even a large amount of training samples would become sparse. Furthermore, generating a set of representative training samples is challenging, especially in less-explored regions. Hence, self-supervised learning (SSL) becomes a viable solution since no training data is required for representation learning purposes. A recently published paper [46] provides a comprehensive review of SSL in remote sensing.
The second challenge arises from the inherent hardware limitations of remote sensing devices, where one can only expect to acquire images of high resolution in the spatial, spectral, or temporal domain, but not all. These devices usually sacrifice spatial resolution to gain spectral and/or temporal resolution for different functional and/or material analyses of the Earth's surface. Ground Sample Distance (GSD) measures the spatial resolution, denoting the physical distance between two adjacent pixels. For example, Landsat-8 [45] has 8 spectral bands at  $30\mathrm{m}$  GSD; Sentinel-2 [17] has 4 bands at  $10\mathrm{m}$ , 6 bands at  $20\mathrm{m}$ , and 3 bands at  $60\mathrm{m}$  GSD; and WorldView-3 [29] has 8 multi-spectral bands at  $1.24\mathrm{m}$  GSD. These multi-scale images, although imaging the same site, might not be aligned at all, and the alignment process can be both time-consuming and expensive.
It is worth noting that in remote sensing imagery, due to the rich and unique data provided along the temporal and spectral domains, algorithms tend to exploit more on the spectral and temporal organization but neglect the spatial characteristics, as pointed out in [4]. So, extracting effective representations from these misaligned multi-scale images poses a significant challenge. In this
paper, we delve into multi-scale analysis and develop novel ways to fully explore the wealth of information offered by images at multiple scales. We present Cross-Scale MAE, a self-supervised model based on the Masked Autoencoder (MAE) [21] that explicitly learns relationships between data at different scales throughout the pre-training process. By leveraging this information, Cross-Scale MAE produces a robust pre-trained model with superior performance across various GSDs and tasks.
Like MAE, Cross-Scale MAE also masks most of the transformed image and then tasks a Vision Transformer (ViT)-based Autoencoder with embeddings of the unmasked components. A decoding light ViT then decodes the entire image from these learned embeddings, where the decoder is later discarded, and the encoder is used to produce representations for an unmasked input image. The main differences can be addressed from two aspects. First, Cross-Scale MAE introduces the "scale augmentation" to synthesize inputs of different scales of one input image to engage the transformer to learn features with varying scales in the encoder. The scale augmentation eliminates the need for aligned multi-scale images of the same ground site as input. The pre-trained model is generic enough to extract arbitrary scale input image representations. The second key difference stems from the term "Cross-Scale", where features extracted from the multi-scale images are not simply concatenated or combined in any ad hoc fashion as many existing fusion frameworks do; instead, Cross-Scale MAE exploits information consistency across the multi-scale images and uses it as constraints to train a more robust Autoencoder model. The cross-scale information consistency is examined at both the encoder and decoder end to ensure consistency is reinforced at both the structural level (i.e., the encoder end) and the semantic level (i.e., the decoder end).
The experimental results overwhelmingly show that Cross-Scale MAE yields more robust multi-scale representations and better performance for downstream tasks as compared to other state-of-the-art methods, across a couple of remote sensing datasets with a variety of scale and resolution characteristics. Our contributions can be summarized as follows:
(1) We develop Cross-Scale MAE, a flexible SSL framework that yields robust representations by enforcing cross-scale information consistency at both structural and semantic levels without the need for aligned multi-scale remote sensing imagery.
(2) We investigate the combination of contrastive learning and masked imagery modeling, specifically, the effect of negative samples on representation at different levels.
(3) We deploy xFormers to realize Cross-Scale MAE, where both the pre-training time and memory usage are improved without performance degradation, making large language model training affordable on a single GPU.
The remainder of this paper is organized as follows. Sec. 2 reviews recent developments in SSL and multi-scale representation learning. Sec. 3 elaborates on the proposed Cross-Scale MAE model design. Sec. 4 presents details about experiments and results. Sec. 5 concludes the paper and provides general directions for further improvements.
We mainly discuss recent works related to SSL and multi-scale representation learning.
Self-Supervised Learning (SSL). Self-supervised learning is a branch of unsupervised learning aiming to learn effective representations for downstream tasks without using training labels. SSL can be divided into two groups, namely, discriminative models and generative models. The discriminative approach, e.g., contrastive learning (CL), requires the representation to be distinctive in information for different inputs. The gist of CL is to make the representations of positive samples close and those of negative samples far from each other, where positive samples are obtained by applying various augmentation schemes on the same image. CL has emerged as a dominant visual SSL method, especially after the advent of MoCo [20] and SimCLR [7]. Negative-free, i.e., non-contrastive, joint-embedding methods have been developed [51, 50, 9], demonstrating comparable performance as CL methods. In the remote sensing community, the CL-based representation learning methods have been developed, such as SeCo [28], mCL-LC [42], where the SeCo, short for Seasonal Contrast, offers a novel approach to the challenge of insufficient labeled data in remote sensing by capitalizing on self-supervised learning mechanisms. The mCL-LC method jointly learns global-local representations to improve the segmentation performance for satellite images. On the other hand, the generative approach, e.g., MAE, requires the representation to preserve as much of the input information as
possible. MAE is an autoencoder with masked prediction, i.e., predicting a property of masked input from unmasked input. Benefiting from the success of masked language models [34, 35, 1, 13], the similar mechanism was investigated in vision tasks and has dramatically advanced the field of SSL [16, 21]. Following the trend in computer vision, the masked image model has also shown strength in the remote sensing field. For example, the GFM model [30] excels in representation learning through continual learning, aiming to enhance the applicability of large language models to satellite imagery via knowledge distillation.
In Cross-Scale MAE, both the discriminative and generative mechanisms are used to guarantee the representation from images of different scales to be consistent and meaningful.
Multi-Scale Representation Learning. As mentioned in Sec. 1, the multi-scale phenomenon is common in vision tasks. To leverage the information in multi-scale sources, the vision community has proposed to extract multi-scale features using traditional approaches, including, for example, spatial pyramids [39, 2, 26], dense sampling of windows [23, 48, 49], and the combination of them [19]. In the last decade, deep convolution neural networks (CNNs) have emerged as the de facto architecture for a wide range of vision tasks. Because of the pooling and multi-convolution kernel operation, the CNN constructs the feature pyramids inherently so that it has been used to build deep multiscale representations [24]. CNN-based multi-scale representation learning is commonly approached in two ways. The first approach involves utilizing external or preset factors, which can include multi-scale kernel architecture [41, 25, 37, 53] and multi-scale inputs architecture [14, 18, 27]. The second approach involves designing internal layers of the network, such as skip and dense connections [40, 33, 5]. Over the past three years, there has been a notable surge of interest in applying transformer-based architectures to computer vision tasks. Among these architectures, the Vision Transformer (ViT) [16] stands out as a particularly successful example, where, compared to CNN, ViT balances the global and local features much better. Recently, other efforts have been made for the multi-scale feature learning, including the multi-scale Deformable Attention and Multi-level Features Aggregation (MSDAM and MLFAM) network [15], and the Shunted Self-Attention (SSA) network [38]. These methods have shown strength in multi-level feature extraction for the natural image containing multiple objects of different sizes. However, in remote sensing images, the multi-scale phenomenon is even more prevalent and dynamic. As a result, there has been a growing recognition of the need for customized approaches to address this. The recently published Scale-MAE [36] is one such example.
Essentially, compared with the most recent two state-of-the-art MAE-based satellite imagery representation learning methods, i.e., SatMAE [12], and Scale-MAE [36], SatMAE is the first paper that applies MAE to extract representations from satellite images with single and fixed scale. Scale-MAE and the proposed Cross-Scale MAE are based on MAE but focus on multi-scale characteristics. Specifically, Scale-MAE develops the GSD position encoding and applies multi-de-convolution after the decoder to reconstruct images of different scales. Nonetheless, Scale-MAE integrates the scale information into the network via hard coding of known GSD, and the de-convolution can only result in a specific scale ratio. The proposed Cross-Scale MAE designs the network to learn the information across different scales. With scale augmentation and multi-level contrastive loss between the scale pair and masked patches reconstruction, Cross-Scale MAE can learn informative and consistent representation across different scales without the need of known GSD. Additionally, we leverage xFormers on a single GPU for pre-training efficiency.
This section elaborates on the Cross-Scale MAE pre-training framework as illustrated in Fig. 1. Cross-Scale MAE is a self-supervised pre-training framework built upon the MAE [21]. Its objective is to learn consistent representations from multi-scale images captured at the same site by harnessing the strengths of both discriminative and generative learning approaches. Cross-Scale MAE incorporates two novel loss functions at the encoder and decoder stages to achieve this objective. Firstly, the representations obtained from images of different scales are enforced to be consistent. This ensures that the learned representations capture relevant information across scales. Secondly, a decoder is employed to reconstruct the corresponding input, promoting the learned representation to be more representative. Additionally, the decoder utilizes cross-prediction, where the embeddings generated by the decoder from an image with a lower GSD are used to predict the embeddings from an image
Figure 1: The architecture of Cross-Scale MAE comprises an encoder and a decoder (Left). The encoder (Top-Right) employs a vision transformer (ViT) backbone, specifically a ViT-base with 12 self-attention blocks. The decoder (Bottom-Right) uses a lightweight ViT backbone with 8 self-attention blocks. A single satellite image undergoes scale augmentation through random cropping and resizing. The contrastive loss is computed using encoder outputs for the two scale inputs. The cross-prediction loss (in MSE) is applied to the last self-attention block's decoder output. Reconstruction loss compares the predicted masked patch and the actual input.
with a higher GSD. This cross-prediction mechanism helps capturing and leveraging the multi-scale information present in the images.
3.1 Basic Setting
Let  $p \in \mathbb{R}^{H \times W \times C}$  represent an input image of height  $H$  and width  $W$  with  $C$  channels. First, multiscale augmentation (Sec. 3.2) is applied to  $p$  to generate two images,  $p_h$  of relatively higher GSD and  $p_l$  of relatively lower GSD. For each of the augmented images, a sequence of  $|S|$  independent patches of height and width of  $n$  pixels are created, where each of the patches,  $s \in S$ , has dimension  $s \in \mathbb{R}^{n^2C}$ . A fraction,  $m$ , of the patches is removed, while the remaining patches are passed through a projection function (e.g., a linear layer) to project the patches  $s$  into  $D$  dimensions,  $f_{emd}: \mathbb{R}^{n^2C} \to \mathbb{R}^D$ , to obtain embedded patches  $S_{emd} = f_{emd}(S)$ . A standard positional encoding vector is added to the embedded patches and fed into the encoder. After the encoder, the removed  $m$  patches are placed back into their original location in the sequence of patches where a learned mask token represents the masked patches that were not encoded. Another positional encoding vector is added to all patches, and a sequence of transformer blocks decodes these patches to form the original input image; these steps are illustrated in Fig. 1.
3.2 Multi-Scale Augmentation
A specialized augmentation method is applied to the input image before patchifying it in the CrossScale MAE framework. This augmentation process generates an additional image of lower GSD for the same site, and the two images are then used as inputs through two independent instances of the MAE network. More specifically, given an input image  $p$  with a certain GSD, the framework randomly selects two scale ratios,  $r_h$  and  $r_l$ , from the range [0, 1], where  $r_l < r_h$ . The image  $p$  is then down-sampled using these scale ratios to generate two images,  $p_h$  and  $p_l$ . In our experiments, we set  $r_h = 1$ , so  $p_h = p$ . Both images,  $p_h$  and  $p_l$ , are subsequently resized to a fixed size of  $128\mathrm{px}^2$ , and the GSD values of  $p_h$  and  $p_l$  are denoted as  $g_h$  and  $g_l$ , respectively.
Note that this augmentation approach differs from traditional augmentation, where the rescaled images are traditionally only used as separate samples. Instead, the multi-scale inputs are generated dynamically at runtime and used as inputs through multiple network branches, where the branches have shared weights. The constraints enforced upon the learned representations of these network branches are detailed in the following sections.
3.3 Cross-Scale Information Consistency in the Encoder
Typically, the standard MAE trains a network to reconstruct an image by masking out a significant portion of its pixels. However, the standard MAE encoder does not explicitly address the issue of representation consistency for multi-scale input sources. In contrast, the proposed Cross-Scale MAE aims to learn meaningful and consistent scale-invariant representations. To achieve this, Cross-Scale MAE focuses on two fundamental properties in the learned representations: (1) consistency of information across different-scale images and (2) representativeness of the raw image. In the Cross-Scale MAE framework, a contrastive learning strategy is employed within the encoder to ensure consistency among representations derived from different-scale images.
The intuitive idea behind Cross-Scale MAE is that different-scale images captured at the same ground site contain highly relevant information despite potential visual differences. Cross-Scale MAE leverages this notion by maximizing the shared information between representations derived from different-scale images of the same site while minimizing the shared information between images from different sites. This approach aligns with the principles of contrastive learning. In contrastive learning, the InfoNCE loss is commonly used to estimate the lower bound of mutual information between data samples. By minimizing the InfoNCE loss, we effectively maximize the mutual information between representations from different scales [44, 32].
Specifically, the encoder  $E$ , uses the ViT-base [16] as the backbone. Upon feeding an input image  $p$  to encoder  $E$ , a high-level representation,  $E(p)$ , can be obtained. Before computing the contrastive loss, a nonlinear projection head  $g_{f}(\cdot)$  is needed, which has been proven to be effective [8]. So, after the encoder and projection head, we obtain the representation,  $z = g_{f}(E(p))$ . The cross-scale consistency loss is thus defined as:
$$
\mathcal {L} _ {c c} = \frac {1}{2 N} \sum_ {k = 1} ^ {N} \left(\ell_ {i n f o} \left(p _ {l} ^ {k}, p _ {h} ^ {k}\right) + \ell_ {i n f o} \left(p _ {h} ^ {k}, p _ {l} ^ {k}\right)\right) \tag {1}
$$
where  $p_h^k$  and  $p_l^k$  are the different-scale images from the same input image  $p_k$ ,  $N$  denotes the number of samples in a mini-batch, and the InfoNCE contrastive loss function  $\ell_{\mathrm{info}}$  is the same as in [6], which is defined as follows:
$$
\ell_ {i n f o} \left(p _ {l} ^ {k}, p _ {h} ^ {k}\right) = - \log \frac {\exp \left(\sin \left(z _ {l} ^ {k} , z _ {h} ^ {k}\right) / \tau\right)}{\sum_ {p \in \Lambda} \exp \left(\frac {\sin \left(z _ {l} ^ {k} , g _ {f} (E (p))\right)}{\tau}\right)} \tag {2}
$$
where  $z_{l}^{k} = g_{f}\left(E\left(p_{l}^{k}\right)\right)$ ,  $z_{h}^{k} = g_{f}\left(E\left(p_{h}^{k}\right)\right)$  and  $\Lambda$  is the sample batch. The sim operator is the cosine distance.
3.4 Cross-Scale Prediction in the Decoder
The InfoNCE loss in the encoder provides consistency for multi-scale input but cannot guarantee that the learned representation adequately represents the input. In other words, the representation learned by the network from multi-scale images should be not only consistent among different scales, but also representative of the semantic information. According to [3], the self-supervised ViT automatically learns class-specific features leading to unsupervised object segmentation; specifically, the attention block's last layer includes abundant semantic information. Motivated by [3], the decoder of the Cross-Scale MAE has two tasks: (1) reconstruction of the multi-scale input as close as possible, (2) cross-scale prediction between attentions from different scales, as shown in the decoder part of Fig. 1 and detailed below.
Similar to the standard MAE, Cross-Scale MAE also adopts a light decoder. Decoding follows the standard MAE decoder where, following the encoder, the removed  $m$  patches are placed back into their original location in the sequence of patches where a learned mask token represents the masked patches that were not encoded, and a positional encoding is added. Then, a series of transformer layers decode all patches. Given the two representations  $f_{e,h}$  and  $f_{e,l}$ , where  $f_{e,h} = E(p_h)$ ,  $f_{e,l} = E(p_l)$ , after the decoder attention block, we will obtain two new representations,  $f_{d,h}$  and  $f_{d,l}$ . Then, the cross-scale prediction is applied between them to get the cross-scale prediction loss, defined as:
Table 1: The information of datasets used in Sec. 4
Dataset	GSD(m)	Number of Images	Number of Categories
RESISC45	0.2-30	31500	45
WHU-19	0.5	1050	19
UC Merced	0.3	2100	21
EuroSAT	10	27000	10
$$
\mathcal {L} _ {c p} = \frac {1}{N} \sum_ {k = 1} ^ {N} \left(\ell_ {p r e d} \left(f _ {d, l} ^ {k}, f _ {d, h} ^ {k}\right)\right) \tag {3}
$$
where  $N$  is the batch size and the prediction loss  $\ell_{pred}$  is defined as:
$$
\ell_ {p r e d} \left(f _ {d, l} ^ {k}, f _ {d, h} ^ {k}\right) = \left| \left| f _ {d, h} ^ {k} - g _ {p} \left(f _ {d, l} ^ {k}\right) \right| \right| _ {2} ^ {2} \tag {4}
$$
It follows the Mean Squared Error (MSE) between the prediction and target representation, where the predictor,  $g_{p}(\cdot)$ , is a multi-layer perceptron (MLP). Besides the cross-scale prediction loss, the decoder reconstructs images of different scales. The reconstruction loss is thus defined as:
$$
\mathcal {L} _ {r e} = \frac {1}{N} \sum_ {k = 1} ^ {N} \left(\left| \left| p _ {l} - \tilde {p} _ {l} \right| \right| _ {2} ^ {2} + \left| \left| p _ {h} - \tilde {p} _ {h} \right| \right| _ {2} ^ {2}\right) \tag {5}
$$
where  $\tilde{p}_l = D(E(p_l))$  and  $\tilde{p}_h = D(E(p_h))$ . Via the cross-scale prediction loss and the reconstruction loss, the consistency and effectiveness of the semantic information of the representation can be better preserved.
The total loss of the Cross-Scale MAE is the sum of cross-scale consistency loss at the encoder  $(\mathcal{L}_{cc})$ , cross-scale prediction loss at the decoder  $(\mathcal{L}_{cp})$ , and the reconstruction loss  $(\mathcal{L}_{re})$ , defined as:
$$
\mathcal {L} = \mathcal {L} _ {c c} + \mathcal {L} _ {c p} + \mathcal {L} _ {r e} \tag {6}
$$
We investigate the quality of representations learned from Cross-Scale MAE pre-training through comparisons to state-of-the-art approaches (Sec. 4.1) and comprehensive ablation studies (Sec. 4.2). Sec. 4.3, we briefly explore their robustness to scale variation and transfer performance to additional tasks. We use SatMAE [12] as the baseline, the state-of-the-art MAE-based approach for remote sensing image analysis. We pre-train Cross-Scale MAE with a ViT-Large model (unless mentioned otherwise) using the Functional Map of the World (fMoW) [11] RGB training set, which consists of  $363.6\mathrm{k}$  images of varying image resolution and GSD. In Sec. 4.1, we present experiments and results to validate the effectiveness of the proposed method, compared with baseline and other state-of-art methods, such as Scale-MAE [36]. Section 4.2 presents a set of ablation experiments to show how different losses and hyper-parameters affect the performance of Cross-Scale MAE. Section 4.3 briefly describes the efficient backbone built through xFormers.
4.1 Comparison with State-of-the-Art
Similar to Scale-MAE, we use both the K-nearest neighbors (KNN) classification accuracy and performance in downstream tasks as metrics to evaluate the quality of the representation learned by Cross-Scale MAE. Specifically, we pre-train the Cross-Scale MAE on the fMoW-RGB dataset and freeze the encoder as a representation generator; then, we test its effectiveness on different datasets via KNN classification.