dataesr/scientific-paragraphs-categorization huggingface dataset:
As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.
Introduction The supervised deep-learning-based state-of-the-art methods in computer vision often rely on large amounts of annotated images in order to learn relevant image features. However, big datasets are very time-consuming and labor-intensive to annotate. One of the biggest annotated image recognition datasets is ImageNet  [1] , with more than 14 million training images, and it has taken several human years to annotate. As a practical approach in many vision-based applied fields, exploiting supervised models pre-trained on ImageNet is a common way to boost the performance of deep neural networks when performing transfer learning or fine-tuning on smaller domain-specific image data. Regarding the concept of transfer learning, using pre-trained ImageNet weights can improve performance over randomly initializing the network weights (i.
The objective of this paper is therefore to review SSL methods which have been developed to tackle scene classification within the last few years in order to provide a guidance to researchers interested in this potential research topic in remote sensing. This paper is organized as follows. In the next section, we briefly present the main approaches in remote sensing scene classification from classical feature engineering to modern deep learning approaches. Then, we provide the main concepts of the most significant self-supervised methods that have been developed in computer vision and have inspired the remote sensing community. In Section 3, we provide a detailed survey of the SSL approaches that have been developed to tackle scene classification task. Section 4 is dedicated to our experimental study where we conduct several experiments to benchmark and compare current state-of-the-art SSL frameworks on two public scene classification datasets.
Background 2. 1. Scene Classification Scene classification in remote sensing is the task of predicting a label given an image from a dataset with different land-cover semantic categories. As scenes often share lowlevel visual similarities and objects, methods modeling only the pixel or object level alone have not been sufficient to perform well on the scene classification task. Indeed, it requires higher levels of understanding and characterization of the relation among objects and regions within each type of scenes  [3]. As an example, both residential and industrial scenes might contain manmade structures, roads, and trees. Thus, scene classification approaches should be able to characterize coarse-to-fine features from the image, as well as take into consideration the spatial appearance and relation among these semantic elements.
As a result, many (or even most) of the state-of-the-art methods for scene classification are now based on deep neural networks  [6]. Similar to many vision-based applied domains, such as industrial and medical imaging that work on image data structures, deep-learning-based remote sensing scene classifications have been mostly based on convolutional neural networks (CNNs)  [11]. CNNs extract features from the local neighborhood in the image using a set of shared weights for each convolutional kernel. The first layers of a CNN have been shown to extract low-level features while deeper layers create object level features when trained on image classification. To obtain an image-level representation, the spatial features are then aggregated (i. e. , pooled) to be processed by a fully connected layer, which produces a score for each semantic class present in the training dataset.
Recently, using transformers to model images as a sequence of visual tokens  [12]  has led to promising improvements in image classification. A practical strategy for scene classification is to use pre-trained weights from a model trained on big datasets for initialization rather than training from scratch (i. e. , with random initialization). Indeed, the authors in  [2]  proved that using pre-trained weights from networks trained on ImageNet  [1]  already improves the classification performance even though the target dataset does not share the same visual features as ImageNet. This source of pre-training is referred to as the transfer learning approach. For scene classification, it is reasonable that transfer learning would be even more beneficial if the pre-training stage is performed on a large remote sensing dataset rather than on ImageNet since it helps to provide more meaningful features.
This remark is confirmed by  [13]. Indeed, samples from the ImageNet dataset have very different properties than remote sensing images. Objects to classify are often centered on the image, whereas class-specific features in remote sensing datasets are often present in the entire image (see Figure  1 ). In order to evaluate and compare the performance of scene classification methods, the remote sensing community has made efforts to create and gather a large variety of datasets for benchmarking. These involve diverse datasets ranging from simple threechannel RGB images to multi-spectral, hyperspectral, or time series datasets. We now briefly present some of the most common datasets that were used to benchmark scene classification methods in the literature, with a focus on optical images. One of the earliest and most well-known ones in the literature is the UC-Merced dataset  [14]  which contains 21 classes, each with 100 images of size 256 × 256 pixels and of resolution of 0.
3 m. Due to the need for higher number of images and classes for training, many bigger datasets, such as NWPU-RESISC45  [3] , the Aerial Image Dataset (AID)  [15] , and the RSI-CB  [16] , were created by collecting and extracting from Google Earth. Among them, NWPU-RESISC45 (namely, Resisc-45 in the rest of the paper) is one of the most exploited, which contains more than 31,500 images with high resolution (from 30 m to 0. 2 m) and covers 45 different scene categories. For lower-resolution images collected from open-access data sources, EuroSAT  [17]  and BigEarthNet  [18]  represent the most famous ones for benchmarking. They are both extracted from images acquired by the Sentinel-2  [19]  sensor. EuroSAT is composed of 27,000 small images of size 64 × 64 with a spatial resolution varying from 10 m to 30 m per pixel, and covers 10 scene categories.
Being much bigger, BigEarthNet is one of the largest remote sensing scene classification datasets available, with more than 590,000 samples of size 120 × 120 extracted from Sentinel-2 data. It is composed of 44 classes. For more details about these datasets and others, we refer readers to deeper description and analysis of scene classification benchmarks in the recent review paper in  [6]. While deep learning methods have become the de facto standard to solve remote sensing scene classification problems, the need for more domain-specific labeled data is becoming a limit to scale the performance of scene classification methods. Indeed, it has been shown that the performance of deep learning models can scale with the amount of labeled data and network size  [20]. Since labeling data can be costly, time-consuming, and biased depending on the annotator, the need for less label-hungry methods has pushed the computer vision community to develop unsupervised representation learning methods.
Self-Supervised Methods Since gathering a large annotated dataset for a specific task can require a lot of work, algorithms have been proposed to learn effective image representations without any supervision, namely, unsupervised learning techniques  [21, 22]. When the training objective is created from the data itself, the methods are referred to as self-supervised learning methods. In principle, a feature representation is encoded from an image using a deep neural network trained on a pretext task for which labels are automatically generated without human annotation. The representations which are learned to solve pretext tasks could be later used as a starting point in supervised downstream tasks. In this section, we briefly review the most significant state-of-the-art self-supervised methods, mostly proposed within the machine learning and computer vision communities.
GANs are trained in an adversarial minimax two-player game where one network, called the generator G(•), learns to transform the random noise z ∼ N (0, 1) to synthetic data x which tries to follows the original data distribution. In an adversary approach, a second network, called the discriminator D(•), learns to classify between images from the generator and real one from the original dataset (see Figure  2 ). The output score of the discriminator is 1 when it is confident that the input image is coming from the real data distribution and 0 for images created by the generator. This adversarial objective can be written as min G max D 1 N N ∑ i=1 log(1 -D(G(z i ))) + 1 M M ∑ i=1 log(D(x i )), (1) where z ∼ N (0, 1) is a set of N random noise vectors and x ∼ p data (x) is a set of M real images.
Predictive The second category of SSL methods involves models which are trained to predict the effect of an artificial transformation of the input image. Such an approach is motivated by the intuition that predicting the transformation requires learning relevant characteristics of semantic objects and regions within the image. By pre-training a model to predict the relative position of two image patches, ref. [31]  managed to boost the performance of a model against a random initialization and to move closer to the performance of the initialization with ImageNet pre-trained weights in iconic computer vision datasets. Other possible predictive pretext tasks have been proposed to learn representations. One of them is the image colorization proposed in  [32]. In such an approach, the input image is first converted to its grayscale version.
In Exemplar CNN, the authors proposed several augmentation classes, including cropping, rotation, color jittering, and contrast modification. f (•) h p(•) Predict rotation Rotate Figure 3. In RotNet  [33] , a random rotation is applied to input images and the model is then tasked to classify which rotation was applied. The model is composed of an encoder f whose output representations h are used by a predictor p to classify the random rotation. By fulfilling one of these aforementioned pretext tasks, an SSL model is able to learn indepth representations of image content. However, depending on the pretext task and on the dataset, the network will not necessarily be able to perform well on all downstream tasks. As an example, predicting random rotations of an image would not perform particularly well on a remote sensing dataset, since the orientation of objects is not as strictly important as in object-centric datasets (see Figure  1 ).
The authors thus define the multilevel feature matching loss as follows: feature_match = min g K ∑ k=1 1 N N ∑ i=1 D k (x i ) - 1 M M ∑ i=1 D k (G(z i )) 2 2 , (7) where D k (•) returns the activations for the k-th layer of D (with a total of K layers); N and M are defined as in Equation (  1 ). The complete MARTA GANs objective for optimization is then adopted by adding a second term of perceptual loss, as follows: L MARTA = feature_match + M ∑ i=1 [log(1 -D(G(z i )]. (8) The MARTA GANs method is tested on two datasets: the UC-Merced dataset [14] (cf. Section 2. 1) and the Brazilian coffee scene dataset  [53] , which contains 64 × 64 pixels images taken in the green, red, and near-infrared bands and labeled as containing coffee plantations or not. The results on these two datasets show that MARTA GANs could provide high-quality fake samples from the generator while also provide promising classification performance with respect to a low number of parameters in a semi-supervised approach.
In this way, the network has to induce an image representation by learning the relationship between data channels. The overall training loss is as follows: L Split-Brain = Loss( f A (x A ), x B ) + Loss( f B (x B ), x A ), (9) where Loss(x, y) is the mean of a pixel-wise distance metric between image x and y. The final image representation is obtained by concatenating the output of both encoders in a single discriminative embedding vector, which can be used for downstream tasks. One limitation of this architecture compared to other asymmetric Siamese methods is the need to use two different encoders, even during inference, since each one is trained on different data channels than the other. Based on their experiments on the Resisc-45 and AID datasets using the RGB and LAB color spaces, the authors show that the method could yield competitive results even with few unlabeled training images  [54].
Predictive To perform a comparative study of the different SSL methods applied to remote sensing scene classification with limited samples, the authors in  [56]  conduct experiments using several pretext tasks, including image inpainting, relative position prediction, and instance discrimination (i. e. , instance-wise contrastive approach). Their experiments are performed on three remote sensing benchmark datasets: Resisc-45, AID, and EuroSAT. By using linear classification as downstream task, they show that the instance discrimination (IDSSL) model provides better performance than the two predictive approaches while being less sensitive to the amount of labeled data. Furthermore, when the number of labeled samples is limited (less than 20 per class in this study), the use of IDSSL pre-trained weights could help to boost the classification performance by around 20% to 25% compared to training from scratch or using the MARTA GAN, and around 5% to 10% compared to ImageNet pre-trained weights (by using the Resnet50 backbone).
In this study, the authors also compare the performance of SSL trained and tested on different data domains (i. e. , transfer learning). Results show a significant performance drop in the case of transfer learning with highly different domains, such as training on low-resolution EuroSAT images and tested on high-resolution AID images. We will confirm these observations in our experimental study in Section 4. In  [57] , the authors propose to improve the scene classification performance by using a multitask learning model with a mixup loss function that combines self-supervised and supervised training strategies. To do so, the self-supervised loss adopted by rotation predictive framework  [33]  (Figure  3 ) is combined with the cross-entropy loss for label prediction. This joint training allows the model to learn both features dependent on both classification and rotation.
This is reasonable since understanding the image orientation could help to characterize its semantic content. During inference, image features are built from the mean of the representations with rotations of [0,  90, 180, 270]  degrees, respectively, applied to the image. To combine the two loss terms, a trade-off parameter λ is sampled from the beta distribution β(α, α) such that the loss is a mixup function of its supervised and self-supervised components as follows: L mixup = λ × L supervised + (1 -λ) × L self-supervised. ( 10 ) This method limits the number of hyperparameters to be selected, since adjusting different loss terms can be difficult and resource intensive. Moreover, it exhibits low sensitivity to the choice of the α parameter compared to using an explicit trade-off λ. By performing extensive experiments on several datasets, including UC-Merced, Resisc-45, AID, and WHU-RS19  [58] , the authors conclude that by introducing image rotation, the network could learn more discriminative feature representations from a limited amount of data, thus providing competitive classification results compared to state-of-the-art performance.
Unlike in masked autoencoders  [28] , the masking processing in this work is performed through the temporal dimension of the time series for each data channel, which means that the model requires a time series dataset to learn the representations. L SITS-BERT = 1 N N ∑ i=1 f (Mask(x i )) -x i 2 , ( 11 ) where f (•) is a sequence-to-sequence transformer. Once the model has learned to reconstruct time series data, a sequence level representation is computed by using either a global pooling operation on each time token or by introducing a learned class token. This helps the SITS-BERT model to learn the final spectral-temporal representations related to land over contents from images.
Applying this method requires either having the coordinate location of each image from the dataset or having very-high-resolution images that can be divided into smaller patches. These requirements seem to be hardly satisfied in most of the public remote sensing benchmark datasets. However, they could be fulfilled in some specific applied remote sensing contexts where images are provided with their geo-information to investigate land-cover and land-use applications. In the paper, the method is tested on two specific datasets composed of images from the National Agriculture Imagery Program (NAIP) as well as the Cropland Data Layer (CDL) raster  [63]. They are multi-spectral images with four bands of red, green, blue, and infrared with a spatial resolution of 0. 6 m and 30 m, respectively. The authors show that Tile2Vec outperforms other unsupervised feature extraction methods such as autoencoder, Kmeans, or PCA.
Given an encoder f (•) and a predictor network g(•), their classification loss is formulated as follows: L(x, x + , x -) = log(1 -g( f (x) ⊗ f (x + ))) + log(g( f (x) ⊗ f (x -)), (12) where x, x + , and x -are the anchor, positive, and negative, respectively, and ⊗ is the element-wise product. Another significant contribution of this work is that the weights of the predictor network g(•) are not updated using the gradient of the loss; rather, they are sampled from the centered Gaussian distribution at each epoch. These randomized layers improve the quality of representations provided by the encoder compared to the use of trained parameters or the regular triplet loss. To compare with Tile2Vec, the authors evaluate their proposed method on both NAIP and CDL datasets  [63]. An improvement of around 3% with the random forest classifier on the top of feature representations is achieved compared against Tile2Vec when the number of randomized layers is set to three.
Another study that leverages both predictive and contrastive methods is developed by Ayush et al. in  [66] , namely, a geography-aware SSL method. Similar to the idea proposed in Tile2Vec, the authors propose to make use of the time and location (i. e. , spatiotemporal) metadata sometimes available in certain remote sensing scene classification datasets. In their approach, time and location are used in different ways. First, as in remote sensing time series, a location can have a sequence of images acquired at different timestamps. Each of them represents a view at the same location and, therefore, their representations should be identical across different timestamps. Since the location is known, it can be used as a label in a training objective added to the main self-supervised objective.
In order to learn consistent representations across different timestamps, the momentum contrast (MoCo)  [40]  method and, more specifically, its updated version MoCo-v2 [67] are adopted. As described previously, MoCo uses a momentum encoder with weights being the momentum averaged weights of the main encoder network and it maintains a sample queue to provide more negative samples in a single batch (see Figure  6 ). Secondly, the authors also investigate the use of location metadata for a classification pretext task (i. e. , geolocation classification). Given an image dataset, a set of K location clusters is built by grouping nearby coordinates in the same cluster. For each image, the geolocation pretext task uses the cross-entropy to maximize the prediction score of the right location cluster.
With these two additional terms, the performance of the feature representations used in downstream tasks is increased compared to a direct application of MoCo-v2 method on remote sensing datasets. However, as the conclusion of the paper, the authors claim that the use of geographic cluster classification does not always improve the final performance compared to the use of temporal positives. Similarly, the authors of  [68]  build a large-scale dataset of unlabeled remotely sensed images along with their geographical locations extracted from the Sentinel-2 image database. In order to learn relevant features, they propose to use this temporal information to pre-train an encoder using the SimCLR loss  [39]  in a framework called seasonal contrast (SeCo). Several predictor heads are used on top of the encoder to learn different invariances.
One of the predictors projects the learned features into a latent space which is invariant to both temporal and synthetic augmentations, while two other predictors focus on learning invariances only with regard to temporal augmentations. The encoder is shared between all predictor heads and thus is forced to learn robust image-level representations. The proposed SeCo pre-training method provides better classification performance on the BigEarthNet and EuroSAT datasets compared to regular self-supervised pre-training methods such as MoCo-v2 as well as MoCo-v2 with temporal positives. The authors also conduct experiments using the Onera Satellite Change Detection  [69]  (OSCD) dataset. With 24 satellite images pairs from Sentinel-2 with 10 m resolution, OSCD is a public benchmark to evaluate change detection frameworks.
For change detection, the convolutional activations of each image patch are compared between two timestamps, from which patches with high difference in activations are then classified as changed. Since change detection task is out of the scope of this review, we refer interested readers to the paper for further details. Based on a similar approach, but when the temporal information is unavailable in the studied datasets, ref. [70]  proposes to create positives for a given image by sampling multiple images geographically near the anchor image. Using this sampling strategy, a single positive representation called a smoothed representation can be created by averaging the representations of all nearby positives. The contrastive loss (Equation (  4 )) can then be applied to maximize the distance between the anchor and negatives and to minimize the distance between the anchor and its corresponding smoothed representation computed from the sampled positives.
The authors claim to achieve better performance in most test scenarios than three benchmark methods, which are SimCLR, MoCo-v2, and Tile2Vec, tested on four public datasets: Resisc-45, UC-Merced, EuroSAT, and the CDL, while the pretext task is trained on xView data  [71]. However, they find that when the amount of training data for downstream task is high (as in the cases of Resisc-45 and EuroSAT), the proposed method is not effective and provides even lower performance than training from scratch. We investigate this observation later in the experimental study in the next section. Also inspired by the SimCLR and MoCo-v2 contrastive paradigms, the authors in  [72]  propose to build a general human-like SSL mechanism, namely, TOV (The Original Vision model) which is task-independent and could target several remote sensing downstream tasks such as scene classification, object detection, and semantic segmentation.