SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
Authors: Yezhen Cong
Samar Khanna
Chenlin Meng
Patrick Liu
Erik Rozi
Yutong He
Marshall Burke
David B. Lobell
Stefano Ermon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Self-supervised learning
Masked Autoencoder
Satellite imagery

Abstract: Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information,  we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/

SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
Yezhen Cong" yzcongdstanford. edu
Samar Khanna"
Chenlin Meng
Patrick Liu
samar www.nna@stanoord......o edu
Erik Rozi
Yutong He
Marshall Burke
David B. Lobell
Stefano Ermon
Stanford University
Abstract
Unsupervised pre-training methods for large vision models have shown to enhance performance downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectre data as groups of bands with distinct spectral positional encodings beneficial. Our approach yields strong improvements over previous state-of tatte-of-thh-art techniques. both in terms of supervised learning performance on benchmark datasets (up 7%), and transfer learning performance downstream remote sensing tasks, including land cover classification (up to f 14%) and semantic segmentation. Code and data are available the project website: https: https::/sustainlabb wwwpraainnnaggruuuuooom group gi thub. Meriiii//atii SStMAE//
1 Introduction
In recent years, self-supervised learning techniques have quickly become the norm for pre-training
models large-scale natural image datasets M
performance on downstream tasks including image classification and have demonstrated strong
image segmentation [3]
111, representation learning I22 [3] [14], image compression 12 L5
image reconstruction [1]. and
image generation [6]. Unlike supervised learning approaches, self-supervised learning techniques do require human labeling, making them appealing in settings where unlabeled data abundant but labeled data are scarce, such as remote sensing data (c.g., satellite imagery). While several large-scale satellite image datasets have been carefully curated in the past few years, including Functional Map of the World (fMoW) [77 Bigharth 18] x View 19) SpaceNet [00], annotating these datasets requires specialized skills and is more expensive than traditional computer vision datasets. Moreover, automatic analysis of satellite imagery is often needed for tasks with large societal impact such as poverty or crop yield prediction 21 2 2
20 30 where acquiring large amounts of labeled data through surveys is impossible or prohibitively expensive. This suggests that self-supervised learning approaches for satellite imagery could be especially valuable.
25 26 2
are mainly designed for natural images. As opposed natural images such as ImageNet RI satellite imagery is usually associated
However, existing self-supervised learning approaches 2
'Equal contribution. Order determined via coin flip.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Postional encoding spatal tempora/spectral dimorsion
tempor of Speecs data
encoder
decoder
Input
reconstruction
patch embed mask pos
decoder
Figure 1: With carefully-designed masking strategies across mutli-spectral and temporal images, and temporal and spectral positional encodings, SatMAE serves powerful SSL vision learner for remote sensing tasks.
with meaningful geographical and temporal information, and can consist of multiple spectral bands representing sensor readings besides visible light (i.c., RGB channels typical in natural images). Depending the data source, satellite imagery
also vary significantly in resolution [32 3k
While self-supervised learning methods for satellite imagery exist (34 35 these approaches cannot learn general representations for both temporal and multi-spectral remote sensing data.
To address this issue we propose SatMAE, a self-supervised learning framework based on masked autoencoders (MAEs) which naturally handles temporal and multi-spectral input data. We show that introducing a positional encoding for the temporal/spectral dimension and independently masking patches across the temporal/spectral dimension benefits pre-trat pre-training. allowing the model to learn representations of the data that are more conducive to finetuning. Specifically, our contributions are:
1. We propose a novel method to leverage temporal or multi-spectral information in satellite imagery improve sule-super pre-training with masked autoencoders (see
2. We introduce MoW-Sentined new Sentinel-2 dataset cross-referenced with fMoW, as benchmark for training models on multi-spectral satellite imagery (see 5.T)
3. We demonstrate the effectiveness of pre-training transformers 33 satellite imagery, achieving significant improvement over previous state-of-the- methods on benchmark datasets well as downstream remote sensing tasks (see 5
2 Related Work
ML for SITS Deep learning has been used for many Satellite Image Time Series (SITS) supervised-
learning tasks such as crop-type mapping 29 28
37
58
yield prediction 39 40 understanding the
economy 31 22
precipitation forcasting 45 and land-cover classification 46
48
These works establish the usefulness of tailoring architectures such as LSTMs, self-attention, and transformers to temporal data. However, outside of their specific task, they are often not directly applicable to other remote-sensing datasets.
SSL for Satellite Imagery Self-supervised learning approach in remote sensing domains. For instance, 34] and 35
has emerged as promising propose incorporating spatially aligned images over time for contrastive self-supervised learning. Despite promising results, these two contrastive learning approaches rely heavily on the quality of positive pairs, which is often hard to control. 49 combines different sensor channels to generate co-located images that serve as
positive pairs. [50 51 2
apply off-the-shelf contrastive learning algorithms to satellite images. (22 utilizes image inpainting and transformation prediction as additional pretext tasks. 5K leverages geographical knowledge to aid SSL, which, however, can be difficult to obtain annotations.

---

Masked Autoencoder MAE 1 is a recent powerful self- superviee learning method. Instead of constructing contrastive objective, proposes the pretext task of reconstructing masked patches of the input, and largely avoids the need for designing specific data augmentation. Inspired by MAE's state-of-the-ar performance on a wide collection vision benchmarks (. many follow-up works extend MAE to different data modalities. VideoMAE
[34]
proposes video tube masking
and reconstruction as a pretext task for video analysis. GMAE 155 adapts MAE to the domain of graphs. MultiMAE [56 takes optional inputs of different modalities and accordingly includes other training objectives to facilitate multi-modality learning. However, these works fail to optimally handle temporal and multi-spectric spectral input. VideoMAE requires squally-spaces image frames in the temporal dimension, which is not the case for satellite data given the temporal irregularity and discontinuity in sampling images of a location. In this work, we incorporate temporal and spectral information into masked autoencoder architecture, and propose a novel self-supervised framework for satellite data.
3 Background
Masked Autoencoder The MAE is an autoencoder with asymmetrical encoding and decoding stages [1] It operates on images I RCxH*W where H. W the height and width of the image, respectively, and C' is the number of channels. The input image I resized to a sequence of non-overlapping patches. S E g2xxP9 where P is the height and width of the patch, and L= (H/P)- (W/P) is the number of patches. Each patch is passed through a patch embedding fp RP + R to create a sequence S' E R D of embedded patch "tokens". A fraction pm of the L tokens are masked and only the remaining (1 11-pu pmm) L "visible" patch tokens are fed to the encoder, a Vision Transformer (ViT) 36 with positional embeddings to capture the spatial location of the patch in the image. The decoder is series of transformer blocks that operates all L tokens (with positional embeddings added), where the LmL encoded visible patches are placed in their original sequence position among (1 Pm) masked patches represented by a learnable mask token. The decoder outputs a reconstructed image i E RCx H xW which compared to the original image using the mean-squared error (MSE) loss, computed per-pixel only on the masked patches (.
Positional encoding Positional encoding allows transformers to make their learned representations the positional encoding is:
position-awand In MAE and in many transformers 57 58
k
k
Encode(k: 21) sin
Encode (, 2i + 1) COS
(1)
22
Ω
a
Here, A is the position, is the index of feature dimension in the encoding, d is the number of possible positions, and Q2 is a large constant (normally set 10000). MAE, position is defined as the index of the patch along the generated by concatenating the encodings of the x and y coordinates.
axes. Therefore, & ranges from 0 to H P (or W)P). The final encoding
4
Method
In this section, we describe SatMAE with temporal 44. and multi-spectral (4.2) satellite images.
4.1
Temporal SatMAE
We now consider input tensors IT E IBTXCXHx where T denotes the number of images in temporal sequence. In video data, T frames are usually equally spaced. However, temporal satellite imagery rarely has images at regular intervals. More commonly, several snapshots, or versions, of given location are taken at irregular times. The length and sample frequency of these sequences of satellite images vary drastically over years and across different regions.
Naively, one could reshape Ir to I E TTC effectively concatenating the temporal sequence of images along the spectral (i.e. channel) dimension, and then apply the MAE machinery verbatim. This method poses few difficulties: (i) the model may be unable to generalise to a temporal ordering different to the one used in pre-training, since it can only understand order through the position of images in the backee-timeeeries (ii) the model cannot reason about the length of time separating two consecutive images in time sequence, which may be variable when images of location are
3

---

sampled at irregular intervals (iii) the model loses access to temporal ane-grained information in deeper layers. as its only direct exposure encode temporal information is through the initial patch embedding Jp (iv) the model is not temporally-shi invariant (i.e. the model would need to separately learn to detect the same event two different segments of a temporal sequence).
To address these challenges and to avoid losing temporal information, we resize the temporal sequence IT to ST E 1 Py Py where LT L-(T/Pr) Pr) (H PH/PP) (W//))) ()/P)))(()PP))) Pr is the "patch size" in the temporal dimension, and I and are defined in 3 Prior works using transformers for video data suggest using Pf 2, where each "patch" is cube of shape 2 16 x 16
54 59 60 Since our
data has much shorter temporal sequence lengths 17. we let Pr = such that LT = L T. In order to operate on inputs of any temporal order, we re-use the same patch embedding fp IR H R for each image in the time series, giving us an embedded sequence of tokens ST- E RRDDD
4.1.1 Temporal Encoding
For each embedded token in the L1 length sequence, need to ensure the model retains information about its spatial and temporal position. As shown in many prior works 34 35). the timestamp of a satellite image is useful for many pre-training or downstream vision tasks. We propose temporal encoding scheme compatible with the masked autoencoder architecture by treating the temporal dimension similarly to the positional dimensions (see 3
D
fp
+
ys, mo, hr
(x, y)
Bnoode
(year, month, bour)
D
The timestamp of satellite image is represented as "year- month-day-hu saaty--hurmmiie minute--econnett entire numerized timestamp into feature encoder, propose only keeping the useful parts. Intuitively, the day, minute, and second should be unrelated to the visual ap- pearance of a region. Thus, including these components in the temporal encoding may not be beneficial, and can be detrimental. In contrast, a landscape may evolve over years due weather, geology, and human activity. The month reflects season and climate, and the hour reflects daylight and temperature.
Instead of passing the
fpj
+
(x, y) (spoctral group)
Bacode
group
Figure 2: Top: Encoding each temporal patch with a shared patch embedding fp. Bottom: Encoding each spectral patch with different patch embedding fp, for each group j.
Then, the temporal encoding is formulated as:
l = CONCAT [Encode (ayyyaarr))).) i), Encode (kimnth i). Encode(khour i)]
(2)
And the final encoding generated by concatenating the temporal encoding to the positional encoding defined in 3 such that the total length of the encoding is D.
4.1.2 Masking Strategies
With an additional temporal dimension, masking a subset of the Lr tokens needs to be treated with care. As seen in figure 3 there are different ways to mask a temporal series of satellite images.
Consistent Masking across all images (fig. 3a). This approach is also used VideoMAE 54 with video input.
Each image is "patchified" separately, but the masked regions are consistent
Independent Masking Each image is "patchified" separately, and masked regions may not be the same across every image. Instead, a fraction Pm of the full sequence of all patch tokens are masked. Another variant is to independently mask the regions of each image, but keep the ratio Pm of masked regions fixed per image. Both variants are equivalent expectation. Effectively, the model may look at unmasked values of a region that is masked in one image but not in others. This setting may lead to an easier task for video data since the model can "cheat" and exploit temporal redundancy in videos with high framerates [54]. However, we argue that this form of "cheating" is less feasible in temporal satellite imagery, given the strong impact of seasonal variation and changing human activity over
periods of time and the much larger time deltas between temporally consecutive images (see fig. Ba).

---

REBB
2 mo 9-10
Infrorad
y-1 m-3.m... x10
Mosture
Original time-series
Consistant Masking Indepandent Masking
Multi-spectrsl imput
Consistent Maeking
Indeper dent Masking
(a) Temporal data
(h) Spectral data
Temporal masking: For images in timeseries, can choose to keep patch fully visible fully masked across time (consistent masking), independently mask all patches (independent masking). In both cases, a fraction Pou patches are masked. Here, T 3, and the leftmost column orders the temporal sequence according to the timestamp features. For example, "y-12, m- 12, h-15" is 12 years from the minimum year (2002), the zero- sero-indexed month 2, and the 15th hour of the day: i.e. roughly 2014, March, 15:00. Pb Spectraa Masking: The same masking strategies adapted groups of the 13 spectral bands in Sentinel-2 images.
Figure 3: 3a
Independent Masking Inconsistent Cropping During data pre-processing, we can crop square regions for input inconsistently that images in the same temporal sequence may be spatially- unaligned. This strategy may help the model learn better representations as may learn to align images in the sequence across the spatial and temporal dimensions.
4.2 Multi-spectral SatMAE
While MAE does operate on images / RCxHXW usually C' for RGB images. Satellite data. on the other hand, can often have multiple spectral bands. For example, Sentinel-2 imagery has C = 13 bands of 10m, 20m and 60m spatial resolution, each of different wavelengths (see A.2.2). Below, we discuss and later experimentally compare various ways to encode spectral information.
Stack Channels
The sequence of patches S e RLXPºC is embedded to sequence of tokens S' 144D thus treating the multi-band image as is. We denote this method SatMAE+Stack.
There are limitations to naively stacking the spectral information, especially that a single convolutional patch embedding may insufficient to fully capture fine- fineggained information present in multiple bands of different wavelengths and spatial resolution. We would like the model to preserve information about the different bands through the encoding and decoding stages.
Group Channels
To address this limitation, we propose grouping subsets of spectral bands. Given C' channels, we form G groups 91. -2,2..... IG such that 91 + + 9G C. This is analogous slicing the image I in the channel dimension, creating images I,,... Ig. where I, E H We use separate patch embedding fr, g g; + RD for each group j. thus allowing the model to best represent each possibly different group of channels as token embeddings. Therefore, each group is first resized from I H to S3 E RRxP'', and then each patch is embedded with fny to produce sequence of embedded tokens S E LRLDD The sequences "iy...,'' concatenated to produce the final set of tokens S' GLLDD
Spectral Encoding Since the tokens in S' correspond to patch location (m, n) in the input image and group of channels 9j. we include an encoding for the group index ky similar 1.1
gkg. Encode
(3)
Note that this encoding simply depends on devr-de channel grouping, and differs from eq. ! since additional metadata for the imagery, like its date, is not needed. The final encoding
5

---

concatenation of the positional Ik,i yk, and the spectral encoding Ik.; such that the total dimension D (see fig. 2. This positional encoding is added S' before inputting it to the encoder. We denote the combined setting of grouping channels and using a group encoding as SatMAEFGroup.
Masking Strategies We consider consistent masking (denoted SatMA BaMMAE+Group-MM and inde-
pendent masking (SaMMAE+Group+IMM) as defined in section 4.1.2 and as visualized in fig. 3b
5 Experiments
In this section, we first introduce the datasets we considered, including a new multi- spectral remote sensing image dataset for downstream task evaluation (5.T. We then present our results on benchmark datasets (..
and various remote sensing transfer-learning and downstream tasks 5.5 For all experiments, we compare with the current state-of-the-art methods Kel 55 and with supervised learning from scratch using the VIT backbone of SatMAE. In summary, our approach demonstrates strong performance on all the tasks we considered, yielding improvements over previous state-of-the- art techniques by up to 6% on supervised learning benchmarks, and up to 14% on remote sensing transfer-learning downstream remote sensing tasks.
5.1 Datasets for Pre-training
Functional Map of the World (fMoW) 1 is dataset of high-resolution satellite image time series across the world, with a task of classification among 62 categories.
fMMoW RGB
fMoW Sentinel We create new dataset based the fMoW RGB dataset. We collect all 13 frequency bands provided by Sentinel-2 (B1-12 and B8A) for the original IMoW locations, at some of the same times as fMoW images plus some extra times, for a total of 7 72,8774 training images, 84,939 validation images, and 84,966 test images. More details are included in appendix A.
5.2
IMoW RGB (non-temporal)
In this section, we perform experiments on fMoW single image classification task. Fol- lowing 34,, we report both the performance of linear probing and finetuning setting. Table μ shows that compared to the previous state- of-the-art self-supervised method using con- trastive momentum encoding approach [+ K our SatMAE achieved 6.29% improvement in top classification accuracy. Interestingly, with- out SatMAE pre-training the ViT-large model could only reach 62.48% at convergence after 50 epochs of finetuning compared to 69.05% achieved by training a ResNet-50 model from seratch. This is likely because the ViT (36 back- bone is harder to finetune from scratch than ResNet50 611. which makes the pre-trained model more valuable.
Method
Backbone
Frozen/Finetune
Sup.* *
ResNe150
-/69.05
Sup.t
ResNet50
-/69.07
GASSL
34
ResNe150
68.32/71.55
Sup.* *
ViT-Large
-/62.48
Sup.t
Varge
-/75.70
Sup.
ViT-Large
-/76.91
SatMAE
ViT-Large
65.94/77.84
Table 1:
Top 1 Accuracy IMoW classification. Frozen: only performing linear classification frozen features the pre-trained model. Finetune: end-to-end finetuning the whole model. is training from scratch, and t is using supervised- -eaarnnng ImageNet weights, and 2 is SSL MAE ImageNet weights.
5.3 iMoW RGB (temporal)
Main experiments We perform image-sequence classification on the temporal version of fMoW RGB to evaluate our temporal SatMAE, The temporal fMoo consists of co-located image sequences with length of As seen in table 2 SatMAE surpasses the previous saae-oo-tthe-ar by 4.48% and improves the non-temporal result by 2.06% in top classification accuracy. We also outperform UTAE 48 a SITS state-of-the-art, by 18%. We can observe from rows 5-8 that this gain not from the larger model to handle sequences of data. Naively stacking the image sequences in the channel dimension performs even worse than the non-temporal SatMAF. Again, SatMAE pre-training crucial for ViT to outperform ResNet50. Training details are in appendix A.3.2

---

Method
Backbone
Top Acc. (1/5)
Method
Backbone
Top Acc. (1/5)
Sup. Learning*
ResNet152
49.12/75.73
Sup.* *
ResNet50
73.24/-
Sup. Learningt
ResNet152
54.46/78.99
ScCo 335
ResNet50
66.80/-
MoCo-v3
ViT-Base
50.45/76.37
GASSL 13
ResNe150
74.11/-
MoCo-v3+Grrou
ViT-Base
51.33/75.68
+
ViT-Large
53,03/37.14
UTAE 48
U-Net
61.59/86.45
SaIMAE+Groupi
ViT-Large
51.61/77.26
Sup. x
ViT-Large
61.89/84.23
SaIMAE+Groupp
Vir-Large
47.57/72.26
SatMAE+Stack
Vir-Large
75.85788.68
SaLMAE+Groupl
ViT-Large
49.49/76.30
MAF+Test Aug.
ViT-Large
78.90/93.31
SatMAE+Stack
ViT-Large
57.37/81.63
MAE]
ViT-Large
76.78/92.01
SaMMAF+Grup--IN
Vir-Large
59.30/82.81
SatMAE
ViT-Large
81.49/93.26
SaLMAE+GroupLLM
ViT-Large 61.48/85.17
Table 2:
Classification results on the temporal [MoW RGB dataset. * means finetuning from scratch. means copying the input image times instead of using tem- poral sequences as input. SatMAE+Stack here means stacking the image sequence along the channel space.
Table 3: Top 1 & Top 5 Accuracy on the fMoW Sen- tinel validation set. The different initializations are: " from scratch, MAE ImageNet weights, supervised ImageNet weights, & SatMAE fMoW RGB weights. Other rows use fMoW Sentinel for pre-training. The last row includes additional data augmentations (.4
Group
Indp.
Spec. Enc.
Temp.
Indep.
Cons. Crop.
Test Aug.
Back.
Top I Acc.
Top 1 Acc.
Strat.
Mask.
Enc.
Mask.
Base
X
59.11
78.07
Large
X
58.87
78.45
Large
X
57.76
79.90
Large
H
V
57.78
79.69
Large
R
58.76
81.49
Large
X
59.30
Table 4: Ablation studies on different components of temporal SatMAE on the temporal IMoW classification task. The first column is whether using temporal encod- ing, the second whether using independent masking, the third is whether cropping consistently, and the last one is whether applying test-time augmentation.
Table Ablation studies on spectral SatMAE on fMoW-Sentinel. The first column denotes using ViT- Base or ViT-Large. The second column is the grouping strategy (see 5.4p. 'hee third column denotes indepen- dent consistent masking. The last column is whether the spectral group encoding Rlis used.
Ablation studies Table provides a comprehensive ablation study on the components of temporal SatMAE. We see that improved performance is mainly due to the temporal encoding and adopting independent masking rather than the consistent masking strategy suggested in VideoMAE [54]. Interestingly, consistent cropping slightly decreases performance, indicating that the model does not rely on perfectly apatially-aligned image sequences. In addition, using test-time augmentations
similar to R4! is beneficial. Further ablations on mask ratio Pm and patch size P are in appendix A.4
5.4 IMoW Sentinel (Multi-spectral)
In this section, we pre-train and finetune SatMAE on the image classification task of the fMo Sentinel dataset. We pre-train SatMAE+ 4.2 and investigate SatMAE+Group+ 1.2 and SatMAE+Group-IMM 4.12 (see 4.2p. The full models are then finetuned on the MMWW-Seninel image classification task. For comparison, we also finetune the ResNet-152 model 61 from scratch and from a supervised ImageNet initialization. We pick the largest model, ResNet- 152, for fairer comparison with ViTs. We also include MoCo-v3 62 3. popular SSL method. Given the differences in applying RGB-image augmentations to satellite imagery, we implement two versions: (i) MoCo-v3: we apply all of the same augmentations, except random grayscale and solarize, to create views of the 10-channel image. (ii) MoCo-v3+Group: we split the 10 bands into two groups suggested by 2 and apply augmentations to each to create positive pair of two 5-channel images.
Model configuration As not all of the 13 Sentinel-2 bands may be useful, in our experiments we drop bands B1, B9 and B10, which correspond to spatial resolution of 60m. Of the remaining 10 bands, we form three groups: (i) RGB+NIR: B2, B3, B4, B8 (ii) Red Edge: B5, B6, B7, B8A (iii) SWIR: B11, B12. We choose this grouping to ensure each group has bands of the same spatial resolution and similar wavelength (see A.2. A.6 Only the last row of table 3 includes additional data augmentations used during finetuning
See A.3.3 for pre-training and finetuning details.
7

---

Method
Backbone
Top Acc.
Method
Backbone
mloU
Sup. (Scratch)
ResNer50
54.46
Sup. (Scratch)
ResNet50
75.57
GASSL [34]
ResNer50
57.63
GASSL [34]
ResNe150
78,51
Sup. (Scratch)
Vir-Large
69.65
Sup. (Scratch)
ViT-Large
74.71
SatMAE
VirgL
71.77
SatMAE
ViT-Laarge
78.07
Table 6: NAIP land cover classification results.
Table 7: SpaceNet vl building segmentation results.
Method
Backbone
Top Acc.
Method
Backbone
mAP
Sup. (Scratch)
ResNet18
63.21
Sup. (Scratch)
ResNet50
69.49
Sup. (IN init.)
ResNet18
86.44
Sup. (IN init.)
ResNet50
80.04
GASSL 34
ResNet18
89.51
GASSL 34
ResNet50
80.20
ScCo 35
ResNet18
93.14
SeCo 35
ResNet50
82.62
SatMAEs
ViT-Large
95.74
Sup. (Scratch)
ViT-Large
80.07
SatMAE
Vir-Large
98.94
SatMAE
ViT-Large
82.13
SaIMAE-Grup-IM
ViT-Large
98.98
Table 9: BigEarthNet multi-label classification results. Following (35) we use mean Average Precision (mAP) as the metric, and use newer set of class labels.
Table 8: EuroSAT land cover classification results. means only the RGB channels of the data.
Our method SatMAE+Group+LM achieves the highest accu- racy, outperforming supervised training from scratch (t 6.27%) and ImageNet-initaaizze backbones (t 4.84%). ImageNet initializations may be less useful than in fMoW-RGB given the larger distri- butional shift multi-spectral input data. We also note the effectiveness of grouping channels over processing all bands only at the patch embedding level (i.e. SatMAE+Stack).
Results We present results in table 3
Ablation Studies We investigate the design of SatMAE for multi-spectral data in table For grouping strategy, we implement alternate band groups to test the hypothesis that grouping bands based on wavelength and resolution is beneficial. X represents the band groups in 5.4. H represents splitting the 10 bands into two halves, (22,3,4,56), 17,8,8,,11,12).. R represents a random split into three groups (66,5,11,12), 483,,,8,3), (7.2)), reflecting the same group sizes as X. As seen, the choice of band groups does influence performance, yielding a gain of about 0.6%. Moreover, ViT-Base performs strongly, suggesting that SatMAE is the reason for improved performance rather than the number of parameters in ViT. Interestingly, independent masking performs the best, which prompts the model to "peck" at unmasked band groups to reconstruct the same region in a masked band group.
We also include further experiments on the length of pre-training (see A.3.3), the impact of mask ratio Pm. and patch size P (see A.5) and the usefulness of the 13 Sentinel-2 spectral bands (see A.6)
5.5 Transfer Learning Experiments
Now, we finetune our pre-trained SatMAE on downstream tasks on remote-sensing datasets, including land cover classification (5.5)
multi-label classification (5.)) and building segmentation (5.
Finetuning details are included in
A.7
A.8
A.9
A.10
Land Cover Classification We perform transfer learning experiments land cover classification using the NAIP and AT 33) dataset. NAIP consists of RGB+CIR images of 66 land cover classes obtained by the USDA's National Agricultural Imagery Program, which are split into 244,471 training and 55,529 validation images. Eu is a small dataset containing 27,000 13-band satellite images of 10 classes based on Sentinel-2. We follow
35 64
for the train/val splits on EuroSAT.
Table 6 and table X shows the remarkable improvement of our SatMAE over the state-of-the-arts Although using the ViT- backbone already achieved good results, initializing the model with SAT-MAE pre-trained weights further increased the accuracy by 2%.3%.
Multi-label Classification We also use the BigHarth [18 dataset for multi-label classification, which consists of I 33-band Sentinel-2 images of 19 classes total. There are 354, 196 images for training and 118,065 images for validation. Following 35 we use 10% subset of the train set.
8

---

T shows SatMAE pre-traie tre-training improves upon the model trained from scratch by over 2%, and achieves comparable results the state-of-the-are GASSL and SeCo were actually trained on larger pre-train dataset (IM Sentinel-2 images v.8. 713k) and with all 13 bands than our fMoW Sentinel. Therefore we expect further improvement when we pre-train SatMAE with more data and for longer.
Building Segmentation In this section, evaluate SatMAE the semantic segmentation down- stream task of the SpaceNet vl dataset 220] The SpaceNet vI dataset consists of 6940 high resolution satellite images with segmentation masks for buildings, which divided into train and test of 5000 and 1940 images, respectively.
The results in table 7 show that method achieves larger performance gain from supervised learning from scratch compared to [4 The incompatibility of the ViT backbone with PSANet could explain why the baseline performance is not strong that of using ResNet50 backbone.
5.6 Visualizing reconstruction quality for SatMAE
temporal Input
masked Input
reconstruction
Figure Reconstruction quality of SaLMAE+M (left) vs. SaIMAE+CM (right). Further results appendix
We show the visualization of the reconstruction quality of two different SatMAE masking strategies in fig. E SatMAE+IM successfully reconstructs all the airplanes even though their number varies across time. In contrast, the SatMAE with Consistent Masking missed some airplanes in the reconstruction.
Conclusion
In this paper, propose new SSL framework based on the MAE architecture [ tailored to remote- sensing data (satellite imagery). Our novel masking strategy a joint positional, temporal/spectral space, along with the temporal and spectral encoding, enables our model to handle temporal and multi-spectral satellite images as input and Iearn useful representations. Experiments on the datasets for pre-training and multiple downstream datasets demonstrate the effectiveness of our pre-trale pre-trained SatMAE model, outperforming previous state-of-the-art results by large margins.
In the future, it would be useful to design more efficient transformer architectures. While SatMAE has similar number parameters for both the temporal and multi- culti-speetrall setting as a regular ViT. the increased length of token sequences can strain computational resources. Moreover, it is also worth exploring optimal positional encodings for spectral and temporal data, as well as optimal groups of spectral bands, either by neural-based search methods, or using prior knowledge. Lastly, investigating better architectures for object detection and semantic segmentation using ViTs will be important in generalising SatMAE to further downstream tasks.

---

Broader Impact
Accurate measurements of economic, social, and environmental phenomena are key inputs into policy decisions made around the world, but the sparsity of labelled data on many outcomes means that such decisions are often framework could relieve the dependence on labelled data for many downstream tasks that use satellite imagery as input. We hope our SatMAE method will help close the gap between SSL performance on natural imagery and the more challenging satellite imagery, and prompt further attention from the MI. community on the usefulness of SSL satellie-imagery-reeacee tasks.
guided by timely accurate data. We demonstrate how pre-training
Better extraction of information from satellite imagery has profound implications for our ability to measure and understand broad array of social, economic and environmental phenomena that are critical for decision making. Our approach further amplifies the usefulness of the sparse amount of labelled data that exist on key human outcomes, and could enable rapid and accurate extraction of imagery features relevant for critical downstream tasks, including poverty prediction, infrastructure development, and population estimation. Such information could aid governments in more rapid and data-informed decision making and ultimately bring large societal benefits.
7
Acknowledgements
This research based upon work supported in part by the Office of the Director of National Intelli- gence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2021-201 1000004, HAI, NSF(#16511566) AFOSR (FA95501910024) ARO (W911NF-21-1-0125)) and Sloan Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes withtttstg any copyright annotation therein.
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We tried our best to be precise.
(b) Did you describe the limitations of your work? [Yes] We described the limitations in the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discussed in Appendix B
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We list the url to our project website in the abstract. The website will contain links the code and data on Github.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We list part of them in the experiments section and part of them in Appendix A
(c) Did you report eтror bars (e.g., with respect the random seed after rumning ex- periments multiple times)? [No] We trained on large datasets with small variation across different runs expected. Limited by computation resources, we only ran each experiment once.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We list that in Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we do that by citing creators of datasets and authors of prior works.
(b) Did you mention the license of the assets? [Yes] We do that in Appendix A
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We releasing new dataset. The instructions and links will be released in codebase mentioned above.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All data we used are released publicly.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We concluded that no data we are using has such concerns.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
References
[1] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.

---

[2] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer. 2020.
[3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1599-1607. PMLR, 2020.
[5] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 13:21271-21288 2020.
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 15750-15758, 2021.
[7] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. CoRR, abs/201 1.00362, 2020.
[8] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural net- works: A survey. IEEE Iransactions on pattern analysis and machine intelligence, 43(11))4037- 4058, 2020.
[9] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[11] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12275-12284, 2020.
[12] Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison. Lossy compression for lossless prediction. Advances in Neural Information Processing Systems, 34, 2021.
[13] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 1920- 2019.
[14] Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation feature decoupling. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 10364-10374, 2019.
[15] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self- supervised learning by compressing representations. Advances Neural Information Processing Systems, 33: 13:1009--2 2020.
[16] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decodiing models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34:12533-12548, 2021.
[17] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In CVPR, 2018.
[18] Gencer Sumbul, Marcela Charfuelan, Begüm Demir, and Volker Markl. Bigearthnet: A large- scale benchmark archive for remote sensing image understanding. IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pages 5901-5904. IEEE, 2019.
12

---

[19] Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. xview: Objects in context in overhead imagery. arXiv preprint arXiv: 1802.0886, 2018.
[20] Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow. Spacenet: A remote sensing dataset and challenge series. arXiv preprint arXiv: 002.00 2018.
[21] Marshall Burke, Anne Driscoll, David Lobell, and Stefano Ermon. Using satellite imagery to understand and promote sustainable development. Science, (716655))eabb86622 2021.
[22] Kumar Ayush, Burak Uzkent, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Efficient poverty mapping from high resolution remote sensing images. Proc. AAAI Conf. Artif. Intell, volume 35, pages 12-20, 2021.
[23] Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon. Gener- ating interpretable poverty maps using object detection satellite images. arXiv preprint arXiv:2002.01612, 2020.
[24] Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353(600 794, 2016.
[25] Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon. Deep gaussian process for crop yield prediction based on remote sensing data. In Thirty-First AAAI conference on artificial intelligence, 2017.
[26] Anna X Wang, Caelin Tran, Nikhil Desai, David Lobell, and Stefano Ermon. Deep transfer learning for crop yield prediction with remote sensing data. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, pages 1-5, 2018.
[27] Mare Rußwurm and Marco Körner. Self-attention for optical satellite time series classifica- tion. ISPRS journal of photogrammetry and remote sensing, 169:421 -35,, 2020.
[28] Jorge Andres Chamorro Martinez, Laura Elena Cué La Rosa, Raul Queiroz Feitosa, Ieda Del' Arco Sanches, and Patrick Nigri Happ. Fully convolutional recurrent networks for mul- tidate crop recognition from multitemporal image sequences. ISPRS Journal of Photogrummelry and Remole Sensing, 171: 171:188-200 281201, 2021.
[29] Rose M Rustowicz, Robin Cheong, Lijing Wang, Stefano Ermon, Marshall Burke, and David Lobell. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 75-82, 2019.
[30] Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Ji- hyeon Lee, Marshall Burke, David B Lobell, and Stefano Ermon. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning. arXiv preprint 0rXiv:2111.04724 2021.
[31] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[32] Yutong He, Dingjie Wang. Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David Lobell, and Stefano Ermon. Spatial-temporal super-resolution of satellite imagery via conditional pixel synthesis. Advances in Neural Information Processing Systems, 34:27903- 27915, 2021.
[33] Yutong He, William Zhang, Chenlin Meng, Marshall Burke, David B Lobell, and Stefano Ermon. Tracking urbanization developing regions with remote sensing spatial-temporal super-resolution. arXiv preprint wwXiv:200.007336 2022.
[34] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David I Lobell, and Stefano Ermon. Geography-aware self-supervised learning. In Proceedings of the IEFF/VV International Conference Computer Vision, pages 10181-10190 2021.
13

---

[35] Oscar Mañas, Alexandre Lacoste, Xavier Giro-i Nieto, David Vazquez, and Pau Rodriguez. Sea- sonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9414 2021.
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[37] Marc Rußwurm and Marco Körner. Convolutional Istms for cloud-robust segmentation of remote sensing imagery. arXiv preprint D.XXiv:811..02777 2018.
[38] Charlotte Pelletier, Geoffrey I Webb, and François Petitjean. Temporal convolutional neural network for the classification of satellite image time series. Remote Sensing, 11(5):523, 2019.
[39] Elisa Kamir, François Waldner, and Zvi Hochman. Estimating wheat yields australia using climate records, satellite image time series and machine learning methods. ISPRS Journal of Photogrammetry and Remote Sensing, 160: 124-135, 2020.
[40] Raí A Schwalbert, Telmo Amado, Geomar Corassa, Luan Pierre Pott, PV Vara Prasad, and Ignacio A Ciampitti. Satellite-basee based soybean yield forecast: Integrating machine learning and weather data for improving crop yield prediction in southern brazil. Agricultural and Forest Meteorology, 107:1078 2020.
[41] Chenlin Meng. Enci Liu, Willie Neiswanger, Jiaming Song. Marshall Burke, David Lobell, and Stefano Ermon. Is-count: Large-scale object counting from satellite images with covariate- based importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12034-12042, 2022.
[42] Evan Sheehan, Chenlin Meng, Matthew Tan, Burak Uzkent, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Predicting economic development using geolocated wikipedia articles. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2698-2706, 2706, 2019.
[43] Enci Liu, Chenlin Meng, Matthew Kolodner, Eun Jee Sung, Sihang Chen, Marshall Burke, David Lobell, and Stefano Ermon. Building coverage estimation with low-resolution remote sensing imagery. arXiv preprint arXiv:2301.01449) 2023.
[44] Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, and Stefano Ermon. Learning to interpret satellite images using wikipedia. In Proceedings of the Tighty-Eighth International Joint Conference on Artificial Intelligence, 2019.
[45] Xingjian Shi, Zhourong Chen, Hao Wang, Van Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
[46] Andrei Stoian, Vincent Poulain, Jordi Inglada, Victor Poughon, and Dawa Derksen. Land cover maps production with high resolution satellite image time series and convolutional neural networks:
Adaptations and limits for operational systems. Remote Sensing, 11(17):1986, 2019.
[47] Xin Yang and CP Lo. Using a time series of satellite imagery to detect land use and land cover changes in the atlanta, georgia metropolitan area. International Journal of Remote Sensing, 1775-1777 2002.
[48] Vivien Sainte Fare Garnot and Loic Landrieu. Panoptic segmentation of satellite image time series with convolutional temporal attention networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 4872-4881, 2021.
[49] Aidan M Swope, Xander H Rudelis, and Kyle T Story. Representation learning for remote sensing: An unsupervised sensor fusion approach. arXiv preprint 1rXiv:2108.088 050944 2021.
[50] Vladan Stojnic and Vladimir Risojevic. Self-supervised learning of remote sensing scene representations using contrastive multiview coding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1182-1191, 2021.
14

---

[51] Pallavi Jain, Bianca Schoen-Phelang and Robert Ross. Multi-modal self-supervised representa- tion learning for earth observation. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 3241-3244. IEEE, 2021.
[52] Wenyuan Li, Hao Chen, and Zhenwei Shi. Semantic segmentation of remote sensing images with self-supervised multitask representation learning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 14:6438-6455 2021.
[53] Wenyuan Li, Keyan Chen, Hao Chen, and Zhenwei Shi. Geographical knowledge-drives representation learning for remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 2021.
[54] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, arXiv preprint arXiv:2203.12602, 2022.
[55] Hongxu Chen, Sixiao Zhang, and Guandong Xu. Graph masked autoencoder. arXiv preprint vrXiv:2202.08391, 2022.
[56] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. arXiv preprint arXiv:2204.01678 2022.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[59] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In International Conference on Computer Vision (ICCV), 2021.
[60] Ze Liu, Jia Ning, Yue Cao,, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[61] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[62] Xinlei Chen Saining Xie* and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.0205: 022.00 2021.
[63] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[64] Maxim Neumann, Andre Susano Pinto, Xiaohua Zhai and Neil Houlsby. In-domain representa- tion learning for remote sensing. arXiv preprint arXiv: 1911199117777 2019.
[65] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In Proceedings of the European conference on computer vision (ECCV), pages 267-283, 2018.
[66] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. CoRR, ab9/1910.09700,, 2019.
15


Reviews:
[{"review_id": "pbnlDfyFokU", "reviewer": "Reviewer_Wua7", "summary": "The paper introduces a new masked autoencoder, namely SatMAE, to pretrain for temporal or multi-spectral satellite imagery. The method aims to reconstruct the masked parts of the input under a high masking rate. More specifically, the paper presents several simple yet useful masking strategies and temporal/spectral encoding schemes based on the properties of remote sensing images. The proposed SatMAE is systematically on several challenging benchmark datasets, showing state-of-the-art results. The authors also create a new dataset based on the fMoW RGB dataset. Overall, the proposed architecture is simple but seems effective for RS data and the paper is well-presented.", "questions": "1. As the most important baseline,  why didn't the standard MAE be compared in the experiment? Especially, in the ablation study,  the standard MAE should be compared as the baseline in Table 4. This is important to indicate the effectiveness of the proposed strategies. \n2. To indicate the superiority of the proposed method, different closely-related SSL architectures could be compared, for example, SimCLR, MoCo, DINO, and recent masking prediction models (e.g. MaskFeat). \n3. The temporal/spectral encoding is unclear. More explanation is needed. For example, what are the differences between positional embedding and temporal/spectral encoding, and how are they encoded with patch embedding?", "limitations": "The authors provided a suitable discussion about the broader impact of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a new self-supervised pertaining architecture for RS data. The proposed method is simple but effective.\n2. The performance is evaluated on challenging datasets, and the experiment results are sufficient and promising. \n3. A new dataset is created.\n\nWeaknesses:\nThe core contribution of the paper is to extend MAE to temporal or multi-spectral satellite imagery and design several new masking and positional encoding strategies accordingly. From this perspective, the novelty of the proposed method is a bit incremental.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "aINivgqYMyr", "reviewer": "Reviewer_vWmP", "summary": "The authors present a pretraining method for satellite images based on masked autoencoders. They perform several experiments on different datasets and tasks, and propose a new dataset combining Sentinel 2 time-series and existing land use labels. The proposed method improves the results of their backbone network.", "questions": "- please explain why channel grouping is important in spectral space, and comment on the results\n- please compare the results with actually dedicated architecture, not spactro-temporal stacked ResNets\n- please give additional details on the fine-tuning", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ multiple experiments on several datasets and tasks\n+ Curated a new dataset from existing data/annotations\n+ An interesting to setting not too well explored in ML/CV yet\n+ Use of modern tools not too explored on remote sensing data\n\nWeaknesses:\n\n- method and experimental settings are somewhat unclear: what is used for pretraining and fine-tuning, etc\n- some design choices are not well motivated (eg. spectral channel grouping)\n- results show that many of the ideas presented have negative effects on the performance, yet this is not addressed\n- no ablation study on the design of the temporal position encoding\n- the authors completely ignore the entire corpus of SITS classification. Claims of state of the art when the baseline is spectral-temporal concatenation are not very convincing\n\nOverall:\na high-effort paper with many experiments on an interesting subject with overall promising results. However, the article completely ignores the literature on SITS, and some design choices seem actually to decrease the performance.\n\nMissing Refs: all these papers present stronger baselines for SITS analysis. Claim of SOTA should include running some of these appraoches, or explanation why they cannot be applied here:\n\nRustowicz etal \"Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods.\" CVPR Workshops, 2019\n\nMartinez etal \"Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences.\" ISPRS Journal of Photogrammetry and Remote Sensing, 2021.\n\nRußwurm etal. \"Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery.\" NeurIPS Workshops, 2018.\n\nGarnot etal \"Panoptic segmentation of satellite image time series with convolutional temporal attention networks.\" ICCV, 2021\n\nShi etal \"Convolutional LSTM network: A machine learning approach for precipitation nowcasting.\" In NeurIPS, 2015.\n\n\nDetails:\n\nFig1 is unclear and hard to read without legend / detailed caption\nIn particular, are the features greyed out and not the input patches?\n\nDo not mention nor cite the large body of work on SITS, eg:\n\nIt is unclear how the temporal patches deal with uneven temporal samplings.\nMotivation for spectral grouping? \n\nWhy would an MLP not be expressive enough to analyze the correlated spectral information. 13 bands is not that many, and it is standard practice. \n\nNotation: j used for spatial and spectral indices...\n\nNo equation number makes it hard to refer to precise\n\nlinesL188 not clear at all, i j are spatial indices; how do they relate to spectral indices?\n\nL190: what is i here?\n\nFlow Sentinel: size of images?\n\nTable2: resnet 50 is a very weak baseline for SITS classification.\n\nUse dedicated networksTab3,7: Group works worse than stacking...\n\nTab4: temporal encoding worsen prediction... Yet the authors say the opposite \n\nL234The experimental setting is very unclear: how much data are used for fine-tuning? All of it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QcIJN5GUNE", "reviewer": "Reviewer_T3bi", "summary": "In this paper, the authors proposed to apply the idea of masked autoencoder to satellite images. The authors proposed to directly handle temporal or spectral sequence of satellite images using a plain vision transformer model with extra encodings. Different masking strategies have been explored. The authors also repurposed existing datasets for multi-spectral satellite imagery.", "questions": "Please see the cons for details.", "limitations": "More discussion on the possible sub-optimal designs in the current method should have been provided.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of masked autoencoder for satellite images is intuitive and straightforward.\n2. Strong performance is shown in the empirical evaluations.\n3. The paper is well presented.\n\nCons:\n1. The main concern is on the insufficient ablation studies. The insights from existing ablation studies are actually limited.\na. Although the authors explored several techniques proposed for satellite images, the relative gains are actually smaller than using test-time augmentation from existing work.\nb. From a subjective view, the distribution of pixels on satellite images is probably quite different from natural images. It is unclear whether the choice on mask ratio and the design of the decoder has similar impact on the final performance, which is actually the case even for MAE and VideoMAE. However, these important ablations that were explored in existing MAE works are not covered.\nc. The selection of the size of the backbone is also limited, where only ViT-Large is used and it makes not directly comparable to existing results.\nd. Moreover, justification on the choice of resolution, number of epochs and other hyper-parameters could be also of great value for future research.\n\n2. An important baseline needs more explanation and exploration. It is not clear whether ImageNet Initialization is obtained from supervised pretraining or MAE. Actually both of them should have been evaluated to understand the contribution of training masked auto-encoder on the same domain of data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MNdyYavr5b-", "reviewer": "Reviewer_FjEo", "summary": "The paper presents a pre-training framework based on masked auto encoders for multispectral and multi-temporal satellite images.", "questions": "NA", "limitations": "Apart from poverty prediction, infrastructure development, and population estimation, most models for satellite imagery are developed to estimate environmental changes happening across the globe. \n\nAs you mentioned in the Conclusion - your model requires expensive computational resources. Although you also mention that you will be working in the future to make the model more efficient, if you could also comment on the carbon footprint generated from your model (or any future work that cites and uses your current model) vs its usability for assessing environmental impacts/climate change/green house gas emissions, that would be great! It would also add a bit broader impact, and trade-off analysis.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The architecture proposed is novel as it addresses the temporal irregularities in satellite images, and builds a masked auto encoder for a variety of purposes such as image classification, segmentation etc. The method has been tested for a variety of benchmark datasets and the results are promising. The authors also acknowledge the large amount of computational resources that are required to build such an architecture. The paper is well written overall. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept