Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a pre-training framework based on masked auto encoders for multispectral and multi-temporal satellite images.
strengths_and_weaknesses: The architecture proposed is novel as it addresses the temporal irregularities in satellite images, and builds a masked auto encoder for a variety of purposes such as image classification, segmentation etc. The method has been tested for a variety of benchmark datasets and the results are promising. The authors also acknowledge the large amount of computational resources that are required to build such an architecture. The paper is well written overall. 
questions: NA
limitations: Apart from poverty prediction, infrastructure development, and population estimation, most models for satellite imagery are developed to estimate environmental changes happening across the globe. 

As you mentioned in the Conclusion - your model requires expensive computational resources. Although you also mention that you will be working in the future to make the model more efficient, if you could also comment on the carbon footprint generated from your model (or any future work that cites and uses your current model) vs its usability for assessing environmental impacts/climate change/green house gas emissions, that would be great! It would also add a bit broader impact, and trade-off analysis.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed to apply the idea of masked autoencoder to satellite images. The authors proposed to directly handle temporal or spectral sequence of satellite images using a plain vision transformer model with extra encodings. Different masking strategies have been explored. The authors also repurposed existing datasets for multi-spectral satellite imagery.
strengths_and_weaknesses: Pros:
1. The idea of masked autoencoder for satellite images is intuitive and straightforward.
2. Strong performance is shown in the empirical evaluations.
3. The paper is well presented.

Cons:
1. The main concern is on the insufficient ablation studies. The insights from existing ablation studies are actually limited.
a. Although the authors explored several techniques proposed for satellite images, the relative gains are actually smaller than using test-time augmentation from existing work.
b. From a subjective view, the distribution of pixels on satellite images is probably quite different from natural images. It is unclear whether the choice on mask ratio and the design of the decoder has similar impact on the final performance, which is actually the case even for MAE and VideoMAE. However, these important ablations that were explored in existing MAE works are not covered.
c. The selection of the size of the backbone is also limited, where only ViT-Large is used and it makes not directly comparable to existing results.
d. Moreover, justification on the choice of resolution, number of epochs and other hyper-parameters could be also of great value for future research.

2. An important baseline needs more explanation and exploration. It is not clear whether ImageNet Initialization is obtained from supervised pretraining or MAE. Actually both of them should have been evaluated to understand the contribution of training masked auto-encoder on the same domain of data.
questions: Please see the cons for details.
limitations: More discussion on the possible sub-optimal designs in the current method should have been provided.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new masked autoencoder, namely SatMAE, to pretrain for temporal or multi-spectral satellite imagery. The method aims to reconstruct the masked parts of the input under a high masking rate. More specifically, the paper presents several simple yet useful masking strategies and temporal/spectral encoding schemes based on the properties of remote sensing images. The proposed SatMAE is systematically on several challenging benchmark datasets, showing state-of-the-art results. The authors also create a new dataset based on the fMoW RGB dataset. Overall, the proposed architecture is simple but seems effective for RS data and the paper is well-presented.
strengths_and_weaknesses: Strengths:
1. The paper presents a new self-supervised pertaining architecture for RS data. The proposed method is simple but effective.
2. The performance is evaluated on challenging datasets, and the experiment results are sufficient and promising. 
3. A new dataset is created.

Weaknesses:
The core contribution of the paper is to extend MAE to temporal or multi-spectral satellite imagery and design several new masking and positional encoding strategies accordingly. From this perspective, the novelty of the proposed method is a bit incremental.

questions: 1. As the most important baseline,  why didn't the standard MAE be compared in the experiment? Especially, in the ablation study,  the standard MAE should be compared as the baseline in Table 4. This is important to indicate the effectiveness of the proposed strategies. 
2. To indicate the superiority of the proposed method, different closely-related SSL architectures could be compared, for example, SimCLR, MoCo, DINO, and recent masking prediction models (e.g. MaskFeat). 
3. The temporal/spectral encoding is unclear. More explanation is needed. For example, what are the differences between positional embedding and temporal/spectral encoding, and how are they encoded with patch embedding?
limitations: The authors provided a suitable discussion about the broader impact of the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present a pretraining method for satellite images based on masked autoencoders. They perform several experiments on different datasets and tasks, and propose a new dataset combining Sentinel 2 time-series and existing land use labels. The proposed method improves the results of their backbone network.
strengths_and_weaknesses: Strengths:

+ multiple experiments on several datasets and tasks
+ Curated a new dataset from existing data/annotations
+ An interesting to setting not too well explored in ML/CV yet
+ Use of modern tools not too explored on remote sensing data

Weaknesses:

- method and experimental settings are somewhat unclear: what is used for pretraining and fine-tuning, etc
- some design choices are not well motivated (eg. spectral channel grouping)
- results show that many of the ideas presented have negative effects on the performance, yet this is not addressed
- no ablation study on the design of the temporal position encoding
- the authors completely ignore the entire corpus of SITS classification. Claims of state of the art when the baseline is spectral-temporal concatenation are not very convincing

Overall:
a high-effort paper with many experiments on an interesting subject with overall promising results. However, the article completely ignores the literature on SITS, and some design choices seem actually to decrease the performance.

Missing Refs: all these papers present stronger baselines for SITS analysis. Claim of SOTA should include running some of these appraoches, or explanation why they cannot be applied here:

Rustowicz etal "Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods." CVPR Workshops, 2019

Martinez etal "Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences." ISPRS Journal of Photogrammetry and Remote Sensing, 2021.

Rußwurm etal. "Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery." NeurIPS Workshops, 2018.

Garnot etal "Panoptic segmentation of satellite image time series with convolutional temporal attention networks." ICCV, 2021

Shi etal "Convolutional LSTM network: A machine learning approach for precipitation nowcasting." In NeurIPS, 2015.


Details:

Fig1 is unclear and hard to read without legend / detailed caption
In particular, are the features greyed out and not the input patches?

Do not mention nor cite the large body of work on SITS, eg:

It is unclear how the temporal patches deal with uneven temporal samplings.
Motivation for spectral grouping? 

Why would an MLP not be expressive enough to analyze the correlated spectral information. 13 bands is not that many, and it is standard practice. 

Notation: j used for spatial and spectral indices...

No equation number makes it hard to refer to precise

linesL188 not clear at all, i j are spatial indices; how do they relate to spectral indices?

L190: what is i here?

Flow Sentinel: size of images?

Table2: resnet 50 is a very weak baseline for SITS classification.

Use dedicated networksTab3,7: Group works worse than stacking...

Tab4: temporal encoding worsen prediction... Yet the authors say the opposite 

L234The experimental setting is very unclear: how much data are used for fine-tuning? All of it?
questions: - please explain why channel grouping is important in spectral space, and comment on the results
- please compare the results with actually dedicated architecture, not spactro-temporal stacked ResNets
- please give additional details on the fine-tuning
limitations: N/A
review_title: null

Meta-review: metareview: Four experts in the field reviewed the paper and recommended Accept, Borderline Reject, Weak Accept, and Weak Accept. According to the reviews, using MAE in satellite imagery is straightforward, but the novelty lies in details, such as using extra tokens to handle temporal consistency, masking strategies, etc. Another major question from the reviewers was about the ablation studies, and the rebuttal addressed it well. Hence, the decision is to recommend the paper for acceptance. We encourage the authors to consider the reviewers' comments and make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!

confidence: Certain
Decision: Accept