Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposed cuboid attention as an alternative to regular attention for forecasting Earth system-like data. The paper presents results on a toy N-body MNIST dataset and two real datasets, one on precipitation nowcasting and one one El-Nino Southern Oscillation




strengths_and_weaknesses: Strength:

The novelty in this paper lies in the fact that they use cuboid attention in their transformer architecture. The motivation is that internal variability in climate may not be captured by the inductive biases of CNN/RNN type architectures. However, there is no reason to think that transformer architectures would also capture internal variability just as much as CNN/RNN or for that matter any DL architecture

Weakness:

This paper has several weaknesses, starting from the motivation behind using cuboid attention and baselines. Furthermore, while I found the N-body system's example as a nice toy example to prove that the proposed algorithm has superiority over other baselines, it is actually not that relevant to weather. I detail it further in my comments. 
questions: 1. The motivation behind not using CNN/RNN is that the inductive bias might not capture the internal variability of the climate system. Still, the proposed architecture also has not been studied in terms of why it should.

2. It's important to understand that for problems in physics, a single number like MSE and MAE may not be a good representative of actually how well the algorithm performs in terms of forecasting. For example, in the motivation, the authors talk about how internal climate variability can affect the prediction performance-- that's a solid motivation but is completely absent in the analysis. Does their model capture large scales better or do they capture small scales for a longer period of time? All of these can be very well explored with a Lorenz-type system instead of this MNIST data set. Weather is more than just chaotic--it is multi-scale without any scale separation, unlike the MNIST data

2. The El Nino and nowcasting predictions results need further elaboration. It's hard to say whether it's good or bad. Also, there are good baselines for both these problems. To my knowledge, this paper is SOTA for DL-based ENSO:https://www.nature.com/articles/s41586-019-1559-7. Can the authors compare their model with the proposed model? 

3. For precipitation nowcasting, I believe this paper is SOTA:https://www.nature.com/articles/s41586-021-03854-z. Is it possible to compare the results directly against this paper?

4. I would finally also like to point out, that even if the proposed model is not outperforming some of the baselines it does not necessarily take anything away from the architecture, but the authors need to do some analysis and show the usefulness or show the connection between how this architecture is really useful for physical/earth-system data. In the current form, it's not really there
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the framework of "cuboid attention" for thinking about different spatio-temporal attention mechanisms. This framework includes vanilla self-attention and various lower-complexity alternatives such as axial attention or divided space-time attention as special cases.

Furthermore, this paper proposes Earthformer, a spatio-temporal transformer model with a particular set of cuboid attention blocks. The Earthformer architecture further features a set of global vectors that all cuboids attend to in order to more-readily share information between cuboids.

This paper also proposes a dataset called N-body MNIST (with N=3), where each dataset example is a sequence 64x64 frames consisting of 3 MNIST digits moving according to chaotic 3-body motion. This dataset contains 22,000 examples.

Finally, the authors evaluate the Earthformer architecture against several existing algorithms (UNet, ConvLSTM, PredRNN, PhyDNet, E3D-LSTM, and Rainformer) on several datasets (MovingMNIST, N-body MNIST, SEVIR, and ICAR-ENSO). The results show that Earthformer consistently outperforms the other models, sometimes by a large margin.
strengths_and_weaknesses: ## Strengths

**S1: New N-body MNIST dataset**\
Compared to MovingMNIST, the N-body MNIST dataset intuitively seems more challenging and representative of realistic dynamical systems. I believe that this dataset could well serve as a standard benchmark for future models.

**S2: Comprehensive ablation study on the benefit of global attention vectors**\
I appreciated the large amount of ablation studies showing the significant performance gain with using global attention vectors on the MovingMNIST and N-body MNIST datasets, regardless of the attention variant.

**S3: Generalization of attention via Cuboid Attention framework**\
The cuboid attention framework for thinking about attention mechanisms serves to provide a common language for discussing various forms of attention. In particular, the authors show how axial attention and divided space-time attention are special cases of cuboid attention.

**S4: Overall impressive results**\
The comparison tables and figure really demonstrate the superiority of Earthformer against the baseline models.

## Weaknesses

**W1: Missing some experimental datasets**\
While the authors show good results of Earthformer on MovingMNIST, N-body MNIST, SEVIR, and ICAR-ENSO, I would be interested in seeing experiments on other types of spatio-temporal datasets, such as action-recognition (Kinetics-400/600, SSV2, Diving-48, or YouTube-8M).

**W2: Missing some comparison models**\
While the set of model comparisons is large, to me it seems that some more generic models are missing. For example, how would a standard TimeSformer model (with divided space-time attention) do on the experiments? A comparison against TimeSformer, for example, would really help cement the importance of (1) cuboid attention, and (2) global attention vectors. An appropriate comparison would also be against TimeSformer, but with global attention added to the TimeSformer model.
questions: **Q1**: To me, it was unclear whether the results of baseline models reported in Tables 4-6 were computed by the Earthformer authors, or the original authors of those baseline models. Could the Earthformer authors please clarify?

**Q2**: Will the N-body MNIST dataset be released publicly?

**Q3**: Why call the dataset "N-body MNIST" instead of "3-body MNIST" if you only tried N=3? Given that you only tried N=3, I think it's a bit misleading to write N-body.

**Q4**: You write for cuboid attention on its own, "the cuboids do not communicate with each other." This sentence is a bit vague, and perhaps misleading. If divided space-time attention is applied sequentially (first space, then time), then there is information passing between all cuboids, right?
limitations: To me, there are no clear limitations / negative impact. The authors did not discuss any negative impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a spacetime transformer and apply it to forecasting events in Earth systems. In detail:
- the authors propose Cuboid attention mechanism which splits the input tensor to non-overlapping parts
  - to reduce computational cost by local attention
  - to compute self-attention in parallel
- the authors generalize previous studied video transformer by stacking cuboid attention layers with various hyperparameters
- the authors fulfill the communication gaps of local Cuboid by introducing shared global vectors
- the authors propose an N-body MNIST dataset developed from MINIST
- the proposed Earthformer achieves SOTA performance on both SEVIR and ICAR-ENSO
strengths_and_weaknesses: Strengths:
 - originality: From my knowledge, it is one of the first paper working on predicting ICAR-ENSO using transformer.
 - quality: The authors use both synthetic dataset--N-body MNIST and SEVIR, ICAR-ENSO to proof the significance improvement of their work.
 - clarity: The Cuboid Attention is introduced very clearly with formula and detailed figure illustration
 - significance: The improvement of the performance on the model is significant

Weaknesses:
 - originality: There are a lot of work of transformers with local attention and shared global connection between these local attentions. The model architecture is not novel from a machine learning model aspects.
 - quality: It would be nicer if the author can use other transformer based space-time model to train on these benchmarks and compare the performance with them. After all, ConvLSTM is from 2015, PredRNN and PhyDNetis not based on transformer, so does E3D-LSTM which is from 2017. Rainformer is the only one using transformer, but not specialized in this task.

questions: - I am not sure why N-body MNIST is related to the chaotic 3-body motion problem and thus performing good on this dataset means the model is capable to handle complex system. It would be better if the authors illustrate the reasons and intuitions behind this?
- It would be better if you can give some training details. For example, how you encode the input data and what is the dimension of the embeddings.

limitations: - The authors do not address the limitation adequately
- This framework is not physical dynamic-aware such as the PhyDNet paper which is also mentioned by the authors. To handle a complex system, I think it is important to introduce knowledge from the dynamics.
- There are few analyses showing why this framework shall only be specialized on this task. They might extend this framework to video frame prediction tasks and comparing to the SOTA in these datasets (e.g. UCF101).
- It would be nicer if they can post some failure cases and discuss the possible issues.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose an earth system forecast model based on transformers. This is achieved by first dividing the entire space-time field as non-overlapping cuboid blocks, applying attention in each cuboid block together with several "global vectors" that allow for interactions between the individual blocks, and then merging the blocks together to yield the predictions. They find state-of-the-art results on several synthetic and real datasets including moving MNIST, N-body MNIST (an extension of moving MNIST), precipitation nowcasting and SST anomaly forecasting.
strengths_and_weaknesses: Strengths:
- Thorough benchmarking to demonstrate the performance on real and synthetic data in addition to nice ablation studies.
- Results on the N-body MNIST look great. Predictions are much less blurry than the baselines, which are a common problem for these data-driven nowcasting/forecasting models.

Weaknesses:
- In terms of novelty, with so many recent papers applying transformers to vision tasks, the idea itself is not very original. There are many papers that applies transformers to video prediction tasks, e.g. "Anticipative Video Transformer (Girdhar and Grauman 2021)", that the authors also acknowledge. However, I have not seen any paper that applies cuboid attention, so this paper may be novel in this respect.
- Comparisons with physics-based models would have been nice to see as for many problems in weather forecasting, physics-based model still perform better than data-driven models. It would be interesting to know how the transformer approach fare against these traditional modelling approaches.

questions: There are several lines throughout the text that I found difficult to comprehend. Please could you clarify this?
- Lines 238-240:
> Note that the MovingMNIST performance on several papers [13] are obtained by training the model with on-the-fly generated digits while we pre-generate the digits and train all models on a fixed dataset.

What are "on-the-fly generated digits"? What was the reasoning behind training the model differently in this paper vs the other papers?

- Lines 184-185:
> generates the prediction via a coarse-to-fine procedure

What exactly is the coarse-to-fine procedure?

- What do the subscripts $(a,b,c)$ in the latent visual code $Z_{(a,b,c)}^{\text{code}}$ mean in the VQ-VAE? This notation is not introduced. Also, shouldn't $Z^{\text{code}}$ in the LHS of equation (9), Appendix D be $Z_i^{\text{code}}$ (since the RHS has index $i$)?

- Could you please explain in more details the idea behind auto-regressive Earthformer? This is to incorporate temporal correlations? But if so, shouldn't the global vectors already take that into account?



limitations: One limitation is that the model may have good performance with respect to standard machine learning metrics, however in practice, this may not necessarily translate to their usability, as demonstrated in the recent Deepmind paper on precipitation nowcasting (Ravuri et al. 2021). This is already visible in Figures 7-10, where the blurry predictions have better MSE score than the more realistic, high fidelity predictions that are clearly better. This indicates a search for better metrics for earth system forecasting or approaches that don't rely on loss optimisation.
review_title: null

Meta-review: metareview: This work proposes a Transformer-based architecture using 3D blocks for spatio-temporal prediction, designed specifically for Earth system forecasting applications. The authors show that this method considerably outperforms other approaches both on two climate/weather forecasting tasks and on unrelated synthetic tasks. The reviewers agree that this is a strong submission, and it addresses an important area of problems too often neglected by our community. While one reviewer held a conflicting opinion to the other three, this review was completed 20 days after the deadline, and no response was made by this reviewer to the author rebuttal despite my requests to the reviewer. I believe that their concerns have been adequately addressed by the authors.

 I accordingly recommend acceptance of this paper.
confidence: Certain
Decision: Accept