CirT: Global Subseasonal-to-Seasonal Forecasting with Geometry-inspired Transformer

Authors: Yang Liu, Zinan Zheng, Jiashun Cheng, Fugee Tsung, Deli Zhao, Yu Rong, Jia Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Weather and climate forecasting

Abstract:
Accurate Subseasonal-to-Seasonal (S2S) climate forecasting is pivotal for decision-making including agriculture planning and disaster preparedness but is known to be challenging due to its chaotic nature. Although recent data-driven models have shown promising results, their performance is limited by inadequate consideration of geometric inductive biases. Usually, they treat the spherical weather data as planar images, resulting in an inaccurate representation of locations and spatial relations. In this work, we propose the geometric-inspired Circular Transformer (CirT) to model the cyclic characteristic of the graticule, consisting of two key designs: (1) Decomposing the weather data by latitude into circular patches that serve as input tokens to the Transformer; (2) Leveraging Fourier transform in self-attention to capture the global information and model the spatial periodicity. Extensive experiments on the Earth Reanalysis 5 (ERA5) reanalysis dataset demonstrate our model yields a significant improvement over the advanced data-driven models, including PanguWeather and GraphCast, as well as skillful ECMWF systems. Additionally, we empirically show the effectiveness of our model designs and high-quality prediction over spatial and temporal dimensions.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730339153463

thread_type:
official_review

<thread_object>
creator: reviewer_iixk

<content>
summary:
The article designs a circular transformer (CirT) to model medium—and long-term numerical forecasts. It takes into account the spherical structure of the ground, designs corresponding circular patches as input tokens of the Transformer, and uses the Fourier transform in self-attention to capture global information to model the spatial periodicity, achieving high accuracy.

strengths:
The article has clear logic and ideas. The proposed method is very consistent with the research problem. It considers the problems faced by Patch Embedding in global meteorological modeling. Compared with most methods, it has achieved excellent results in high-latitude regions.

weaknesses:
1、The data in the experiment of the article is 1.5° instead of 0.25° as used in the Pangu model. Obviously, the author should give a detailed explanation of the experiment.
2、The article lacks experimental conditions for key variables such as wind u and wind v.
3、The article states that the usage of the Transformer network with Fourier transform is an important component, but both AFNO and Fourcast use related structures, which is not innovative enough.

questions:
1、The author should explain in detail how the results of the 1.5° experiment of the Pangu model were obtained. In global forecasts, there is currently a 0.25° prediction model. Is the 1.5° model necessary?
2、The experimental conditions of variables such as wind u and wind v.
3、Most weather models are tuned after training is complete, the article does not perform this work and fine-tuning can be used to further improve model performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730445322824

thread_type:
official_review

<thread_object>
creator: reviewer_phhy

<content>
summary:
CIRT propose a novel network for accurate S2S climate forecasting. The key idea is to model the cyclic characteristic of the graticule, consisting of two designs: (1) decomposing the weather data by latitude into circular patches; (2) Use fourier transform in self-attention. Extensive experiments on ERA5 demonstrate the effectiveness of the proposed CIRT network.

strengths:
1. The paper is well-written, and the idea and motivation are intuitive. Figure 1 clearly highlights the limitations in previous methods, providing a strong rationale for the proposed approach.

2. CIRT introduces significant innovations to address the lack of geometric inductive biases in prior work. Specifically, it partitions the graticule uniformly by latitude and leverages the Fourier transform to capture global features, which are then processed with self-attention.

3. The experimental results effectively demonstrate the superiority of CIRT, comparing favorably with both numerical and data-driven models.

weaknesses:
GraphCast employs a graph neural network for weather forecasting, sharing a similar initial motivation with CIRT in aiming to reduce inappropriate geometric inductive biases. The authors should consider providing a more in-depth discussion on the distinctions between GraphCast’s model design and system architecture compared to their own, highlighting the unique aspects and advantages of their final approach.

questions:
Add discussion with GraphCast.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730512023426

thread_type:
official_review

<thread_object>
creator: reviewer_pwhl

<content>
summary:
This paper propose an improved approach to the Subseasonal-to-Seasonal (S2S) climate forecasting problem. The method mainly has two novel parts: 1. A better input patching method which utilize the circular nature of the graticule. 2.  Propose to intervene Discrete Fourier Transform throughout the transformer layers. 

The propose methods demonstrate improvement over all the existing methods, both numeric methods and data-driven method. And the propose method is especially effective in the long time horizon prediction compared to other autoregressive data-driven methods which suffer from error aggregation. The method also show better performance in the hard-to-predict areas like polar areas.

strengths:
1. The experiments are extensive, solid, and the results are good. The paper compares the proposed method with both numerical method and ml method, and the metrics on latitude-weighted RMSE and Anomaly are strong and solid. The visualization graphs are very useful to understand the impact of the proposed method in different dimensions.

2. The proposed method well utilize the characteristics of the input data modality. The circular patching strategy introduce a effective inductive bias, and the DFT well utilize the latitudinal spatial periodicity.

weaknesses:
1. The comparison to existing method does not include computation complexity/model size. This weakens the comparison to other methods because the reader don't know whether the improvement is brought by the network architecture design or simply due to the increase in model size.

2. It would be a plus if the paper could also include a ablation study of autogressive prediction vs directly predicting all the future values, to ablate the effectiveness of the autoregressive formulation. But I understand this might be consuming.

questions:
I am wondering why the proposed method patch all the different longitudinal areas into the same patch. What prevents the model from taking a 2d input HxWxD and do e.g. axial transformer (intervene between attention along H axis and then W axis) with Fourier Transform.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257602328

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
Dear Reviewers,

We sincerely thank all the reviewers (`pWHL`, `PHHy`, `iiXK`) for their valuable feedback. We are glad that the reviewers appreciated the effectiveness and consistency of our proposed framework (`pWHL`, `PHHy`, `iiXK`), the extensiveness of our experiments (`pWHL`, `PHHy`), the solidness of experimental results (`pWHL`, `PHHy`, `iiXK`), and the overall quality of our paper's writing (`PHHy`, `iiXK`).

We have made every effort to faithfully address your comments in the responses. As suggested by the reviewers, we add

* Additional comparison of model computation complexity and model size ([pWHL](https://openreview.net/forum?id=YslOW2SO6S&noteId=ufTafZz5op)).
* Additional ablation study between autoregressive and direct prediction ([pWHL](https://openreview.net/forum?id=YslOW2SO6S&noteId=ufTafZz5op)).
* Additional results on wind $u$ and $v$ ([iiXK](https://openreview.net/forum?id=YslOW2SO6S&noteId=XVzzP1Of3X)).
* Additional results of fine-tuning CirT ([iiXK](https://openreview.net/forum?id=YslOW2SO6S&noteId=XVzzP1Of3X)).
* Discussion with GraphCast ([PHHy](https://openreview.net/forum?id=YslOW2SO6S&noteId=FB1YN049Ho)) and AFNO ([iiXK](https://openreview.net/forum?id=YslOW2SO6S&noteId=XVzzP1Of3X)).
* Detailed explanation of baseline results ([iiXK](https://openreview.net/forum?id=YslOW2SO6S&noteId=XVzzP1Of3X)) and CirT motivation ([pWHL](https://openreview.net/forum?id=YslOW2SO6S&noteId=ufTafZz5op)).

We have incorporated the suggested modifications in the revised version, which are highlighted in blue. If you are satisfied with the revisions, we kindly request your approval to consider an improved score.

Thanks for all the reviewers' time again. 

Best regards, 

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546092048

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A kind reminder: the discussion period will end in two days

comment:
Dear Reviewers,

We sincerely thank you for providing valuable feedback. We kindly remind you that the responses to your concerns have been posted. As the discussion period will end in two days, we would be grateful if you could allocate some time to review our responses.

Thanks for all the reviewers' time again.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a geometry-inspired transformer for numerical weather forecasting. It involves two main improvements: (1) decomposing weather data into patches by geometry, and (2) using Fourier transform. The model performs well in subseasonal-to-seasonal weather forecasting on the ERA5 data.

All three reviewers recommended borderline acceptance, but two of them were not very confident about the recommendation. The AC reads the paper and believes that the paper is of merit to the community, in particular, AI for scientific computing is an important and rising field and ICLR should encourage studies in this direction. The AC recommends acceptance, but leaves the following comments for the authors to address in the final version.
* The comparison may be unfair to other models (FourCastNet, Pangu-Weather, GraphCast) because these models were not optimized for seasonal forecasting and these models have a resolution of 0.25 degrees (which requires more details; but adding more spatial details can be harmful for long-range weather forecasting).
* The authors should clearly mention the relationship between the claimed technical contributions and prior works. The use of geometry (e.g. circular patches) is highly related to the Earth-specific prior of Pangu-Weather, and the Fourier transform is highly related to FourCastNet.
* The work uses 2D neural networks; it is good to discuss whether it is better to use 3D neural networks to integrate various pressure levels (cf. Pangu-Weather, Fengwu, Fuxi, etc.).