CirT: Global Subseasonal-to-Seasonal Forecasting with Geometry-inspired Transformer
Authors: Yang Liu
Zinan Zheng
Jiashun Cheng
Fugee Tsung
Deli Zhao
Yu Rong
Jia Li
Venue: ICLR 2025 Poster (2025)
Keywords: Weather and climate forecasting
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Accurate Subseasonal-to-Seasonal (S2S) climate forecasting is pivotal for decision-making including agriculture planning and disaster preparedness but is known to be challenging due to its chaotic nature. Although recent data-driven models have shown promising results, their performance is limited by inadequate consideration of geometric inductive biases. Usually, they treat the spherical weather data as planar images, resulting in an inaccurate representation of locations and spatial relations. In this work, we propose the geometric-inspired Circular Transformer (CirT) to model the cyclic characteristic of the graticule, consisting of two key designs: (1) Decomposing the weather data by latitude into circular patches that serve as input tokens to the Transformer; (2) Leveraging Fourier transform in self-attention to capture the global information and model the spatial periodicity. Extensive experiments on the Earth Reanalysis 5 (ERA5) reanalysis dataset demonstrate our model yields a significant improvement over the advanced data-driven models, including PanguWeather and GraphCast, as well as skillful ECMWF systems. Additionally, we empirically show the effectiveness of our model designs and high-quality prediction over spatial and temporal dimensions.

Published as a conference paper at ICLR 2025
CIRT: GLOBAL SUBSEASONAL-TO-SEAASE CASTING WITH GEOMETRY-INSPIRED TRANSFORMER
FORE-
Yang Liu Zinan Zheng'; Jiashun Cheng" Fugee Tsung!? Deli Zhao?, Yu Rong ! Jia Li The Hong Kong University of Science and Technology (Guangzhou) The Hong Kong University of Science and Technology DAMO Academy, Alibaba Group
ABSTRACT
(S2S) climate forecasting is pivotal for decision-making including agriculture planning and disaster preparedness but is known to be challenging due to its chaotic nature. Although recent data-driven models have shown promising results, their performance is limited by inadequate consideration of geometric inductive biases. Usually, they treat the spherical weather data as planar images, resulting in inaccurate representation of lo- cations and spatial relations. In this work, we propose the geometric-inspired Circular Transformer (CirT) model the cyclic characteristic of the graticule. consisting of two key designs: (I) Decomposing the weather data by latitude into circular patches that serve as input tokens to the Transformer; (2) Leveraging Fourier transform in self-attention to capture the global information and model the spatial periodicity. Extensive experiments on the Earth Reanalysis (ERA5) re- analysis dataset demonstrate our model yields significant improvement over the advanced data-driven models, including Pangu Weather and GraphCast, as well as skillful ECMWF systems. Additionally, empirically show the effectiveness of our model designs and high-quality prediction predictiooooooi over spatial and temporal dimen- sions. The code link is: https:/ tett:s://gihhie /ttpt/ttttttt///////////tt. com/ cowpasszzn/ /Crr..
Accurate Subseasonal-to-Seuane
INTRODUCTION
Tubseasonal-to-seassne (S2S) forecasting, which predicts meteorological variables 2 to weeks in advance, is crucial for agriculture, resource allocation, and disaster preparedness (e.g., heatwaves and droughts) (Mouatadid et al., 2024). Despite its high socioeconomic benefits, such task has long been considered a "predictability desert" (Vitart al., 2012) due the chaotic nature of the atmo- sphere. Compared with medium-range (up to 15 days) and seasonal predictions (3-6 months) (Vitart al., 2017), the S2S timescale is long enough to lose much of the memory of atmospheric initial conditions, while it is too short for slowly evolving earth system components such as the ocean that strongly influence the atmosphere (Black al., 2017: Phakula et al., 2024). The existing S2S near- real-time forecasting models rely physics-based Numerical Weather Prediction (NWP) models that discretize governing equations of thermodynamics. fluid flow, etc (Nathaniel et al., 2024). How- ever, these models generally suffer considerable biases (Mouatadid et al., 2023) and require mas- sive computational resources to perform numerical integration at fine-grained resolutions (Schneider al., 2023).
Multiple studies utilize the potential of data-driven models to mitigate the above weakness, in which most works (Hwang et al., 2019; He et al., 2022; Mouatadid al., 2024) focus regional fore- casting. However, the regional weather is often influenced by conditions in other areas on the S2S timescale, indicating the insufficiency of relying solely on regional inputs for S2S forecasting (Vitart et al., 2012; Lau & Waliser, 2011: L; Robertson et al., 2015). With the development of the high high-quality Earth Reanalysis 5 (ERA5) dataset (Hersbach et al., 2020) and weather foundation models (Pathak et al., 2022; Lam et al., 2022; Bi et al., 2023), few studies (Chen et al., 2024; Nguyen et al., 2023; Weyn et al., 2021) have proposed global data-driven S2S forecasting models and achieved promis- ing results. Specifically, they treat weather parameter values on the latitude-lonittdee grid (i.e..
"quual contribution.
Corresponning authors (ji ee@ust. hk, ronghhgehot com)

---

Published as a conference paper at ICLR 2025
graticule) as image data, represented as 3-dimensional tensors, and employ the Transformer (Doso- vitskiy, 2020) to forecast the future weather parameter values as an image generation task. Despite the promising results, the inconsistency between the planar and sphere geometry leads to signifi- cant distortions in learning dynamics, resulting in incorrect spatial relations. Figure depicts the example of this heavy distortion in planar projection.
Planar View meter temperature)
Therefore, we re-investigate the transformer design for graticule by considering two geometric inductive biases. First, existing methods decompose the planar latitude- longitude image into the fixed-degree patch, such as 3 x 39, ignoring that parallels have unequal geometric lengths. For example
Patch
Patch
Figure 1, although the patch size is the same in the planar view, the area of Patch is significantly larger than that of Patch Thus, the generated patches are of varying sizes and shapes in the sphere, especially in high-altitude regions, leading to an uneven distribution of information across patches. Sec- ond, the graticule demonstrates latitudinal spatial period- icity. Overlooking such inductive bias results in inac- curate spatial relation modeling. As shown in Figure I, the left and right boundaries of the planar view are con- nected while appear separated.
Spherical View
Figure I: Planar and the spherical view of 2-metre temperature. Treating it as an image results in distortion.
In this work, propose Circular Transformer (CirT)
that implements an equidistant circular patching strategy and self-attention incorporating spatial periodicity. To construct undistorted spatial relations among patches, CirT partitions the graticule uniformly by latitude, treating weather variables distributed on each parallel as a patch. Thus, the generated patches are in the same shape with geometric lengths determined by latitudes. Meanwhile, the adjacent patches are equidistant. Considering the weather signals are spatially periodic on the circular patch, we treat it as a spatial signal of 2n periodicity and leverage the Fourier transform to extract the global features and perform self-attention to mix patches the frequency domain. The frequencies are inversely transformed into the spatial domain to make the final forecasting. Finally, instead of learning autoregressive models like previous works (Chen et al., 2024; Nguyen et al., 2023; Weyn et al., 2021), we directly train CirT to predict in S2S timescale, which avoids the large accumulation errors and learns the connections between the initial and target states. Through extensive experiments on the ERA5 dataset, we find that
Remarkably, CirT outperforms skillful numerical S2S systems including UKMO, NCEP, CMA, and ECMWF, as well as state-of-the data-driven models including ClimaX, FourCastNetV2, PanguWeather, and GraphCast on S2S forecasting.
Most methods, including data-driven and numerical models, achieve a larger bias in high-latitude areas. contrast, we show that CirT produces more structurally consistent results with ground truth and performs better in these areas.
Ablation studies show that the proposed two simple designs, circular patching and patch mixing in the frequency domain, significantly enhance the model performance.
2 PRELIMINARY
Problem Definition We study the bi-weekly forecasting of K weather parameters at the latitude- longitude grid g E "xxx2 H and W are the height and width of the grid that depends on the resolution of latitude and longitude, and Gn, = (At, @w)) E Ω = 90", 909] x [-180,, 180%). At day f, the state of global weather is represented by a 3-dimensional tensor X, E R H x Following previous works (Chen al., 2024; Mouatadid et al., 2023; Nguyen et al., 2023), given the initial condition (g, X,,,,,,, ). our objective is to learn neural network to predict the average value of weather variables over weeks 3-4 and weeks 5-6, as shown in the following:
(8i1sita & ( fe(g, x,,),
(1)
where denotes the parameters neural networks. E Eistta and X Ntagita are predicted average value over weeks 3-4 (from day 15 to day 28) and weeks 5-6 (from day 29 to day 42). Distinet from
2

---

Published as a conference paper at ICLR 2025
Week 5-6 Prediction
Week 3-4 Prediction
Transformer Encoder
Lx
Circular Patch Examples
Add Norm
Output Head
2 Meter Temperature
Feed Forward
Transformer Encoder
IDFT
Add Norm
Patch
Circular Patching and Embedding
Patch
Multi-Head
Patching by Latitude
90
90"
A tttentoon
180
180
OF
DFT
X
Patch Embedding
Initial States
Figure 2: CirT architecture and circular patching examples. The input tensors are first decomposed by latitudes, resulting in set of circular patches. Then they are fed into series of Transformer blocks where DFT and IDFT are applied each block to transform information between frequency and spatial domain. Finally, the output head maps the represeniation to biweekly predictions.
data-driven medium-range models that iteratively produce the results, aim to learn a model that directly predicts these two values.
Fourier Transform and Inversion Fourier Transform is known be effective tool extract features from periodic signals. Consider sequence of N grid-based real-valued observations of a function, denoted by . SN) E R., The Discrete Fourier Transform (DFT), represented by F, converts this sequence into the frequency domain with a periodicity of 2 as follows:
k - - Sn sin 2nT T²
S = Σ Sn COS
2T N
A Bgi,
#=1
==1
where S = = (S) SN) E R and is the imaginary unit. A and Bg are the real and imaginary parts of the complex number S₃ in the frequency domain, respectively. The inverse transformation, which reconstructs the original sequence from the frequency domain, is given by:
Se cos 2 kk)
tisin (2T n kxm) k))
8n
or cquivalently by substituting S = A Bri
Sn
A cos 2TT n2 k)
By sin 2n 72
k))
(4)
where the imaginary unit is canceled out. We express the inverse transform as s = F-)(() for symmetry.
3
CIRT MODEL
Our CirT architecture is illustrated in Figure 2. The input X,, is split to embedded circular patches and then fed into the Transformer encoder to predict the Weeks 3-4 and Weeks 5-6 results. In the following, we elaborate on the model structure including circular patching and transformer encoder.
3

---

Published as a conference paper at ICLR 2025
Circular Patching CirT first divides the imput Xn into H latitudinal non-overlapping patches (x (1) 4
Jhm1 where ( Rw*X and its w-th row X R denotes the weather values at the coordinate (A,, Dw). Thus, the geometric distance of adjacent patches X and w+11 is fixed to RAØ where AΦ is the longitude resolution and the earth radius. Meanwhile, the geometric length of the patch ( can determined by 2T R (os())). These patches are then flattened and stacked into a matrix X E (R'x((-K), which is subsequently projected into latent space to generate initial embedding E E RI follows:
E = + W Pos:
(5)
where W () and W pos HR denote the learnable projection matrix and additive position embedding, respectively. Subsequently, the initial embedding is fed into the Transformer encoder for further processing.
CirT Transformer Encoder Considering the circular patch satisfies X Xw+ Therefore, instead of directly inputting X into the transformer, we consider its Fourier transform, composed of coefficients from series of periodic basis functions. The Fourier Transform offers insights into the wave frequencies present in the periodic signals, which aligns well with data with inherent pe- riodicities (Zhou et al., 2022; Wu al., 2023). Our approach aims incorporate such an inductive bias into the learning process by operating in the frequency domain. Specifically, at the (-th trans- former block, we begin by applying DFT to each row of input embedding EXI) E R#xD, where h.1
E = (E() E E IR corresponds to the embedding of the h-th patch. According to Eqn. 2,
E!ll cos 2n k n
k. Σ E sin 2m ") 1 B
Σ
(6)
where 500 = F(E RD represents the complex frequency embedding, and A = Re(s
and 1 lm () represent its real and imaginary parts, respectively. These components of all patches are then stacked into matrices (1) E RNxD and (()) E NxD,, which are then jointly fed
to the multi-head attention consider their correlation.
For the m-th attention head, we compute the query, key, and value matrices of (()) = [[A B01 e Hx2D following standard attention operations:
Ql..m = (OW' T K (l.m) wk m
((.m) = c( w m.
(7)
where WO/KKN 2D222D are learned projection matrices. The attention output Al.m) and
(l.m) are then computed using scaled production:
(.l..) = = (l.m). B(l.m)) softmax @a.m) (1.m) [(l.m)
(8)
where (l.m) consists of the first columns the output (1.m) while (1.m) corresponds to the rest. For the h-th patch, the processed frequency-domain representation is then rearranged as
5((,7)) A( (1,m) B (l.m)) i. This representation is then converted back to the original domain by applying the inverse DFT following Eqn. 4:
D
(l..) b,n
1
D
(l.m) h.k
cos 2T n k)
B.(l.om) sin 2 hikk
ni T kx):
D
k=1
where Ef(.m) - -S-(((() is the trress--transformedd representation of the patch. By stacking these transformed representations. we obtain the reconstructed embedding E(l.m). Finally, a feed- forward network used generate the output embedding E(++) by passing the reconstructed embeddings from all heads:
E(++1 MLP([E B(LMO)).
(10)
For brevity, we omit the LayerNorm layers the forward process of the Transformer block, as shown in Figure 2.
4

---

Published as a conference paper at ICLR 2025
Training Given the output representation of the Transformer encoder, flattened MLP is used to obtain the prediction results of weeks 3-4 and weeks 5-6. The model is trained to minimize the discrepancy between the prediction and ground truth. The loss in each variable and location is gathered and averaged over weeks 3-4 and 5-6 to calculate the overall objective loss:
2x 2x [1 X L1sstan 122 11 t29:142 X toaitaal 11)).
(II)
where X fisites and X t2201422 are the ground truth.
4 EXPERIMENTS
Dataset We evaluate the effectiveness of CirT on the ERAS reanalysis dataset (Hersbach et al., 2020) which provides the comprehensive pressure and single levels climate variables. The resolu- tion is set to 1.5", resulting in a 121 x240 latitude-longitude grid. We use 6 pressure level variables, including geopotential (z), specific humidity (g), temperature (4), u component of wind (u), V com- ponent of wind (v), and vertical velocity (w) at 10 pressure levels: 10, 50, 100, 200, 300, 500, 700, 850, 925 and 1000 hPa. Besides, we integrate 3 more single levels variables: 2m temperature (f2m), 10m u component of wind (10w), 10m v component of wind (10v), totaling 63 variables. We use the 1979-2016 (38 years of) data for training, the 2017 data for validation, and the 2018 for testing.
Metric Following existing works (Rasp et al., 2024; Nathaniel et al., 2024), we adopt latitude- weighted RMSE and Anomaly Correlation Coefficient (ACC) to evaluate the model performance with K = 1, which are defined as follows:
Ene ath) X/, VEn. a(h) R2 Enn a(h)
RMSE
Σ ( X2, ACC= K,
(12)
HW
where ahh) = cos() H En cos(Aw) is the latitude weighting factor. X = X Nuan1 E lllW is
the ground truth for specific day with its prediction X. X = X C and X h,w0 X -C, where C' is the observational climatology (i.e., empirical mean of observational data).
Data-driven baselines Following the existing S2S benchmark (Nathaniel et al., 2024), we com- pare CirT with ofate-of-the-arr data-driven models, including FourCastNetV2 (Pathak et al., 2022), PanguWeather (Bi et al., 2023), GraphCast (Lam et al., 2022) and ClimaX (Nguyen et al., 2023). Among them, FourCastNetV2,, is a graph neural network.
and ClimaX follow a ViT process, while GraphCast
Physics-based baselines To further evaluate the model performance of CirT, we compare it with various advanced physics-based models, including UK Meteorological Office (UKMO) (Williams et al., 2015), National Centers for Environmental Prediction (NCEP) (Saha et al., 2014), China Me- teorological Administration (CMA) (Wu al., 2019), European Centre for Medium- Range Weather Forecasts (ECMWF) (Molteni et al., 1996). Among them, ECMWF is recognized as the most skill- ful $2$ modeling system (Chen al., 2024; Domeisen et al., 2022). More details are shown in the Appendix.
Implementation details We use the following hyper- parameters for all direct training baselines: Batch size 16, the hidden dimension 256, and the attention head 16. All models are set to 8 layers and the learning rate is 0.01. All models are trained for 20 epochs. All models are implemented based on Pytorch Lightning, trained on GeForce RTX 4090 GPU. We train ClimaX the same as CirT and download trained Feu by ECMWE. We perform the inference of Download models in NVIDIA A800 80G GPU. We use the download parameters and do not finetune them to the S2S timescale due to the limited computational resources. Note that although FourCastNetV2 and Pangu Weather report the 2-meter predictions, the retrieved model in ECMWF does not include its inference and GraphCast is out-of- of--moory when performing inference for Weeks 5-6.
Weather and GraphCast through API' provided
5

---

Published as a conference paper at ICLR 2025
Table 1: Global S2S forecasting results of data-driven models. The lower RMSE and higher ACC
indicate better results.
RMSE (L)
ACC (1)
Metric
FourCastNetV2 GriphCust PanguWeather ClimaX CirT
FourCastNerV2 GraphCast PanguWeather ClimaX CirT
2500 (m² /$9)
615
618
649
602
477
0.947
0.973
0,963
0.977
0.984
Weeks 3-4
z850 (m² /s")
402
411
416
372
304
0.896
0,931
0.926
0.963
1500 (K)
2,093
2.176
2:271
2.886
1.687
0.966
0,979
0.966
0.981
0.988
1850 (K)
2.390
2.370
2.569
2.618
1.903
0,957
0,982
0,963
0.981
0.988
12m (K)
2.158
2.998
2,007
0.991
0,985
0.993
u1O (m/s)
2.328
2,431
2.334
1.806
0.830
0.812
0.817
0.896
v10 (m/s)
1,896
1,984
1.906
1.511
0.712
0.686
0.667
0.811
2500 (m² (s)
652
754
619
471
0.943
0.956
0,976
0.985
Weeks 5.6
2850 (m² /s*)
426
461
325
301
0.889
0.911
0,948
0.964
1500 (K)
2,250
2,829
2.254
1.672
0.963
0.958
0.980
0.988
1850 (K)
2.567
2.741
0,953
0.957
12m (K)
3.168
2.026
0.984
0.992
u10 (m/s)
2.479
2.679
2.355
1.809
0.812
0.783
0.814
0.895
v10 (m/s)
1,800
2.104
1.939
1.512
0.691
0.655
0.655
0.812
4.1 OVERALL PERFORMANCE
Compared with data-driven models We display the model performance in 7 target variables: geopotential 500hPa (2500), geopotential 850hPa (z850), temperature at 500hPa (#500), tem- perature at 850hPa (t850), 2m temperature (f2m), 10 metre U wind component (u10) and 10 metre wind component (v10) in Table Based on these results, we have the following observations:
CirT consistently outperforms all baselines in all cases. Specifically, compared to the best base- line, the average RMSE improvement geopotential (m² (m²/s²) and temperature (K) over Weeks 3-4 and Weeks 5-6 is 96.5, 0.369, and 111, 0.843, demonstrating significant improvement.
CirT achieves larger improvement over Weeks 5-6 than Weeks 3-4 predictions. The iterative mod- els (i.e., FourCastNett22 and PanguWeath) accumulate errors in each step, leading to inaccurate predictions when the iterative step large. In contrast, the direct prediction models aim to capture the relations between initial and subseasonal states, resulting in lower performance drops.
Compared with ViT-based iterative models, GraphCast achieves relatively better performance in Weeks 3-4 predictions. The reason be attributed to that it employs mesh to model the sphere geometry, leading to lower accumulated errors.
We can find that wind forecasting is more challenging than other comparing variables. For exam- plc, Weeks 3-4 t850 ACC of FourCastNetV2 is 0.957 while u10 ACC is 0.830. Under such cases, CirT still performs the best, further verifying its effectiveness.
In addition, we provide relative RMSE comparison in Figure 3. We can observe that CirT generally achieves lower errors across all pressure levels. When the lead time increases from Weeks 3-4 to 5-6, the performance of baselines significantly reduces, especially in Temperature. In contrast, CirT maintains relatively low errors.
Compared with numerical models To better investigate the performance CirT, we compare and numerical models in Figure where lighter colors indicate lower RMSE. The ACC results shown in the Appendix. From the results, we can find that:
CirT remarkably outperforms numerical models in almost all cases, demonstrating the effective- ness of direct data-driven models. In addition, numerical methods underperform at low-pressure levels, especially at the 10 hPa level, while CirT still performs better.
Compared with the results of Weeks 3-4 and 5-6, we can observe that similar to iterative data- driven models, the performance of numerical models decreases when the lead time increases but the performance drop is smaller than data-driven models. Such results show that physics-based models are more stable than iterative data-driven models. Overall, CirT which directly predicts the biweekly states maintains the best performance.

---

Published as a conference paper at ICLR 2025
Weeks 3-4
Weela 5-6
Weeks 3-4
Werks 5-6
1.0
FCN
0.8
0.6
0.4
ClimaX
198 33r say TWU 250 v2s 19n0 30 100 2M
300 200
299
300
925 1966
180 FIR 100 200 sue see TH0 850 925 1000
0.0
(lhh))
12m Weeks
Weeks 5-6
GC
ClimaX
CirT
ClimaX
CirT
LKMO
0.8
NCEP
0.6
CMA
BorT normalll
0.4
ECMWP
0.2
Leval (bPaa)
Laval (MP))
Lamdl (bits)
Lavel (aits
12m
UKMO
ECMWF
UKMO
ECMWF
CirT
u Wieks 3-4
Werks S-6
Works 3-
Works
FCN
0.8
0.6
PW
0.4
ClimaX
0.2
Cirl
51 190 206 291 59 THE
VS5 1991 56 106 bi 348 188 701 XSI
935 1866 58 100 20 388 588
701 300
425 1008 30 100 201 top SaB THO 1S6
925 1000
0.0
Weeks
Weeks
Weeks
Weeks
1.0
UKMO
RAISE after Mita Mox
NCEP
wwrmaltzatoo
0.6
Lavdl (MF))
Lovel (MPaa)
Lavil (MP))
Lavil (kiPa)
Figure 3: RMSE comparison between CirT and data-driven and numerical methods on geopoten- tial temperature wind and of different pressure levels. FCN, GC, and PW are short for FourCastNetV2, GraphCast, and PanguWeather lighter color indicates better results: CirT con- sistently outperforms all models.
4.2 ABLATION STUDY
To validate the effect of each model design on the overall model performance, we compare CirT with several invariants based whether they grid circular patching well Fourier transform. The results are shown Table 2, where we can observe that:
Fourier transform is a strong inductive bias and directly applying it does not always enhance model performance. From the table, we can observe that employing FT for grid patches increases the z850 and (2m the necessity of designing suitable patching approaches.
in Weeks 3-4 well z850, 1850, and t2m errors in Weeks 5-6, indicating
Employing the circular patching strategy improves model performance, especially when the model uses the FT. For example, when no FT. employing circular patching reduces z500 Weeks 3-4/5-6 RMSE from 516/501 to 502/498. In contrast, the case the FT, employing circular patching significantly reduces z500 Weeks 3-4/5-6 RMSE from 497/494 to 477/471. Such results not only suggest the effectiveness our model designs but also validate the effectiveness utilizing the FT to extract the spatial periodic signal from circular patches.

---

Published as a conference paper at ICLR 2025
Table 2: Ablation studies of patching strategies and Fourier transform.
Fourier
RMSE (4)
ACC (1)
Patch
Transform
z500
z850
1500 1850 12m
u10
v10 z500 z850 t500 1850
12m
u10
v10
Grid
516
319
1,910 2.168 2.554 1.946 1.616 0.983 0.959 0.986 0.987 0.990 0.877 0.782
Weeks
Circular
502
313
1.74 2.077 2,000
1.827
1.503
0.983 0.961 0.987 0.988 0.993 0.893 0.811
Grid
497
324
1.733
2.050 2.583 1.970
1.614 0.983 0.958 0.988 0.988 0.990 0.875 0.782
Circular
V
477
1.687
1.903 2.007
1.806
1.511 0.984 0.963 0.988 0.988 0.993 0.896 0.811
Weeks 5-6
Grid X
501
319
1,808 2.113 2.578 1.932 1.61400.933
0.959 0.987 0.987 0.989 0.879 0.783
Circular
Grid X
498
311
1.707 2.008 2.178 1.812 1.515 0.984 0.962 0.987 0.989 0.992 0.895 0.810
494
320
1.737
2.112 2.650 1.963
1.621 0.983 0.960 0.988 0.988 0.989 0.877 0,781
Circular
471
301
1.672
1.933 2.026
1.809
1.512
0.985 0.964 0.988 0.989 0.993 0.895 0.812
Table 3: RMSE comparison w.r.t latitude. CirT generally achieves the best performance and has higher relative improvement in mid-/high-latiudee areas.
Weeks 3-4
Weeks 5-6
Variable FourCastNetV2 GraphCast Pangu Weather ClimaX CirT FourCastNetV2 GraphCast Pangu Weather ClimaX CirT
z500
200
229
293
234
223
468
242
185
2850
134
150
168
159
112
153
236
158
107
Low- Low-Lat. Lat.
(500
1.000
1.357
1.490
1.321
1.199
2.278
1.364
1.008
(850
1,403
1.488
1.762
1,902
1.310
1,564
2.332
2,002
1.290
t2m
1.363
2.044
1.308
2.156
1.264
u10
1.789
1.953
2.119
L,966
2,295
2.165
1.467
v10
1.399
1.494
1.713
1.165
1,503
1.679
1.779
1.177
2500
799
809
842
837
6.32
852
936
860
633
z850
521
539
541
506
404
552
584
511
406
Mid-Lat.
2.630
2.749
2,832
2.082
2.819
3,226
2.932
2.072
(850
2.750
2.787
2.984
3.093
2.217
2.950
3,349
3.249
2.260
12m
2.212
3,276
2.031
3,496
2.071
2,967
3.004
2.474
v10
2.327
2.422
2.015
1.829
2.388
2.492
2.014
1.824
Z500
1804
1019
1238
861
756
1856
1299
880
743
2850
1050
715
566
1074
569
498
High-Lott
(500
5.249
3.297
4.667
2,833
2.414
5,465
4.857
2,879
2.497
1850
7.927
3.832
6.265
3,560
2.833
8,244
6.459
3.678
2.955
12m
4.153
4,874
3.601
5,086
3.674
u1o
2.559
2.567
2.641
1.918
2.414
2,453
2.377
1.765
v10
2.625
2.661
2.657
1.927
2,457
2,493
2.392
1.755
4.3 EMPIRICAL ANALYSIS
Latitudinal forecasting To investigate the model performance w.r.t. the latitude, we compare their results at low-latitude (09-30)) mid-latitud (309-609, and ligh-latitude (602-909) areas. The re- sults are displayed in Table From them, we can discover that CirT generally outperforms baselines in all areas. Moreover, CirT achieves larger relative improvement in mid-latitude and high-latitude areas. For example,
the 1500 Weeks 3-4 prediction, CirT has 1.2% relative improvement in low-latitude areas over the best bascline and 19.6%/16.22 in had-hhigh-latitude areas. Such im- provement can be attributed to the consideration of geometric inductive bias
Global visualization To provide a global view of model predictions, we visualize the Weeks 3-4 RMSE distribution in Figure 4. More visualizations such Weeks 5-6 predictions and different variables can be found in the Appendix. As shown in the figure, all models achieve the best re- sults in the equatorial region. Higher RMSE values are primarily concentrated in south polar areas and the continents of the Northern Hemisphere, especially in North America and the Bering Strait. Among these methods, CirT demonstrates significantly lower errors, even the above areas, further validating our framework of incorporating geometric information.
Performance w.r.t. month We further compare the model performance over different months, which shown in Figure The results of other variables can be found in the Appendix. CirT has superior predictive capabilities in forecasting all times, outperforming all competing methods
8

---

Published as a conference paper at ICLR 2025
Cirl
Pangu Weather
NCEP
RMSE
ClimaX
GraphCast
ECMWF
FourCast Ves
UKMO
CMA
2
The global RMSE distribution of 1850 with lead times weeks 3-4 in testing set: CirT demonstrates significant performance across different areas.
Figure 4:
1500 Weeks 3-4
Weeks
4.0
3.5
RMSE
3.0
RMSE
3.0
2.5
2.5
2.0
2.0
1.5
Jan
Apr
Jul
Oct
Jan
Apr
Jul
Oct
ECMWF
NCEP
CMA
UKMO
PW
FCNV2
GC
ClimaX + CirT
Figure 5: The monthly RMSE of t500 in testing set: CirT outperforms baselines across all months.
at all months. In particular, it has the largest improvement in June for both Weeks 3-4 and 5-6 predictions.
RELATED WORK
The advances in numerical weather prediction have dominated weather and climate modeling over the last century. They model the complex Earth dynamics coupled physical systems such as earth system models (ESM) (Hurrell et al., 2013), integrating the simulations of the atmosphere, cryosphere, land, and ocean processes. With the development of machine learning models (Ho et al.. 2019; Guibas et al., 2021) and the accessibility of high-quality weather data, various data- driven models have been proposed to mitigate the weaknesses of NWP such as high computational demands and sensitivity to initial conditions. Early studies target regional forecasting on specific variables, such as precipitation nowcasting in Hong Kong (Shi et al., 2015), wind prediction in Stuttgart (Harbola & Coors, 2019), and air temperature prediction in Australia (Deo, 2016). A notable progress is the publication of the ECMWF reanalysis v5 (ERA5) dataset (Hersbach et al., 2020), which combines historical observations with results from high-fidelin integrated Forecast- ing System (IFS) (Wedi et al., 2015). Based on such a dataset, pioneer works (Scher, 2018: Weyn al., 2019; 2020; Rasp et al., 2020; Rasp & Thuerey, 2021) study the global forecasting of specific

---

Published as a conference paper at ICLR 2025
variables such 500 hPa Geopotential and 300 hPa zonal wind, but at relatively coarser resolu- tions (Verma et al., 2024).
Recently, the development of foundation models has significantly advanced data-driven weather and climate forecasting. They are trained on large-scale high-resolution global data and target various weather variables. FourCastNet (Pathak al., 2022) and a follow-up work SFNO (Bonev et al., 2023) are built on the framework of the Fourier Neural Operator (Li et al., 2020; Guibas et al., 2021). Graph-based models (Lam et al., 2022: Keisler, 2022) such as GraphCast (Keisler, 2022) first create a mesh grid on the spherical surface and perform message passing (Liu et al.. 2024; Zheng et al., 2024; Liu al., 2022) on it. Besides these studies, a series of works are based on Transformer (Vaswani, 2017: Dosovitskiy, 2020). Pangu-Weather Weather (Bi et al., 2023) leverages sliding window attention to model spatial relations. Based on a similar backbone, Fuxi (Chen et al., 2023b) and FengWu (Chen et al., 2023a) improve training strategies to reduce accumulation errors and incorporate multi- malli-model/takk perspectives respectively. Climax (Nguyen et al., 2023) demonstrates its ability for various weather and climate tasks and CaFA (Li et al., 2024) considers the spherical geometry.
Despite the progress of current foundation models, S2S forecasting receives less attention due to its difficulty. Hwang et al. (Hwang et al., 2019) and He et al. (He et al., 2022) study regional S2S forecasting via traditional machine learning models such as AutoKNN and XGBoost (Chen & Guestrin, 2016). Weyn et al. (Weyn et al., 2021) designs an ensemble system based on convolution neural networks predict six atmospheric variables. It's only been recently that Climax (Nguyen et al., 2023) and Fuxi-S2S (Chen et al., 2024) have been developed to try to tackle these issues based on pre-trained foundation models. Therefore, how build an effective data-driven S2S forecasting model is still an open problem. In this work, we propose CirT and study the performance of the direct prediction model and show that it outperforms current iterative models. Moreover, existing S2S models generally treat global data as planar which introduces geometric inconsistency while we leverage spherical inductive bias in model designs to alleviate such problems.
Although both GraphCast (Lam et al., 2022) and CirT leverage geometric inductive biases, Graph- Cast focuses on local state aggregation and relies on message passing (Liu et al., 2023) to aggregate local information without explicitly accounting for spatial periodicity. In contrast, CirT employs circular patching to normalize patch geometry and leverages its Fourier representation, consisting of coefficients of periodic basis functions, as inputs to the transformer encoders. Compared with Four- castNet (Pathak et al., 2022) which aims to design an efficient token mixer for Vision Transformers that can effectively handle high-resolution inputs, CirT performs multi-head attention in the fre- quency domain to model the interactions among weather patches across various latitudes. Moreover, FourcastNet employs regular grid patching while CirT introduces circular patching standardize patch geometry. Pangu Weather also designs an earth-specific positional bias to integrate spherical information, encoding relative coordinates with learnable parameters into the attention weight com- putation. Nevertheless, they still employ cube patching and implicitly learn the geometric bias from data, while CirT explicitly leverages spherical bias in the patching strategy.
CONCLUSION AND FUTURE WORK
In this work, we highlight the geometric inductive bias in Transformer designs for S2S forecasting and introduce CirT, consisting of a circular patching strategy and latitudinal spatial periodicity mod- eling. It learns to mix patch embeddings in frequency domains and inverse transform to the spatial domain. Finally, it is trained to predict the future states in the S2S timescale. Extensive experiments on the ERA5 dataset demonstrate that CirT not only outperforms advanced data-driven models but also skillful numerical methods. Ablation studies have further substantiated the effectiveness of model designs and additional empirical analysis illustrates the superior performance in spatial and time dimensions. In the future, we are interested in incorporating slowly evolving earth system components including ocean, land, and sea ice in the proposed framework. In addition, CirT is a 2D transformer that encodes the inputs of different pressure levels into a single embedding. Neverthe- less, failure to incorporate the vertical inductive biases may result in incomplete cross-pressure level interaction. We plan to generalize the 2D CirT to the 3D transformer to incorporate such inductive biases.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is funded by National Natural Science Foundation of China Grant No. 72371217, the Guangzhou Industrial Informatic and Intelligence Key Laboratory No. 2024A03J0628, the Nansha Key Area Science and Technology Project No. 2023ZD003, and Project No. 2021JC02X191.
REFERENCES
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(79770:533-5538 2023.
Jiaxin Black, Nathaniel C Johnson, Stephen Baxter, Steven B Feldstein, Daniel S Harnos, and Michelle L L'Heureux. The predictors and forecast skill of northern hemisphere teleconnection patterns for lead times of 3-4 weeks. Monthly Weather Review, (4(7))2855-2877) 2017
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dy- namics on the sphere. In ICML, pp. 2806-2823, 2023.
Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, Jing-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang, Rui Su, et al. Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304. 02948, 2023a.
Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. Fuxi: A cascade machine learning forecasting system for 15-day global weather forecast. npj Climate and Almospheric Science, 190) 2023b.
Lei Chen, Xiaohui Zhong. Hao Li, Jie Wu, Bo Lu, Deliang Chen, Shang-Ping Xie, Libo Wu, Qingchen Chao, Chensen Lin, et al. A machine learning model that outperforms conventional global subseasonal forecast models. Nature Communications, 15(1))6425, 2024.
Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In KDD, pp. 785-794, 2016.
RC Deo. Monthly prediction of air temperature in australia and new zealand with machine learning algorithms. Theoretical and applied climatology, 125:13-25, 2016
Daniela IV Domeisen, Christopher J White, Hilla AfrrgannGerstman,
Gerstman, Ángel G Muñoz, Matthew A Janiga, Frédéric Vitart, Ole Wulff, Salomé Antoine, Constantin Ardilouze, Lauriane Batté, al. Advances in the subseasonal prediction of extreme events: Relevant case studies across the globe. Bulletin of the American Meteorological Society, 103(6))1143-E1150 2022.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1et20000..1199, 11999, 2020.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Efficient token mixing for transformers via adaptive fourier neural operators. In ICLR, 2021.
Shubhi Harbola and Volker Coors. One dimensional convolutional neural network architectures for wind prediction. Energy Conversion and Management, 195:70-75, 2019.
Sijie He, Xinyan Li, Laurie Trenary, Benjamin Cash, Timothy DelSole, and Arindam Banerjee. Learning and dynamical models for sub-seasonal climate forecasting: Comparison and collabo- ration. AAAI, volume 36, pp. 4493-4503, 2022.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 1997730):199920000 2020.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. CoRR, ths//192..11800 2019.

---

Published as a conference paper at ICLR 2025
James W Hurrell, Marika M Holland, Peter R Gent, Steven Ghan, Jennifer E Kay, Paul J Kushner, J-F Lamarque, William Large, D Lawrence, Keith Lindsay, et al. The community earth system model: framework for collaborative research. Bulletin of the American Meteorological Society, 94(9):133--136 2013.
Jessica Hwang, Paulo Orenstein, Judah Cohen, Karl Pfeiffer, and Lester Mackey, Improving sub- seasonal forecasting in the western with machine learning. In KDD, pp. 2325-2335, 2019.
Ryan Keisler. Forecasting global weather with graph neural networks.
arXiv preprint
arXiv:2202.07555. 2022.
Remi Lam, Alvaro Sanchez- -oobzazzG,,zzazz,, Matthew Willson, Peter Wirmsberger, Meire Fortunato, Fer- ran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Grapheast: Learning skillful medium-range global weather forecasting. arXiv preprint vrXiv2222.112994 2022.
William K-M Lau and Duane E Waliser. Intraseasonal variability in the atmosphere-oceom climate system. Springer Science& & Business Media, 2011.
Zijie Li, Anthony Zhou, Saurabh Patil, and Amir Barati Farimani. Cafa: Global weather forecasting with factorized attention sphere. arXiv preprint arXiv:2405.07395, 2024.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08899 2020.
Yang Liu, Liang Chen, Xiangnan He, Jiaying Peng, Zibin Zheng, and Jie Tang. Modelling high- order social relations for item recommendation. IEEE Trans. Knowl. Data Eng., 34(9):4385- 4397, 2022.
Yang Liu, Yu Rong, Zhuoning Guo, Nuo Chen, Tingyang Xu, Fugee Tsung, and Jia Li. Human mobility modeling during the COVID-19 pandemic via deep graph diffusion infomax. In AAAI, pp. 14347-14355, 2023.
Yang Liu, Jiashun Cheng, Haihong Zhao, Tingyang Xu, Peilin Zhao, Fugee Tsung, Jia Li, and Yu Rong. SEGNO: generalizing equivariant graph neural networks with physical inductive biases. In ICLR, 2024.
Franco Molteni, Roberto Buizza, Tim N Palmer, and Thomas Petroliagis. The eemwf ensemble prediction system: Methodology and validation. Quarterly journal of the royal meteorological society, 1226519):73-119, 1996.
Soukayna Mouatadid, Paulo Orenstein, Genevieve Flaspohler, Judah Cohen, Miruna Oprescu, Ernest Fraenkel, and Lester Mackey. Adaptive bias correction for improved subseasonal fore- casting. Nature Commanications, 14(1):3482, 2023.
Soukayna Mouatadid, Paulo Orenstein, Genevieve Flaspohler, Miruna Oprescu, Judah Cohen, Franklyn Wang, Sean Knight, Maria Geogdzhayeva, Sam Levang, Ernest Fraenkel, et al. Subsea- sonalclimateusa: a dataset for subseasonal forecasting and benchmarking. NeurIPS, 36, 2024.
Juan Nathaniel, Yongquan Qu, Tung Nguyen, Sungduk Yu, Julius Busecke, Aditya Grover, and Pierre Gentine. Chaosbench: A multi-channel, physics-based benchmark for subseasonal-to- seasonal climate prediction. arXiv preprint arXiv:2402.00712, 2024.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: 2023.
A foundation model for weather and climate. In ICMI., pp.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Four- castnet: A global data-driven high-resolution weather model using adaptive fourier neural opera- tors. arXiv preprint 22Xiv:2202.1122 12214, 14, 2022.
Steven Phakula, Willem A Landman, and Christien J Engelbrecht. Literature survey of subseasonal- to-seasonal predictions in the southern hemisphere. Meteorological Applications, 31(1):21170, 2024.
12

---

Published as a conference paper at ICLR 2025
Stephan Rasp and Nils Thuerey. Data-driven medium- raage weather prediction with resnet pre- trained on climate simulations: A new model for weatherbench. Journal of Advances in Modeling Earth Systems. 13()):22000MS002405,, 2021.
Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: a benchmark data for data-driven weather forecasting, Journal of Advances in Modeling Earth Systems, 12(1 ):220200SS00020,, 2020.
Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez-Gonzalee Vivian Yang. Rob Carver, Shreya Agrawal, et al. Weatherbench 2: A benchmark for the next generation of data-driven global weather models. Journal of Advances in Modeling Earth Systems, 1660):6202335004001 2024.
Andrew W Robertson, Arun Kumar, Malaquias Peña, and Frederic Vitart. Improving and promoting subseasonal to seasonal prediction. Bulletin of the American Meteorological Society, 96(3):E849 ES53, 2015.
Suranjana Saha, Shrinivas Moorthi, Xingren Wu, Jiande Wang, Sudhir Nadiga, Patrick Tripp, David Behringer, Yu-Tai Hou, Hui-ya Chuang, Mark Iredell, et al. The ncep climate forecast system version 2. Journal of climate, 27(6):21 180-2228,8222 2014.
Sebastian Scher. Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning. Geophysical Research Letters, 45(22):12-616, 2018.
Tapio Schneider, Swadhin Behera, Giulio Boccaletti, Clara Deser, Kerry Emanuel, Raffaele Ferrari, Ruby Leung. Ning Lin, Thomas Müller, Antonio Navarra, et al. Harnessing ai and computing to advance climate modelling and prediction. nature climate change, (3())8887-889 2023.
Xingjian Shi, Zhourong Chen, Hao Wang, Din-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: machine learning approach for precipitation nowcasting. NIPS, 28, 2015.
Vaswani. Attention all you need. NeurIPS, 2017.
Yogesh Verma, Markus Heinonen, and Vikas Garg. Climode: Climate and weather forecasting with physics-informed neural odes. arXiv preprint arXiv:2404. 10024, 2024.
Frédéric Vitart, Andrew Robertson, and David LT Anderson. Subseasonal seasonal prediction project: Bridging the gap between weather and climate. 2012.
Frederic Vitart, Constantin Ardilouze, Axel Bonet, Anca Brookshaw, M Chen, Codorean, M Déqué, L Ferranti, E Fucile, M Fuentes, et al. The subseasonal to seasonal (s2s) prediction project database. Bulletin of the American Meteorological Society, 98(1):163-173, 2017.
Zhou Wang, Ecro Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Seeeeny-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402, 2003.
NP Wedi, P Bauer, W Denoninck, M Diamantakis, M Hamrud, C Kuhnlein, S Malardel, K. Mo- gensen, G Mozdzynski, and PK Smolarkiewicz. The modelling infrastructure of the Integrated Forecasting System: Recent advances and future challenges. European Centre for Medium-Range Weather Forecasts, 2015.
Jonathan A Weyn, Dale Durran, and Rich Caruana. Can machines learn to predict weather? using deep learning to predict gridded 500-hpa geopotential height from historical weather data. Journa! of Advances Modeling Earth Systems, 11(8):2680-268) 2019.
Jonathan A Weyn, Dale R Durran, and Rich Caruana. Improving data-driven global weather pre- diction using deep convolutional neural networks on a cubed sphere. Journal of Advances in Modeling Earth Systems, 220)=:2220MM00000,, 2020.
13

---

Published as a conference paper at ICLR 2025
Jonathan A Weyn, Dale R Durran, Rich Caruana, and Nathaniel Cresswell- -lay. Sub-seasonal fore- casting with a large ensemble of deep-learning weather prediction models. Journal of Advances in Modeling Earth Systems, 13(7):2202 2021.
KD Williams, CM Harris, A Bodas-Salcedo, Camp, RE Comer, Copsey, D Fereday, Graham, R Hill, T Hinton, et al. The met office global coupled model 2.0 (gc2) configuration. Geoscientific Model Developmenr, 88(55):1509-1522 2015.
Liming Wu, Zhichao Hou, Jirui Yuan, Yu Rong, and Wenbing Huang. Equivariant paapooltemporal attentive graph networks simulate physical dynamics. In NeurIPS, volume 36, pp. 45360- 45380, 2023.
Tongwen Wu, Yixiong Lu, Yongjie Fang, Xiaoge Xin, Laurent Li, Weiping Li, Weihua Jie, Jie Zhang, Yiming Liu, Li Zhang, et al. The beijing climate center climate system model (bee-csm): The main progress from cmip5 to cmip6. Geoscientific Model Development, 12(4):1573-1600, 2019.
Zinan Zheng, Yang Liu, Jia Li, Jianhua Yao, and Yu Rong. Relaxing continuous constraints of equivariant graph neural networks for broad physical dynamics learning. In KDD, pp. 4548- 4558, 2024.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, pp. 27268 27286,, 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL EXPERIMENTAL DETAILS
A.1.1 BASELINE
UKMO The UK Meteorological Office uses the Global Seasonal forecast system version (GloSca6) model to generate daily control forecasts for 60-day lead time.
NCEP The National Centers for Environmental Prediction uses the Climate Forecast System (CFSv2) model to generate daily control forecasts for 45-day lead time.
CMA The China Meteorological Administration uses the Beijing Climate Center (BCC) fully- coupled BCC-CSM2-HR model to generate control forecasts each Monday and Thursday for 60- day lead time.
ECMWF The European Centre for Medium-Range Range Weather Forecasts uses the operational In- tegrated Forecasting System (IFS) to generate control forecasts each Monday and Thursday for 46-day lead time. We use the CY48RI version forcast.
FourCastNetV2 Iterative data-driven model built upon Vision Transformer. FourCastNetV2 patches all the variables and uses the Adaptive Fourier Neural Operator mix the spatial patches. We use the API https: //ttts::/gieee //g/tttttuu///ggg com/e cowff- albbaaaaaaa aaaaaimmmm miimoddes to perform inference. Due the data loss in October 2018, we utilized the available data from the remaining 11 months.
GraphCast Iterative data-driven model built upon Graph Neural Network. It use multi-mesh method to construct the graph lear the complex dynamics. We use the API https: / //github. com/ wwwwf-laaaaa aaaaimmoooe -models to perform inference. The GPU's memory only allows to perform inference data spanning maximum of 4 weeks. Therefore, only present the results for weeks 3-4.
Pangu Weather Iterative data-driven model built upon Vision Transformer. We the API ht hpsps: /ttps://gli h////tttubb com/ saenwf-lab/ate lab/ai-models to perform inference. PanguWeather patches the pressure level and single-level data separately and merge them in the transformer. Using hierarchical temporal aggregation method to train the mdoel.
ClimaX Direct training data-driven model built upon Vision Transformer. Each variable is inde- pendently tokenized and aggregated by variable aggregation.
A.1.2 DATASET
For all data-driven models, utilize the same approach to transform 0.25 x 0.25 grid to 1.5" x 1.5% grid, including obtaining the prediction of FourCastNetV2, Graphcast, and Pan- first obtain the results of
guWeather as well as the training grid data of CirT. Specifically, 0.25 x 0.25 models (e.g.. Pangu Weather), which are represented on a 721 1440 grid. This grid corresponds to the coordinates (X, Φ) within the domain Ω = [-900 -9.75 ..... 89.75% 909) x [-180 -179.77 75 75.,..., 179.75. 180%). where À denotes longitude and & denotes latitude. Sub- sequently, we retrieve the results of coordinates (A, Φ) that correspond the 1.5e 1.5e grid, which is represented on a 121 x 240 grid within the domain Ω = [-90", -88.5 88.5%, 28.5",90"] [-180%, 178..5 ..... 178.5 180%).
A.2 ADDITIONAL RESULTS
Ace results ACC comparison between CirT and numerical models at all pressure levels are shown in Figure 6 and ACC comparison w.r.t. latitudes are displayed in Table 9.
Regional forecasting We additionally evaluate models in regional forecasting, constrained the bounding boxes of North America and Europe. The results are shown in Table 7. We can observe that CirT outperforms baselines all cases.
Additional visualization The global visualization of t850 Weeks 5-6 predictions and other vari- ables including t500, 2500, and z850 shown in Figure 7-13.
15

---

Published as a conference paper at ICLR 2025
Table 4: Computation complexity and model size comparison.
Model
GraphCast
Pangu Weather
CirT
FLOPs
1
168 T
2.2G
Parums
37M
256M
16M
Table 5: Ablation study of autoregressive prediction vs directly predicting all the future values.
RMSE (.)
ACC (f)
Model
z500
z850
1500
1850
12m
u10
v10
z500
z850
1500
1850
12m
u10
v10
Autoregressive
781
453
3,406
4.014
4.584
2.806
2.267
0.962
0.922
0.956
0,957 0.968 0.763
0,610
Direct
477 304
1.687
1,903
2.007
1.806
1.511
0.984
0.963
0.988
0.988
0.993 0.896 0.811
Autoregressive Direct
813
455
3.636
4.357
5.047
2.855
2.324
0.960
0.923
0.950
0.949
0.960
0.758 0.599
471
301
1.672
1.933
2.026
1.809
1.512
0.985
0.964
0.988 0.989 0.993 0.895 0.812
Additional results w.r.t month Figure 14-16.
Additional results of variables t850, z500, and z850 are shown in
Comparison of model computation complexity/size We compare CirT's Floating point opera- tions (FLOPs) and parameters with two representative models, Graphcast and PanguWeather. The results are shown in Table 4. We can observe that CirT achieves better S2S predictivity with less computation and smaller model size, verifying our model designs.
Comparison of autoregressive and direct prediction We adapted CirT's output head to forecast next-day weather variables based on the imput date for autoregressive prediction. For inference, iteratively predicts next-day weather variables up to the S2S timescale. The results are shown in Table From the results, we can observe that the direct method performs better. The autoregressive CirT still accumulates errors, resulting in inaccurate S2S predictions.
Additional results on fine-tuning CirT. We further evaluate the performance of fine-tuning the trained autoregressive CirT. We freeze the transformer encoder and replace the embedding layers and output head with newly initialized networks to forecast weather variables for Weeks 3-4 and 5-6. The results in Table 6. From the result, best in most cases. Mcanwhile, we find that fine-tuned embedding layer and decoder improve the performance in several variables such as 1850.
observe that direct training still performs
Multi-scale structural similarity Following the previous work (Nathaniel et al., 2024), also compare Multi-Scale Structural Similarity (Wang et al., 2003) of the data-driven models. The result are shown in Table 8. CirT achieves the best performance. GraphCast is the best baseline for Weeks 3-4 predictions. The reason can be attributed to that it employs mesh to model the sphere geometry, consistent with the observations in Table 1.
Weeks 3-4
Woeks 5-6
1 Weeks 3-4
I Werks S-6
UKMO
NCEP
seemoliratin
CMA
FCMWF
CirT
50 180 380 201 RaY NOI
2000
19 SE 180
300 700 R06 700 506 1997
90 190
310 300
S3B THE 935 1000 TO Sa 100 780 339 Sas 788 ROI 935
(20 Werks 3-4
12nn Wedks S-6
UKMO
NCEP
CMA
ECMWE
CIrl
UKMO
NCEP
CMA
ECMWF CITT
Figure 6: The ACC comparison between numerical models and CirT.
16

---

Published as a conference paper at ICLR 2025
Table 6:
Ablation study of fine-tuning CirT model.
RMSE ())
Model
z500
z850
1500
1850
12m
u10
v10
Fine- tuning embedding and decoder
480
315
1.660
1.870
1.983
1.842
1,530
Finc-tuning Decoder
540
346
1.885
2.327
2.715
2.013
1.619
Direct Training
477
304
1.687
1.903
2.007
1.806
1,511
Fine-tuning embedding and decoder
485
312
1.679
1.923
2.032
1.847
1,535
Fine-tuning Decoder
588
354
2.190
2.702
3.145
2,043
1.650
Direct Training
471
301
1.672
1,933
2.026
1.809
1.512
Table RMSE comparison w.r.t. regions. CirT generally achieves the best performance and has higher relative improvement in mid-/high-latituee areas. N-America is short for North America
Weeks 3-4
Week 5-6
Variable FourCastNerV2 GraphCast PanguWeether ClimaX CirT FourCastNetV2 GraphCast Weathee ClimaX CirT
N-America
2500
1017
1014
996
879
801
1037
1063
916
774
z850
709
714
690
606
552
715
744
634
538
1500
2,770
2.704
2.793
2.392
2.128
2,858
3,221
2.431
2.136
1850
3.012
2,962
3.041
2.802
2.333
3.110
3.454
2.826
2.347
(2m
3.184
3.618
2.556
3.669
2.617
z500
892
905
909
855
651
953
854
644
Europe
2850
606
617
620
551
427
636
675
355
415
1500
2.862
2.828
2.869
2,573
2.111
3,094
3,282
2.516
2.163
1850
2.848
2,910
3.035
2,827
2,083
3.078
3.382
2.828
2.139
t2m
2.972
3.561
2.334
3.619
2.396
Table 8: Multi-scale structural similarity of data-driven models.
Week 3-4
Week 5-6
Variable
FourCastNerV2 GraphCast PanguWeather ClimaX
CirT
FourCastNesV2 GraphCast PunguWeather ClimaX
CirT
2500
0.814
0.872
0.862
0.909
2850
0.799
0.811
0.802
0.794
0.874
0.786
0.772
0.789
0.874
1500)
0.866
0.889
0,882
0.875
0.925
0.857
0.860
0.869
0.924
1850
0.882
0.919
0.913
0.893
0.942
0.876
0.885
0.942
12m
0.966
0.928
0.969
0.921
0.968
Table 9:
ACC comparison w.r.t latitude. CirT generally achieves the best performance and has higher relative improvement in mid-/high-latitude areas.
Variable FourCastNerV2 GraphCast PanguWeather Weather ClimaX CirT FourCastNetV2 GraphCast Pangu Weather ClimaX CirT
Weeks 3-4
Week
>500
0.998
0.998
0.998
0.997
0.999
0.998
0.997
0.997
0.999
Low-J Lat.
2850
0.993
0.991
0.990
0.991
0.995
0.991
0,983
0.991
0.996
t500
0.997
0.996
0.996
0.995
0,996
0.994
0.995
0.997
(850
0.995
0.995
0.995
0.993
0.997
0.995
0.993
0.992
0.996
t2m
0.997
0.994
0.998
0.994
0.998
z500
0.932
0.927
0.921
0.920
0.955
0.923
0.902
0.915
0.954
Mid- Latt.
0.888
0.878
0.874
0.887
0.876
0.886
(500
0.943
0.937
0.932
0.923
0.959
0.932
0.907
0.917
0,959
1850
951
0.947
0.939
0.931
0.965
0.90 0.942
0.922
0.923
0.963
0,944
z500
0.927
0.969
0.952
0.978
0.983
0.923
0.948
0.977
0,984
High-Latt. Lat.
2850
0.870
0.925
0.918
0.952
0.961
0.863
0.904
0.951
0.963
t500
0.951
0.978
0.952
0.982
0.987
0.948
0.952
0.981
0.988
(850
0.942
0.982
0.946
0,983
0.989
0.938
0.944
0.982
0,989
(2m
0.990
0.986
0.992
0.984
0.992
17

---

Published as a conference paper at ICLR 2025
CirT
Pangu Weather
RMSE
7
ClimaX
CMA
ECMWF
3
FourCastNerV2
UKMO
1
Figure 7: The global RMSE maps of 1850 with lead times weeks 5-6 in 2018.
CirT
Pangu Weather
NCEP
RMSE
ClimaX
GraphCast
ECMWF
FourCastNett2
UKMO
CMA
Figure 8: The global RMSE maps of t500 with lead times weeks 3-4 in 2018.
18

---

Published as a conference paper at ICLR 2025
CirT
Pangu Weather
NCEP
RMSE
ClimaX
C.MA
ECMWF
FourCastNett2
UKMO
Figure 9: The global RMSE maps of t500 with lead times weeks 5-6 in 2018.
CirT
Pangu Weather
NCEP
RMSE
1600
1400
1200
ClimaX
GraphCast
ECMWF
1000
800
600
FourCastNerV2
UKMO
CMA
400
200
Figure 10: The global RMSE maps of z500 with lead times weeks 3-4 in 2018
19

---

Published as a conference paper at ICLR 2025
CirT
Pangu Weather
NCEP
RMSE
1600
1400
1200
ClimaX
CMA
ECMWF
1000
800
600
FourCastNert2
UKMO
400
200
Figure 11: The global RMSE maps of z500 with lead times weeks 5-6 in 2018.
CirT
PanguWeather
NCEP
RMSE
1200
1000
ClimaX
GraphCast
ECMWF
800
400
FourCastNerV2
UKMO
CMA
200
Figure 12: The global RMSE maps of z850 with lead times weeks 3-4 in 2018
20

---

Published as a conference paper at ICLR 2025
CirT
Pangu weatherther
NCEP
RMSE
1200
1000
ClimaX
CMA
ECMWF
800
600
FourCastNerV2
UKMO
400
200
Figure 13: The global RMSE maps of z850 with lead times weeks 5-6 in 2018.
1850 Weeks 3-4
t850 Weeks 5-6
5.5
5.0
4.5
4.5
RMSE
4.0
3.0
3.0
2.5
2.5
2.0
2.0
Jan
Apr
Jul
Oct
Jan
Apr
Jul
Oct
ECMWF
NCEP
CMA
UKMO
PW
FCNV2
GC
ClimaX + CirT
Figure 14:
The monthly RMSE of 1850 testing set: CirT outperforms other models across all
months.
z500 Weeks 3-4
>500 Weeks 5-6
1200
1100
1100
1000
1000
RMSE
900
RMSE:
900
800
800
700
700
600
600
500
500
Apr
Apr
Jul
ECMWF
NCEP
CMA
UKMO
FCNV2
GC
ClimaX
CirT
Figure 15:
The monthly RMSE of z500 in testing set: CirT outperforms other models across all
months.
21

---

Published as a conference paper at ICLR 2025
1850 Weeks 3-4
2850 Weeks 5-6
700
700
600
600
RASE
RMSE
500
500
400
400
300
300
Apr
Jul
Oct
Jan
Apr
Jul
Oct
ECMWF
NCEP
CMA
UKMO
PW
FCNV2
GC
ClimaX
CirT
Figure 16:
The monthly RMSE of z850 in testing set: CirT outperforms other models across all
months.
22


Reviews:
[{"review_id": "ufTafZz5op", "reviewer": "Reviewer_pWHL", "summary": "This paper propose an improved approach to the Subseasonal-to-Seasonal (S2S) climate forecasting problem. The method mainly has two novel parts: 1. A better input patching method which utilize the circular nature of the graticule. 2.  Propose to intervene Discrete Fourier Transform throughout the transformer layers. \n\nThe propose methods demonstrate improvement over all the existing methods, both numeric methods and data-driven method. And the propose method is especially effective in the long time horizon prediction compared to other autoregressive data-driven methods which suffer from error aggregation. The method also show better performance in the hard-to-predict areas like polar areas.", "questions": "I am wondering why the proposed method patch all the different longitudinal areas into the same patch. What prevents the model from taking a 2d input HxWxD and do e.g. axial transformer (intervene between attention along H axis and then W axis) with Fourier Transform.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are extensive, solid, and the results are good. The paper compares the proposed method with both numerical method and ml method, and the metrics on latitude-weighted RMSE and Anomaly are strong and solid. The visualization graphs are very useful to understand the impact of the proposed method in different dimensions.\n\n2. The proposed method well utilize the characteristics of the input data modality. The circular patching strategy introduce a effective inductive bias, and the DFT well utilize the latitudinal spatial periodicity.", "weaknesses": "1. The comparison to existing method does not include computation complexity/model size. This weakens the comparison to other methods because the reader don't know whether the improvement is brought by the network architecture design or simply due to the increase in model size.\n\n2. It would be a plus if the paper could also include a ablation study of autogressive prediction vs directly predicting all the future values, to ablate the effectiveness of the autoregressive formulation. But I understand this might be consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FB1YN049Ho", "reviewer": "Reviewer_PHHy", "summary": "CIRT propose a novel network for accurate S2S climate forecasting. The key idea is to model the cyclic characteristic of the graticule, consisting of two designs: (1) decomposing the weather data by latitude into circular patches; (2) Use fourier transform in self-attention. Extensive experiments on ERA5 demonstrate the effectiveness of the proposed CIRT network.", "questions": "Add discussion with GraphCast.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, and the idea and motivation are intuitive. Figure 1 clearly highlights the limitations in previous methods, providing a strong rationale for the proposed approach.\n\n2. CIRT introduces significant innovations to address the lack of geometric inductive biases in prior work. Specifically, it partitions the graticule uniformly by latitude and leverages the Fourier transform to capture global features, which are then processed with self-attention.\n\n3. The experimental results effectively demonstrate the superiority of CIRT, comparing favorably with both numerical and data-driven models.", "weaknesses": "GraphCast employs a graph neural network for weather forecasting, sharing a similar initial motivation with CIRT in aiming to reduce inappropriate geometric inductive biases. The authors should consider providing a more in-depth discussion on the distinctions between GraphCast’s model design and system architecture compared to their own, highlighting the unique aspects and advantages of their final approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XVzzP1Of3X", "reviewer": "Reviewer_iiXK", "summary": "The article designs a circular transformer (CirT) to model medium—and long-term numerical forecasts. It takes into account the spherical structure of the ground, designs corresponding circular patches as input tokens of the Transformer, and uses the Fourier transform in self-attention to capture global information to model the spatial periodicity, achieving high accuracy.", "questions": "1、The author should explain in detail how the results of the 1.5° experiment of the Pangu model were obtained. In global forecasts, there is currently a 0.25° prediction model. Is the 1.5° model necessary?\n2、The experimental conditions of variables such as wind u and wind v.\n3、Most weather models are tuned after training is complete, the article does not perform this work and fine-tuning can be used to further improve model performance.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The article has clear logic and ideas. The proposed method is very consistent with the research problem. It considers the problems faced by Patch Embedding in global meteorological modeling. Compared with most methods, it has achieved excellent results in high-latitude regions.", "weaknesses": "1、The data in the experiment of the article is 1.5° instead of 0.25° as used in the Pangu model. Obviously, the author should give a detailed explanation of the experiment.\n2、The article lacks experimental conditions for key variables such as wind u and wind v.\n3、The article states that the usage of the Transformer network with Fourier transform is an important component, but both AFNO and Fourcast use related structures, which is not innovative enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)