Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Authors: Tung Nguyen
Rohan Shah
Hritik Bansal
Troy Arcomano
Romit Maulik
Veerabhadra Kotamarthi
Ian Foster
Sandeep Madireddy
Aditya Grover
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
transformers
weather forecasting
climate modeling
AI for climate
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce Stormer, a simple transformer model that achieves state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone.

Abstract: Weather forecasting is a fundamental problem for anticipating and mitigating the impacts of climate change. Recently, data-driven approaches for weather forecasting based on deep learning have shown great promise, achieving accuracies that are competitive with operational systems. However, those methods often employ complex, customized architectures without sufficient ablation analysis, making it difficult to understand what truly contributes to their success. Here we introduce Stormer, a simple transformer model that achieves state-of-the art performance on weather forecasting with minimal changes to the standard transformer backbone. We identify the key components of Stormer through careful empirical analyses, including weather-specific embedding, randomized dynamics forecast, and pressure-weighted loss. At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. During inference, this allows us to produce multiple forecasts for a target lead time and combine them to obtain better forecast accuracy. On WeatherBench 2, Stormer performs competitively at short to medium-range forecasts and outperforms current methods beyond 7 days, while requiring orders-of-magnitude less training data and compute. Additionally, we demonstrate Stormer’s favorable scaling properties, showing consistent improvements in forecast accuracy with increases in model size and training tokens. Code and checkpoints are available at https://github.com/tung-nd/stormer.

Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Tung Nguyen' Rohan Shah Hritik Bansal'
Troy Arcomano
Romit Maulik
Veerabhadra Kotamarthi* Ian Foster²
Sandeep Madireddy
Aditya Grover' *Penn State University
UCLA ²CMU AArgonne National Laboratory
Abstract
Weather forecasting is fundamental problem for anticipating and mitigating the impacts of climate change. Recently, data-driven approaches for weather fore- casting based on deep learning have shown great promise, achieving accuracies that are competitive with operational systems. However, those methods often employ complex, customized architectures without sufficient ablation analysis, making it difficult to understand what truly contributes their success. Here we introduce Stormer, a simple transformer model that achieves state-of-the-art performance weather forecasting with minimal changes to the standard trans- former backbone. We identify the key components Stormer through careful empirical analyses, including weather-specific embedding, randomized dynam- ics forecast, and pressure-weighted loss. At the core of Stormer is a random- ized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. During inference, this allows us to produce multi- ple forecasts for target lead time and combine them obtain better forecast accuracy. On WeatherBench 2, Stormer performs competitively at short to medium- range forecasts and outperforms current methods beyond days, while requiring orders-of-magne maggiuu less training data and compute. Additionally, we demonstrate Stormer's 's favorable scaling properties, showing consistent improvements with increases in model size and training tokens. Code and checkpoints are available at https://github woom/ung--aoi ttngtnd/gg///ggg//et//eo///de stormer
26 December 2020 08:00 UTC
31 Decobbr 2029 08:01 UTC
31 Decmmer 2021 08:0 UTC
1
Initial Conditions
5-day Forecast
Ground Truth
Figure 1: Illustaaioon an example 3-day forecast of near-surface wind speed (color-fill) and mean sea level pressure (contours). On December 31, 2020, an extratropical cyclone impacted Alaska setting new North Pacific low-pressure record. We evaluate the ability of Stormer predict this record-breaking event days in advance. Using initial conditions from 0000 UTC, 26 December 201 1, Stormer successfully forecasts the location and strength of this extreme event with great accuracy.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Weather forecasting is fundamental problem for science and society. With increasing concerns about climate change, accurate weather forecasting helps prepare and recover from the effects of natural disasters and extreme weather events, while serving as an important tool for researchers to better understand the atmosphere. Traditionally, atmospheric scientists have relied on numerical weather prediction (NWP) models (2]. These models utilize systems of differential equations describing fluid flow and thermodynamics, which can be integrated over time to obtain future forecasts [29] 2). Despite their widespread use, NWP models suffer from many challenges, such as parameterization errors of important small-scale physical processes, including cloud physics and radiation 44). Numerical methods also incur high computation costs due to the complexity of integrating a large system of differential equations, especially when modeling at fine spatial and temporal resolutions. Furthermore, NWP forecast accuracy does not improve with more data, as the models rely on the expertise of climate scientists refine equations, parameterizations, and algorithms 30
To address the above challenges of NWP models, there has been an increasing interest in data-drives approaches based on deep learning for weather forecasting
48). The key idea involves training deep neural networks to predict future weather conditions using historical data, such as the ERA5 reanalysis dataset [16, 17 40 41. Once trained, these models can produce forecasts in few seconds, as opposed to the hours required by typical NWP models. Because of the similar spatial structure between weather data and natural images, early works in this space attempted to adopt standard vision architectures such as ResNet [39 8] and UNet [49] for weather forecasting, but their performances lagged behind those of numerical models. However, significant improvements have been made in recent years due to better model architectures and training recipes, and increasing data and compute [22 35
42
26 5 7 Pangu-Weate Weather B3. a 3D Earth-Specific Transformer model trained on 0.25 data x 1440 grids), was the first model to outperform operational IFS [47] Shortly after, GraphCast 26 scaled up the graph neural network architecture proposed by Keisler [22 to 0.25 data and showed improvements over Pangu- weather. Despite impressive forecast accuracy, existing methods often involve highly customized neural network architectures with minimal ablation studies, making it difficult to identify which components contribute to their success. For example, it is unclear what the benefits of 3D Earth-Specific Transformer over a standard Transformer are, and how critical the multi-mesh message-passingg in GraphCast is to its performance. A deeper understanding, and ideally a simplification, of these existing approaches is essential for future progress the field. Furthermore, establishing a common framework would facilitate the development of foundation models for weather and climate that extend beyond weather forecasting [32
32
3
In this paper, we show that simple architecture with a proper training recipe can achieve state-of-the- art performance. We start with standard vision transformer (ViT) architecture, and through extensive ablation studies, identify the three key components to the performance of the model: (1) a weather- specific embedding layer that transforms the input data to a sequence of tokens by modeling the interactions among atmospheric variables; (2) randomized dynamics forecasting objective that trains the model to predict the weather dynamics random intervals; and (3) a pressure-weighted loss that weights variables at different pressure levels in the loss function to approximate the density at each pressure level. During inference, our proposed randomized dynamics forecasting objective allows a single model to produce multiple forecasts for a specified lead time by using different combinations of the intervals for which the model was trained. For example, one can obtain a 3-day forecast by rolling out the 6-hour predictions 12 times or 1 12-hour predictions 6 times. Combining these forecasts leads to significant accuracy improvements, especially for long lead times. We evaluate our proposed method, namely Scalable transformers for weather forecasting (Stormer), on WeatherBench 2 [41], a widely used benchmark for data-driven weather forecasting. Stormer achieves competitive forecast accuracy of key atmospheric variables for 1-7 days and outperforms the state-of-the-art beyond 7 days. Notably, Stormer achieves this performance by training on more than 5x lower-resolution data and orders-of-magninn shows that the performance of Stormer improves consistently with increases in model capacity and data size, demonstrating the potential for further improvements.
fewer GPU hours compared to the baselines. Finally, our scaling analysis
2 Background and Preliminaries
Given a dataset D {X,), of historical weather data, the task of global weather forecasting is to forecast future weather conditions Xx e V xW given initial conditions Xo E V XXW
2

---

X
Xr
fo
C
fo
Xx
X²
Xa
X
6t2
fo
fo
fo
fo
X
X
T
X
X
6t1
Direct forocasting
Continuos forecasting
Henative forecasting
Randomized incrutvve forecasting
Figure 2: Different approaches to weather forecasting. Direct and continuous methods output forecasts directly, but continuous forecasting is adaptable to various lead times by conditioning on T. Iterative forecasting generates forecasts at small intervals 8t, which are rolled out for the final forecast, Our proposed randomized iterative forecasting combines continuous and iterative methods.
in which T' is the target lead time, e.g., 7 days; V is the number of input and output atmospheric variables, such as temperature and humidity; and H W is the spatial resolution of the data, which depends on how densely we grid the globe. This formulation is similar to many image-to-image tasks in computer vision such as segmentation or video frame prediction. However, unlike the RGB channels in natural images, weather data can contain up to 100s of channels. These channels represent actual physical variables that can be unbounded in values and follow complex laws governed by atmospheric physics. Therefore, the ability to model the spatial and temporal correlations between these variables is crucial to forecasting.
There are three major approaches to data-driven weather forecasting. The first and simplest is direct forecasting, which trains the model to directly output future weather X,
(X)) for each
Since the weather is chaotic system, forecasting the future directly for large T is challenging, which may explain the poor performances of these early models. Moreover, direct forecasting requires training neural network for each lead time, which can be computationally expensive when the number of target lead times increases. To avoid the latter issue, continuous forecasting uses T' as an additional input: X fo (o. T), allowing a single model to produce forecasts at any target lead time after training. MetNet [43] 22 II employed the continuous approach for nowcasting at different lead times up 24 hours, WeatherBench [39 considered continuous forecasting one of the baselines, and ClimaX [32] used this approach for pretraining. However, since this approach still attempts to forecast future weather directly, it suffers from the same challenging problem of forecasting the chaotic weather in one step. Finally, iterative forecasting trains the model to produce forecasts at small interval X5c = fe(Xo), in which öt is typically from 6 to 24 hours. To produce longer-horizon forecasts, we roll out the model by iteratively feeding its predictions back in as input. This is common paradigm in both traditional NWP systems and the two state-of-the-an deep learning methods, Pangu- Weather and GraphCast. One drawback of this approach is error accumulation when the number of rollout steps increases, which can be mitigated by multi-step loss function [22
target lead time T. Most early works in the field adopt this approach [T] 42 48
7 iterative forecasting, one can forecast either the weather conditions X or the weather dynamics Ast Xat Xo. and X can be recovered by adding the predicted dynamics to the initial conditions. this work, we adopt the latter approach, which we refer to as iterative dynamics forecasting. We show empirically that approach achieves superior performance relative to the former approach in Section 4.2) Figure summarizes these different approaches.
3 Methodology
We introduce Stormer, skillful method for weather forecasting, and show that simple architecture can achieve competitive forecast performances with a well-designed franework. We first present the overall training and inference procedure of Stormer, then describe the model architecture we use in practice. Section 4.2 empirically demonstrates the importance of each component of Stormer.
3

---

72m
12m 10
2.50
122
2.3
1.13
3.50
1.2
1.2
0.75
Lead Urme (days)
Lead Lime (oayg)
Load time (Okyss)
VIT Errbesinng
Weather brbbood
AdGEN0 Embbudding
AUUUN
(a) Time interval comparison.
(b) Patch embedding comparison.
(c) Time embedding comparison.
Figure 3: Preliminary results on forecasting surface temperature that led to the design choices of Stormer: (a) Different intervals better different lead times, (b) Weather-specific embedding is superior to standard ViT embedding, and (c) Adaptive layer norm outperforms additive embedding.
3.1 Training
We adopt the iterative approach for Stormer, and train the model to forecast the weather dynamics Ast Xst - Xp. which is the difference between two consecutive weather conditions, Xo and Xst. across the time interval 51, A common practice in previous works [2] 226 is to use small fixed value of such as 6 hours. However, as we show in Figure 3a while small intervals tend to work well for short lead times, larger intervals excel at longer lead times (beyond days) due less error accumulation. Therefore, having a model that can produce forecasts at different intervals and combine them in effective manner has the potential improve the performance of single-interval models. This motivates our randomized dynamics forecasting objective, which trains Stormer to forecast the dynamics random intervals öt by conditioning on öt:
(0) (X,e [1],x(Xn ōt) Aa
(1)
in which P(öt) is the distribution of the random interval. our experiments, unless otherwise specified, P(ot) is uniform distribution over three values öt ~ 24(6, 12, 24). These three time intervals play an important role atmospheric dynamics. The 6 and 12-hour values help encourage the model to learn and resolve the diurnal cycle (day-night cycle), one of the most important oscillations the atmosphere driving short-term dynamics (e.g., temperature over the course of day). The 24-hour value filters the effects of the diurnal cycle and allows the model to learn longer, synoptic-scale dynamics which are particularly important for medium-range weather forecasting [19].
From a practical standpoint, this randomized objective provides two benefits. First, randomizing öt enlarges the training data, serving as data augmentation. Second, it allows single trained model to generate various forecasts for specified lead time T by creating different combinations of intervals ót that sum T. For example, forecast 7 days ahead, one could use 12-hour forecasts 14 times or 24-hour forecasts 7 times. Our experiments show that combining these forecasts is crucial for achieving good accuracy, especially for longer lead times. While both our approach and the continuous approach use the time interval instead of direct forecasting. This avoids the challenge of directly modeling chaotic weather and offers more flexibility for combining different intervals at test time.
additional input, we perform iterative forecasting
3.1.1 Pressure-weighted loss
Due to the large number of variables being predicted, we use a physics-based weighting function to weigh variables near the surface higher. Since each variable lies on a specific pressure level, we can use pressure as a proxy for the density of the atmosphere at each level. This weighting allows the model prioritize near-surface variables, which important for weather forecasting and have the most societal impact. The final objective function that we use for training is:
w L( - -A
( E VHW ΣΣΣ
(2)
The expectation over 8t, Xo. and X8s which we omit for notational simplicity. In this equation, w(v) is the weight of variable v, and L(i) is the laiihue-weighting factor commonly used in previous

---

works to account for the non non-uniformity when we grid the spherical globe 40 22 35 32 3 26 The pressure-wiighee loss was first introduced by GraphCast [26], and we show that also helps with a different architecture.
3.1.2 Multi-step finetuning
To produce forecasts lead time beyond the training intervals, roll out the model several times. Since the model's forecasts are fed back as input, the forecast error accumulates as we roll out more steps. To alleviate this issue, finetune the model on multi-step loss function. Specifically, for each gradient step, we roll out the model K times, and average the objective (2) over the K steps:
V H w(v)
ΣΣΣΣ (()) E KVHW 1mm1 uill
köt
(3)
In practice, we implement a three-phase training procedure for Stormer. In the first phase, we train the model to perform single-step forecasting, which is equivalent to optimizing the objective in (2) In the second and third phases, we finetune the trained model from the preceding phase with 4 and K = 8, respectively. We use the same sampled value of the interval öt for all K steps. We tried randomizing öt at each rollout step, but found that doing so destabilized training as the loss value at each step is of different magnitudes, hurting the final performance of the model. Multi-step finetuning was used in FourCastNet [35] and also adopted in more recent works [2] 26
3.2 Inference
At test time, Stormer can produce forecasts at any time interval öt used during training. Thus the model can generate multiple forecasts for target lead time by creating different combinations of öt that sum to T'. We consider two inference strategies for generating forecasts:
Homogeneous In this strategy, only consider homogeneous combinations of öt, i.e., combinations
with just one value of öt. For example, for T = 24 we consider [6, 6, 6, 6]. [12, 12], and [24].
Best m in We generate n different, possibly heterogeneous combinations of 51, validate each combination, and pick m combinations with the lowest validation losses for testing.
The two strategies offer trade-off between efficiency and expressivity. The homogeneous strategy only requires running three combinations for each lead time T', while best m provides greater expressivity. Upon determining these combinations and executing the model rollouts, we obtain the final forecast by averaging the individual predictions. This approach achieves similar effect to ensembling in NWP, where multiple forecasts are generated by running NWP models with different perturbed versions of the initial condition
(27). target lead times extend beyond 5-7 days and individual forecasts begin to diverge due to the chaotic nature of the atmosphere, averaging these forecasts is a Monte Carlo integration approach to handle this sensitivity to initial conditions and the uncertainty in the analyses used as initial conditions 31. We note that our inference strategy is distinguished from that used in Pangu- -eather: while Pangu-Weather Weather trains separate model for each time interval ôt, we train a single model for all öt values by conditioning on öt. Additionally, while Pangu- Weather relies on a single combination of intervals to minimize rollout steps, our method improves forecast accuracy by averaging multiple forecasts derived from diverse combinations.
3.3 Model architecture
We instantiate the framework in Section 3.1 with simple Transformer [45 [ -based architecture. Due to the similarity of weather forecasting to various dense prediction tasks in computer vision, one might consider applying Vision Transformer (ViT) [TO] for this task. However, weather data is distinct from natural images, primarily due to its significantly higher number of input channels, representing atmospheric variables with intricate physical relationships. For example, the wind fields are closely related to the gradient and shape of the geopotential field, and redistribute moisture and heat around the globe. Effectively modeling these interactions is critical to forecast accuracy.
3.4 Weather-speeiffic embedding
The standard patch embedding module in VIT, which uses linear layer for embedding all input channels within patch into vector, may not sufficiently capture the complex interactions among
5

---

input atmospheric variables. Therefore, we adopt for our architecture a weather-specific embedding module, consisting of two components, variable tokenization and variable aggregation.
Variable tokenization Given an input of shape V x H x W, variable tokenization linearly embeds sequence of shape (H/p) x (W/p) x D, in which p is the patch size and D is the hidden dimension. We then concatenate the output of all variables, resulting in sequence of shape (p) x (W/p) x V D.
each variable independently
Variable aggregation We employ a single-layer cross-attention mechanism with a learnable query vector to aggregate information across variables. This module operates over the variable dimension on the output of the tokenization stage to produce a sequence of shape [H x (W/p) x D. This module offers two primary advantages. First, it reduces the sequence length by factor of V. significantly alleviating the computational cost as we use transformer to process the sequence. Second, unlike standard patch embedding. the cross-attention layer allows the models to learn non-linear relationships among input variables, enhancing the model's capacity to capture complex physical interactions. We present the complete implementation details of the weather-specific embedding Section B
Figure 3b shows the superior performance of weather-specific embedding to standard patch embedding at all lead times from 1 to 10 days. A similar weather-specific embedding module was introduced by ClimaX (3) to improve the model's 's flexibility when handling diverse data sources with heterogeneous input variables. We show that this specialized embedding module outperforms the standard patch embedding even when trained on a single dataset, due to its ability to model interactions between atmospheric variables through cross-attention effectively.
3.4.1 Stormer Transformer block
Following veather-specifee embedding, the tokens are processed by a stack of transformer blocks 451 In addition to the input Xn. the block also needs to process the time interval öt. We do this by replacing the standard layer normalization module used in transformer blocks with adaptive layer normalization
(adaLN) (37]. In AdaLN, instead of learning the scale and shift parameters y and / as independent parameters of the network, we regress them with an one-layer MI.P from the embedding of St. Compared to ClimaX 332 which only adds the lead time embedding to the tokens before the first attention layer, a is applied to every transformer block, thus amplifying the conditioning signal. Figure 3c| shows the consistent improvement of adaLN over the additive lead time embedding used in ClimaX. Adaptive layer norm was widely used in both GANs (21 4 and Diffusion [9] 36) to condition additional inputs such as time steps or class labels. Figure 7 illustrates Stormer's architecture. We refer to 32 for illustrations of the weather speciffc embedding block.
4 Experiments
We compare Stormer with state-of-the-arr weather forecasting methods, and conduct extensive ablation analyses to understand the importance of each component in Stormer. We also study Stormer scalability by varying model size and the number of training tokens. We conduct all experiments on WeatherBench 2 (WB2) 41,, a standard benchmark for data-driven weather forecasting.
Data: We train and evaluate Stormer on the ERA5 dataset from WB2, which is the curated version of the ERA5 reanalysis data provided by ECMWF [T7]. In its raw form, ERA5 contains hourly data from 1979 to the current time at 0.25 (721x x 1 1440 grids) resolution, with different atmospheric variables spanning 137 pressure levels plus the Earth's 's surface. WB2 downsamples this data to 6-hourly with 13 pressure levels and provides different spatial resolutions. In this work, we use the 1.40625 (128x256 grids) data. We use four surface-level variables - 2-meter temperature (T2m), 10-meter U and V components of wind (U10 and V10), and Mcan sea-level pressure (MSLP), and five atmospheric variables - Geopotential (Z), Temperature (T), U and V components of wind (U and V), and Specific humidity (Q), each at 13 pressure levels (50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 850, 925, 1000). We use 1979 to 2018 for training. 2019 for validation, and 2020 for testing.
Stormer architecture: For the main comparison in Section 4.1 we report the results of our largest Stormer model with 24 transformer blocks, 1024 hidden dimensions, and a patch size of 2, which equivalent to ViT-I. except for the smaller patch size. We vary the model size and patch size in the scaling analysis. For the remaining experiments, we report the performance of the same model as for the main result, but with larger patch size of 4 for faster training.

---

T2m (K)
U10m (mVs)
V10m (m/s)
4-
2.5
2.0
1.5
1.0
6
10
12
14
10
12
14
10
MSLP (Pa)
Z500 (m²)s?)
T850 (K)
600
RMSE
2
1.0
12
14
10
12
14
2
10
12
14
Q700 (g/kg)
U850 (m/s)
V850 (mvss)
2.00
1.55
150
125
3
2
100
32 14
300
12
14
2
10
12
14
Load time (days)
Climatoloyy
Pangu-Weather
GraphCast
Stormer
Figure 4: Global forecast results of Stormer and the baselines. We show the latitude-weighted RMSE for select variables. Stormer is on par or outperforms the baselines for the shown variables. During the later portion of the forecasts, Stormer gains day of forecast skill with respect to climatology compared to the next best deep learning model. We note that Stormer was trained on much lower resolution data (1. (1.40625)) compared to Pangu- Weather (0.259) and GraphCast (0.25%).
Training: For the main result in Section we train Stormer three phases, as described in Section 3.1.2] We train the model for 100 epochs for the first phase, 20 epochs for the second, and 20 epochs for third. We perform early stopping on the validation loss aggregated across all variables. and evaluate the best checkpoint of the final phase the test set. For the remaining experiments, only train Stormer for first phase due to computational constraints.
Evaluation: We evaluate Stormer and two deep learning baselines on forecasting nine key variables: T2m, U10, V10. MSLP. Z500, T850, Q700, U850, and V850. These variables are also used report the headline scores in WB2. For each variable, we evaluate the forecast accuracy at lead times from days, using the weiihtee root-mean-square error (RMSE) metric. For the main results, we use best m in inference for rolling out Stormer as it yields the best result, with 32 and 128 chosen randomly from all possible combinations. For the remaining experiments, we use homogeneous inference for faster evaluations. We provide results on the non-ensemble version of Stormer, probabilistic metries with IC perturbations, a comparison between two inference strategies, and additional ablation studies in Appendix C
4.1 Comparison with State-of-the-art models
We compare the forecast performance of Stormer with Pangu-Weather Weather S and GraphCast (26] two leading deep learning methods for weather forecasting. Pangu-Weather employs a Earth-Specific Transformer architecture trained on the same variables as Stormer, but with hourly data and higher spatial resolution 0.25*. GraphCast is graph neural network that was trained on 6-hourly FRA5 data at 0.25%, using 37 pressure levels for the atmospheric variables, and two additional variables,
7

---

17m
1.2
1.5
1.21
1.0
0.5
Lead time stayO
Lead mm Idayn)
Load Kma 153y5)
5-65
5,12
24
Stommr
Unereighted
Waigttd
Xy ferecast
Impact of randomized forecasting.
Impact of weighted loss.
(c) Absolute vs. dynamics forecast.
Figure 5: Ablation studies showing the importance of different components in Stormer: (a) Random- ized forecasting, (b) Fressure-weighter loss, and (c) Dynamics forecasting.
and GraphCast are iterative methods. GraphCast operates at 6-hour intervals, while Weather uses four distinct models for 1-, 3-, 6-, and 24-hour intervals, and combines them produce forecasts for specific lead times. We include Climatology as a simple baseline. We also compare Stormer with IFS HRES, the state-of-the-an numerical forecasting system, and IFS ENS (mean), which is the ensemble version of IFS. Since WB2 does not provide forecasts of these numerical models beyond 10 days, we defer the comparison against these models Appendix C.T
total precipitation and vertical wind speed. Both
Figure 4 evaluates different methods on forecasting nine key weather variables at lead times from to 14 days. For short-range, 1-5 day forecasts, Stormer's accuracy is par with exceeds that of Pangu-Weather, Weather, but lags slightly behind GraphCast. At longer lead times, Stormer excels, Pangu-Weather
Results:
consistently outperforming both Pangu- Weather and GraphCast from day 6 onwards by large margin. Moreover, the performance gap increases as we increase the lead time. At 14 day forecasts, Stormer performs better than GraphCast by 10% 20% across all 9 key variables. Stormer is also the only model in this comparison that performs better than Climatology at long lead times, while other methods approach or even do worse than this simple baseline. The model's superior performance long lead times is attributed to the use randomized dynamics training, which improves forecast accuracy by averaging out multiple forecasts, especially when individual forecasts begin diverge.
Moreover, we also note that Stormer achieves this performance with much less compute and training data compared to the two deep learning baselines. We train Stormer 6-hourly data of 1 1.40625e with 13 pressure levels, which is approximately 190x less data than Pangu-Weather's at 0.25 and 90x less than that used for GraphCast, which also uses 6-hourly data but at 0.25 resolution with 37 pressure levels. The training of Stormer was completed under 24 hours on 128 A100 GPUs. In contrast, Pangu- Weather took 60 days to train four models 192 V100 GPUs, and GraphCast required 28 days on 32 TPUv4 devices. This training efficiency will facilitate future works that build upon our proposed framework.
Weather's 's hourly data
4.2 Ablation studies
We analyze the significance of individual elements within Stormer by systematically omitting one component at time and observing the difference in performance.
Impact of randomized forecasts: We evaluate effectiveness our proposed randomized iterative forecasting approach. Figure 5a the forecast accuracy surface temperature of Stormer and three models trained with different values öt. Stormer consistently outperforms all single- interval models at all lead times, and the performance gap widens the lead time increases. We attribute this result to the ability of Stormer to produce multiple forecasts and combine them to improve accuracy. We note that Stormer achieves this improvement with computational overhead compared to the single-interval models, as the different models share the same architecture and were trained for the same duration.
Impact of pressure-weighted loss: Figure 5b shows the superior performance Stormer when trained with the pressure-weigheee loss. Intuitively, the weighting factor prioritizes variables that are nearer to the surface, as these variables are more important for weather forecasting and climate science.

---

72m
T2m 80
25
2.0
0.-
Load Lime (lays!
Load time (days)
Starmer-S
Stormest
Stemenk
p-16 p-8
Figure 6: Stormer improves consistently with larger models (left) and smaller patch sizes (right).
Dynamics vs. absolute forecasts: We justify our decision forecast the dynamics Ast by comparing with a counterpart that forecasts Xot- Figure [5c] shows that forecasting the changes in weather conditions (dynamics) consistently more accurate than predicting complete weather states. One possible explanation for this result is that it is simpler for the model to predict the changes between two consecutive weather conditions than the entire state of the weather; thus, the model can focus on learning the most significant signal, enhancing forecast accuracy.
4.3 Scaling analysis
We examine Stormer's scalability in terms of model size and training tokens. We evaluate three variants - Stormer-S, Stormer-B, B, and Stormer-L, with parameter counts similar to ViT-S, ViT-B, and ViT-L, respectilly. To understand the impact of training token count, we vary the patch size from to 16, quadrupling the training tokens each time the patch size is halved. Figure(Jshowe improvement in forecast accuracy with larger models, and the performance gap widens with increased lead time. Since we do not perform multi-step fine-tuning for these models, minor performance differences at short intervals may magnify over time. While multi-step fine-tuning could potentially reduce this gap, it is unlikely eliminate it entirely. Reducing the patch size also improves the performance the model consistently. From a practical view, smaller patches mean more tokens and consequently more training data. From climate perspective, smaller patches capture finer weather details and processes not evident in larger patches, allowing the model to more effectively capture physical dynamics that drive weather patterns.
significant
5
Related Work
Deterministic weather forecasting Deep learning offers a promising approach to weather fore-
casting due its fast inference and high expressivity. Early efforts (T 48] attempted training simple architectures on small weather datasets. To facilitate progress, WeatherBench [40] provided standard datasets and benchmarks, leading to subsequent works that trained Resnet (15) and UNet architectures [49] for weather forecasting. These works showed the potential of deep learning but still displayed inferior accuracy to numerical systems. However, significant improvements have been made in the last few years. Keisler [22] proposed a graph neural network (GNN) that per- forms iterative forecasting with 6-hour intervals, performing comparably with some NWP models. FourCastNet 35 trained an adaptive Fourier neural operator and was the first neural network to run on 0.25 data.
3. with 3D Earth-Specific Transformer design, trained on high-resolution data, surpassed the benchmark IFS model. Following this, GraphCast (26 scaled up Keisler's GNN architecture to 0.25%, achieving even better results than Pangu-Weather. Weather. FuXi [6] was a subsequent work that trained Swin V2 (28) 0.25% data and showed improvements GraphCast at long lead times. However, FuXi requires finetuning multiple models specialized for different time ranges, increasing model complexity and computation. Feng [5] was a concurrent
work with FuXi that also focused on improving long- oon-hhoizznn forecasts, but has not revealed complete model architecture and training details. ClimODE (46) introduced physical inductive biases to provide better interpretability but was empirically inferior to existing methods.
Probabilistic weather forecasting In addition to high accuracy, desired ability of weather forecasting model is to quantify forecast uncertainty. One common approach to achieve this is to combine existing architecture with a probabilistic loss function. Gencast [38) was one of the first

---

works in this direction, combining the Graphcast architecture with diffusion objective [88 21]. followed by Graph-EFM [33]. which combined hierarchical variant of Graphcast with the VAE objective 24 This approach allows the model to generate multiple forecasts and estimate uncertainty after training. In an orthogonal approach, NeuralGCM [25] proposed a hybrid forecasting system that combined a differentiable dynamical core with MI components for end-to-end training. The dynamical core allows the method to leverage powerful general circulation models and generate forecast ensembles via IC perturbations similar to NWP. However, the dynamical core in NeuralGCM more computationally expensive than auwaaddppsssngg neural network and can limit the method's performance with imperfect circulation model.
6
Conclusion and Future Work
This work proposes Stormer, simple yet effective deep learning model for weather forecasting, We demonstrate that standard vision architecture can achieve competitive results with carefully designed training recipe. Our novel approach, randomized iterative forecasting, trains the model to forecast at different time intervals, enabling produce and combine multiple forecasts for each target lead time for better accuracy. Experiments show Stormer's 's competitive accuracy in short-range forecasts and exceptional performance beyond days, all with significantly less data and computing resources. Future research could explore using multiple forecasts to quantify uncertainty, randomizing other model components like input variables to increase variability and accuracy, and evaluating Stormer on higher- heslerone data and larger model sizes due to its favorable scaling properties.
7 Acknowledgments
AG acknowledges support from Google, Cisco, and Meta. SM supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, through the SciDAC-RAPIDS2 institute under Contract DE-ACO2-06CHI 1357. RM and VK are supported under a Laboratory Directed Research and Development (LDRD) Program Argonne National Laboratory, through U.S. Department Energy (DÔE) contract A-W.-CC0-HHHHHHH 1357. TA is supported by the Global Change Fellowship in the Environmental Science Division at Argonne National Laboratory (grant no. LDRD 2023-0236). RM acknowledges support from DOF-FOA-2499 "Data intensive scientific machine learning". : An award for computer time provided by the U.S. Department of Energy's (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program and Argonne Leadership Computing Facility Director's discretionary award. This research used resources from the Argonne Leadership Computing Facility, U.S. DOE Office of Science user facility at Argonne National Laboratory, which supported by the Office of Science of the U.S. DOE under Contract No. AE-AC02-06C
10

---

References
[1] Marcin Andrychowicz, Lasse Espeholt, Di Li, Samier Merchant, Alex Merose, Fred Zyda, Shreya Agrawal, and Nal Kalchbrenner. Deep learning for day forecasts from sparse observa- tions. arXiv preprint arXiv:2306.06073 2023.
[2] Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 525(7567):47-55, 2015.
[3] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3D) neural networks. Nature, 619(7970):533- 538, 2023.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv:180..10000 2018.
[5] Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, ling-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang. Rui Su, al. Fengwu: Pushing the skillful global medium-range maange weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02948. 2023.
[6] Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. Fuxi: a cascade machine learning forecasting system for 15-day global weather forecast. upj Climate and Atmospheric Science, 6(1):190, 2023. doi: 00.00888416112-023-0000122 URL
https://doi.orge
[7] Lei Chen, Xiaohui Zhong, Feng Zhang. Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. FuXi: cascade machine learning forecasting system for 15-day global weather forecast. arXiv preprint arXiv:2306. /2873, 2023.
[8] Mariana CA Clare, Omar Jamil, and Cyril Morcrette. Combining distribution-basee neural net- works to predict weather forecast probabilities. Quarterly Journal of the Royal Meteorological Society, 147(741)44337-435, 2021.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8778 2021.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini arXiv:2010. 1929, 2020.
[11] P. D. Dueben and P. Bauer. Challenges and design choices for global weather and climate models based on machine learning. Geoscientific Model Development, 11(10):3999-4009 2018. doi: 10.5194/gmd- 11-3999--199 URL https: /tt.///gd.././// copernicus....c.mocm. .og.uus..rr///..o www.lleroggrrrrreeeeeeeeeeee articles/ 11/3999/2018/
[12] Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz, Jason Hickey, et al. Deep learning for twelve hour precipitation forecasts. Nature communications, 13(1):1-10, 2022.
[13] William A Falcon. PyTorch lightning. Gitllub, 3, 2019.
[14] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Vir- tanen, David Coumapeau,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten van Kerkwijk, Matthew Brett, Allan Hal- dane, Jaime Fernández del Río, Mark Wiebe. Pearu Peterson, Pierre Gérard-Marcham Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis Oliphant. Array programming with NumPy. Nature, 585/7825) September 2020. doi: 00.1088/44188--222 2649-2. URL httphhtts:///oi.ooo
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[16] Hans Hersbach, Bill Bell, Paul Berrisford, Gionata Biavati, András Horányi. Joaquin Muñoz Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Iryna Rozum, Dinand Schepers, Adrian Simmons, Comell Soci, Dick Dee, and Jean-Noël Thépaut. ERA5 hourly data on single levels from 1979 to present. Copernicus Climate Change Service (C3S) Climate Data Dtore (CDS), 10(10.24381) 2018.
[17] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Elías H6lm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean-Noël Thépaut. The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Sociery, 146(730):1999-225 2049, 2020.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearul information processing systems, 33:6840-6851, 2020.
[19] James R. Holton. An Introduction to Dynamic Meteorology. International Geophysics Series. Elsevier Academic Press, Burlington, MA, 4 edition, 2004. ISBN 9780123540155
[20] Stephan Hoyer and Joe Hamman. xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software, 5(1):10, April 2017. doi: 10.5334/jors. 148.
[21] Tero Karras, Samuli Laine, and Timo Aila, A style- based generator architecture for generative adversarial networks. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[22] Ryan Keisler. Forecasting global weather with graph neural networks. arXiv preprint arXivv2202.07575. 2022.
[23] Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xfv:1412.6980, 2014.
[25] Dmitri Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Klöwer, James Lottes, Stephan Rasp, Peter Düben, al. Neural general circulation models for weather and climate. Nature, 632(8007):1000-1066 2024.
[26]
Remi Lam, Alvaro Sanchez-Gonzalezz Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Science, 00))c33)3)33 2023. doi: 10. 1226/scinnc..ai2333... Utthttps: http://wwwwciinneccce org/doi/abs 10. wwwp://////ciee...ee ad32336
[27] John M. Lewis. Roots of ensemble forecasting. Monthly Weather Review, 133(7): 1885, 2005. doi: http://bb..rrr/1.1155MwwW
URL httphttt////ouraall.. ametsoc org/
1.
[28] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[29] Peter Lynch. The origins of computer weather prediction and climate modeling. Journal of Computational Physics, 227(7)3431-3444 2008.
[30] Linus Magnusson and Frland Källén. Factors influencing skill improvements in the ecmwf forecasting system. Monthly Weather Review, 141(9):3122-3152 2013.
12

---

[31] N. Metropolis and S. Ulam. The Monte Carlo method. Journal of the American Statistical Association, 44:335, 1949.
[32] Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. ClimaX: A foundation model for weather and climate. arXiv preprint arXiv:2301. 10343, 2023.
[33] Joel Oskarsson, Tomas Landelius, Mare Peter Deisenroth, and Fredrik Lindsten. Probabilistic weather forecasting with hierarchical graph neural networks. arXiv preprint arXiv:2400.04759, 2024.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, segh-performanc deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[35] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadeneshelii Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. FourCastNet: global data- driven high-resolution weather model using udaptive Fourier neural operators. arXiv preprint arXiv:2202.11214 2022.
[36] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[37] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[38] Ilan Price, Alvaro Gonzalz, Ferran Alet, Timo Ewalds, Andrew El-Kadi, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. Gencast: Diffusion- based ensemble forecasting for medium-range weather. arXiv preprint 13X152231..15796, 2023.
[39] Stephan Rasp and Nils Thuerey. Data-driven medium-range weather prediction with resnet pretrained on climate simulations: new model for WeatherBench. Journal of Advances in Modeling Earth Systems, 13():e2020MS000440,, 2021.
[40] Stephan Rasp. Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. WeatherBench: benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(1 1xx2000MS00020,,, 2020.
[41] Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sha. WeatherBench 2: A benchmark for the next generation of data-driven global weather models. arXiv preprint ar arXiv:2308. 15560, 2023.
[42] Sebastian Scher. Toward data-driven weather and climate forecasting: Approximating simple general circulation model with deep learning. Geophysical Research Letters, 45(22):12-616 2018.
[43] Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Salimans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXi::2003.12140 2020.
[44] David J Stensrud. Parameterization Schemes: Keys Understanding Numerical Weather Prediction Models. Cambridge University Press, 2009.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
13

---

[46] Yogesh Verma, Markus Heinonen, and Vikas Garg. Climode: Climate and weather forecasting with thysics-informed neural odes. In The Twelfth International Conference Learning Representations.
[47] NP Wedi, P Bauer, W Denoninck, M Diamantakis, M Hamrud, C Kuhnlein, S Malardel, K Mogensen, Mozdzynski, and PK Smolarkiewicz. The modelling infrastructure of the Integrated Forecasting System: Recent advances and future challenges. European Centre for Medium-Range Weather Forecasts, 2015.
[48] Jonathan A Weyn, Dale R Durran, and Rich Caruana. Can machines learn to predict weather? Using deep learning predict gridded 500-hPa geopotential height from historical weather data. Journal of Advances in Modeling Earth Systems, 188):2680-2693, 2019.
[49] Jonathan A Weyn, Dale R Durran, and Rich Caruana. Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. Journal of Advances in Modeling Earth Systems, 1209::20020MS0020099 2020.
[50] Ross Wightman. PyTorch image models.
https://gitaubb.omm/rrgggaaa
pytorch- models 2019.
14

---

A Borader impacts
Weather and climate modeling crucial for understanding and tackling climate change. Creating better models based on deep learning could offer faster and cheaper alternatives to expensive numerical simulations. These models could improve weather predictions, extreme event forecasts, and climate projections. They might also help reduce the carbon footprint, better prepare for natural disasters, and enhance our knowledge of the Earth. However, relying only on deep learning models requires careful checks and monitoring, especially when predicting new or uncertain scenarios.
B Experiment details
B.1 Stormer architecture
Figure iillustrates the architecture of Stormer. The variable tokenization module tokenizes each variable of the input Xo E RY x separately, resulting in sequence of V x (H/p) x (W/p) tokens, where p is the patch size. The variable aggregation module then performs cross-attention over the variable dimension and outputs sequence of (H/p) (W/p) tokens. The interval öt is embedded and fed the Stormer backbone together with the tokens. The output of the last Stormer block is then passed through a linear layer and reshaped to produce the prediction Ast. Each Stormer block employs adaptive layer normalization to condition on additional information from őt. Specifically, the scale and shift parameters (m1, (1) and (12, (22 are output by an MLP which takes òl embedding as input. This MI.P network additionally outputs 01 and 02 to scale the output of the attention and fully connected layers, respectively.
A 6t
Scale
Linear and Reshape
Hully Connected
Scale,
N
Stormer block
Layer Norm
Scale
Aggregate
Embed
'T.. PL
Scale, shift
Tokenize
Layer Norm
MLP
Input embeddling
X
Figure 7: Stormer architecture. The initial condition goes through tokenization and aggregation, before being fed to stack of N Stormer blocks together with öt. Each Stormer block employs adaL.N for Ål conditioning.
In all experiments, the variable tokenization module is standard patch embedding layer usually used in VIT, and the aggregation module a single-layer multi-head cross-attention. The first embedding of 5t is linear layer, and the adaLN module in each block employs 2-layer MLP. For the main comparison with the current methods, we train a Stormer model with patch size of 2, 1024 hidden dimensions, and 24 Stormer blocks. For the scaling experiments, we vary the hidden dimensions, number of blocks, and patch size. For the rest of the ablation studies, we use patch size of 4, hidden dimension of 1024, and 24 blocks.

---

B.2
Training and evaluation details
B.2.1 Data normalization
Input normalization We compute the mean and standard deviation for each variable in the input across all spatial positions and all data points in the training set. This means each variable is associated with a scalar mean and scalar standard deviation. During training, we standardize each variable by subtracting from the associated mean and dividing it by the standard deviation.
Output normalization Unlike the input, the output that the model learns to predict is the difference between two consecutive steps. Therefore, for each variable, we compute the mean and standard deviation of the difference between two consecutive steps in the training set. What it means to be "consecutive" depends on the time interval 8t. 8t 6, we collect all pairs in training data that are 6-hour apart, compute the difference between two data points in each pair, and then compute the mean and standard deviation of these differences. Since we train Stormer with randomized őt, we repeat the same process for each value of öt.
B.2.2 Three-phase training
As mentioned in Section 3
we train Stormer in three phases with the following objective:
(()) E
1 KVHW
ΣΣΣΣ ((t)
A
(4)
where the number of rollout steps K is equal to 1, 4, and 8 in phase 1, 2, and 3, respectively. For phases and 3, we finetune the best checkpoint from the preceding phase.
B.2.3 Pressure weights
For pressure-Level variables, we assign weights proportionally to the pressure level of each variable. For 4 surface variables, assign = 1 for T2m and 0.1 for the remaining variables U10, V10, and MSLP. The surface weights were proposed by GraphCast [26] and we did not perform any additional hyperparameter tuning.
B.2.4 Optimization
For the 1st phase, train the model for 100 epochs. We optimize the model using AdamW [24] with learning rate of 5e 4, parameters (3,1 = 0.9, 32 = 0.95) and weight decay of le We used linear warmup schedule for 10 epochs, followed by cosine schedule for 90 epochs.
For the 2nd and 3rd phases, we train the model for 20 epochs with a learning rate of 5e 6 and 5e - 7, respectively. We used a linear warmup schedule for 5 epochs, followed by cosine schedule for 15 epochs. Other hyperparameters remain the same.
We perform early stopping for all phases, where the criterion is the validation loss aggregated across all variables at lead times of day, 3 days, and days for phases and 3, respectively. We save the best checkpoint for each phase using the same criterion.
B.2.5 Software and hardware stack
We use Py PoTch [34, Pytorch Lightning [3], timm [50] numpy 14 and xarray 20 for data processing and model training. We trained Stormer 128 40GB 100 devices. We leverage
mixed-precision training, Fully Sharded Data Parallel, and gradient checkpointing to reduce memory.
B.3 Evaluation protocol
As different models are trained on different resolutions of data, we follow the practice in WB2 to regrid the forecasts of all models to the same resolution of 1 1.40625 (128 x 256 grid points). We then calculate evaluation metries on this shared resolution. Similarly to WB2, we evaluate forecasts with initial conditions 00/ /12UTC for all days in 2020.

---

12mm (K)
1/1 Dim Im/e]
01 (mm))
23
2.0
15
1.0
14
MSSPP (Po)
2300 (m²)
18.50 (K)
1000
140
RMSE
400
10
18
12
14
14
07000 (p/leg)
Im/s)
VR50 (mpss)
2.00
1.75
8
20
15
12
14
12
24
Brme (days)
Climatology
Pangu-Weetter
GraphCast
Starmer
11.5 HRRES
LINS (meer))
Figure 8: Global forecast verification results Stormer and the baselines from to 14-day lead times. We show the latitude-eiihted RMSE for select variables. Stormer is par outperforms each of the benchmark models for the shown variables. During the later portion of the forecasts, Stormer significantly outperforms the current methods.
(mps)
(mik)
3.8
0.2
2.4
15
14
10
14
10
24
2.0
2.5
2.9
3.5
3.2
1.0
2.4
2.2
Lead time
GraphCast
Figure 9: Global forecast verification results of Stormer and the baselines from 1- 14-day lead times. We show the latinude-weighted ACC for select variables. Stormer is on par or outperforms each of the benchmark models for the shown variables. During the later portion of the forecasts, Stormer significantly outperforms the current methods.
3

---

Additional results
C.1 Complete comparison with SoTA models
Figure compares Stormer with both deep learning and numerical methods. We take IFS and IFS ENS from WB2 which is only available until day 10. Similar to its deep learning counterparts, Stormer achieves lower RMSE compared the IFS model for most variables, except for near-surface temperature (T2m) at initial lead times, and only performs slightly worse than IFS ENS. To the best of our knowledge, Stormer is the first model trained 1.40625 data surpass IFS.
Additionally, we compare Stormer and the baselines on latitude- -eettude-weigtted ACC, another common verification metric for weather forecast models. ACC represents the Pearson correlation coefficient between forecast anomalies relative to climatology and ground truth anomalies relative to climatology. ACC ranges from -1 to 1, where 1 indicates perfect correlation, and indicates perfect anti- correlation. We refer to WB2 41 for the formulation of ACC. Figure ? shows that similarly to RMSE, Stormer achieves competitive performance from 1 to days, and outperforms the baselines by a large margin beyond 6 days.
C.2 Impact of multi-step fine-tuning
We verify the importance multi-step fine-tuning by comparing Stormer after the 1st phase (K = 1) and after the 3rd phase (K 8). Figure 10 that multi-step fine-tuning significantly improves performance at long lead times.
12m (K)
U100m (mis)
V1.0m (m/s)
2.0
3.5
1.5
2.0
1.5
1.0
1.0
30
2
10
MSLP (Pa)
2500 (mi/js))
T850 (K)
5.0
600
500
RU5E
300
200
200
100
100
1.0
10
10
Q700 (g/kgg)
U850 (m/s)
V850 (m/s)
1.0
2.4
1.2
1.0
3
0.8
10
10
10
Load time (days)
- K-
Figure 10: Performance of Stormer without (K = 1) and with (K = 8) multi-step fine-tuning.
C.3 Non-ensemble performance Stormer
Even though our inference strategy can be considered ensemble forecasting, we note that it is much cheaper and more efficient than common techniques such as training multiple networks, dropout, or IC perturbations, only have to train single neural network and do not need extensive

---

T2m (5)
U10m (m/s]
V10m (n1/))
25
10
12
MSLF
2500 (m"
TE50
1009
-20
10
12
10
12
C700 (g/kg))
2.00
1.75
2.50
3.30
0.75
0.0
Lead time (days)
Climatoloyy
Pangu-Weatter
GraphCast
Stormer
-
IFS HRES
+
IFS ENS (mean)
Stormer Pangu Style
Figure 11: Non-ensemble version Stormer vs the baselines.
hyperparameter tuning. However, to provide more insights into the performance of Stormer, we additionally compare the non-ensemble version of Stormer with the baselines. Specifically, performed the Pangu-style inference, where we only used the 24-hour interval forecasts to roll out into the future, instead of combining different intervals. Figure IT shows that non-ensemble Stormer outperforms Pangu and performs competitively with Graphcast.
C.4 Probabilistic forecasting with IC perturbations
Since Stormer can produce forecast ensembles after training, can consider it probabilistic forecast system. However, our preliminary results suggested that different forecasts from Stormer are underdispersive and should not be used for uncertainty estimation. To make Stormer probabilistic forecast system, we need to introduce more randomization to the forecasts via IC perturbations. To do this, for each combination of intervals during the Best in inference, added 4 different noises sampled from Gaussian distribution, resulting in a total 128 ensemble members.
Figure 12 shows that IC perturbations improve the probabilistic metric significantly. but may hurt the deterministic performance at short lead times. Moreover, is difficult find an optimal noise level for the spread-skill ratio across different variables and lead times. We can further improve this by using a better noise distribution or variable-dependent and lead-tine-dependent noise scheduling, which we defer to future works.
C.5 Comparison of different inference strategies
Our inference strategies, Homogeneous and Best in n, provide tradeoff between efficiency and forecast accuracy. Figure 13 compares the performance of these two strategies across different variables at different lead times. The results show that Homogeneous performs competitively with Best m in n, while being much more efficient, requiring only 3 forward passes compared to
5

---

2503 (m))))
T8L00 05
12m 04)
600
2.50
3.00
1.7
24
200
13
140
1.0
300
232
14
3.5
0.3
035
14
(Dugga
Canse
0.05 10000
0.000
0.1
Usyaaology
Figure 12: Probabilistic performance of Stormer with different levels of IC perturbations.
12m
U10m (m/s)
V10m
2.25
3.5
2.00
1.75
2.5
2.5
1.5
0.75
3.50
(Pt)
2500 (m3/s?)
T850
600
400
400
2.0
100
200
200
1.0
(g/kg)
V850
1.2
0.0
0.6
(days)
Homogeneous
Best mn in
Figure 13: Comparsion of Homogeneous vs Best in inference strategies.

---

C.6 Qualitative results
We visualize forecasts produced by Stormer lead times from days days for key variables. All forecasts are initialized at OUTC January 26th 2020. Each figure illustrates one lead time, where each row is for each variable. The first column shows the initial condition, the second column shows the ground truth at that lead time, the third column shows the forecast, and the last column shows the bias, which is the difference between the forecast and the ground truth.
IC
Ground truth
(Forecast:
330
30
12m
1800
1300
1000
2500
$3500
$250e
50500
07000
200
Figure 14: 1-day lead time

---

(jan
3-day (Forecast
2590
40000
1000
-200
0700
T850
260
Figure 15; 3-day lead time

---

(jan
5-day (Forecast
2550
40000
1000
-200
T850
260
Figure 16: 5-day lead time

---

(jan
7-day (Forecast
2500
4000
1000
-200
TR50
200
200
Figure 17: 7-day lead time
10

---

Ijan
10-day
(Forecast
2550
4000
1000
-000
T850
240
Figure 18: 10-day lead time

---

Ijan
14-day (Forecast
V1
2550
40000
1000
-00
200
T850
240
Figure 19; 14-day lead time
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper introduces Stormer, scalable and skillful method for medium- range weather forecasting, Both the abstract and introduction reflect this contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Stormer currently operates on 1.40625 data which is lower-resolution than other leading methods. We mention this limitation in Section G
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide architectural, training, and evaluation details in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code, data, and checkpoints will be released publicly upon acceptance. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblicgguuuee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide architectural, training, and evaluation details in Section 3 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, reporting error bars is too computationally expensive and is not standard in data-driven weather forecasting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the compute resources for training Stormer in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of the paper in Section A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used and cited Weatherbench 2, an open-source benchmark for data-driven weather forecasting.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18


Reviews:
[{"review_id": "ahC5spkipf", "reviewer": "Reviewer_vUij", "summary": "The authors introduce a vision transformer-based method, Stormer, designed for medium-range weather forecasting. Ablations identify multiple important components of the method, including a weather-specific patch embedding, \"randomized\" dynamics forecast, and a pressure-weighted loss. The randomized forecasting component is similar to prior continuous models but complemented with iterative rollouts and an ensemble technique \"best m in n\" that exploits the multiple discretizations possible to forecast a specific lead time iteratively.\nThe experiments show competitive results in terms of RMSE and ACC against deep learning and physics-based baselines.", "questions": "- It would be very insightful to see an ablation of the different prediction approaches in weather forecasting. That is, continous, iterative, and randomized iterative forecasting. Exploring their (dis-)advantages would be beneficial and very interesting. This would also tie well with the paper's desire to carefully ablate important components of the design stack.\n- Can you expand on how the \"best m\" combinations are chosen? By combinations you mean e.g. (3h+6h=9 and 6h+3h=9h, but not 9*1h?) Are you choosing them on the validation set? How do you choose the \"n\" combinations that you validate? Also, some lead times won't have as many combinations (e.g. 6h only has one, 12h two etc.), right? That would seem worth mentioning as a potential limitation of this ensembling technique.\n- Why does the time embedding module predict the second scale parameter? Why no second shift? Did you ablate this choice? What if you remove this second time embedding part (or the first)?\n- For completeness, consider specifying the parameter sizes of your models in section 4.3. explicitly. Also, do you discuss the exact hyperparameters used for the smaller model sizes?\n- Is input normalization done per variable or per variable AND pressure level?", "limitations": "The authors have not adequately addressed the limitations, mentioning the spatial resolution as the only limitation of their method (but also claiming it to be an advantage in other parts of the paper). A more upfront discussion of (potential) limitations would be welcome. For example, there is no discussion on how the proposed cross-attention-based patch embedding might scale to higher spatial resolutions or how it might impact inference speed.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Tackles an important problem and achieves strong results on a popular weather forecasting benchmark, Weatherbench2.\n- The weather-specific patch embedding and randomized iterative ensemble forecasting are interesting methodological contributions.\n- Careful ablation of key design choices is insightful and valuable.\n- Clearly written, easy to follow, and a relatively simple approach are valuable for the community, especially if coupled with a good code release.\n\n********************* After rebuttal: Raising score from 5 to 6.", "weaknesses": "- The evaluation is somewhat unfair. \n\nThe paper is essentially using an ensembling technique but comparing against deterministic models. It is well known that ensembling improves ensemble-mean RMSE and ACC scores, especially for longer-range horizons (which is were Stormer, unsurprisingly, shines the most against the non-ensemble baselines). This can be actually clearly observed with the physics-based baselines IFS HRES (deterministic) and IFS ENS (ensemble mean) in Figures 8 and 9, where the ensemble mean shines the most on long-range horizons. Thus, a fairer comparison would be to either 1) ensemble the deterministic ML-based baselines (e.g. through input perturbations or with lagged ensembles as in [1]) or 2) show results of Stormer without ensembling (even a \"m=1 in n\" forecast would be much fairer than the way it is now). \n\nAdditionally, the proper way to evaluate an ensemble weather forecast is via probabilistic metrics such as the CRPS and spread-skill ratios. This should be included to properly assess the quality of the homogenous or \"best m in n\" ensembles. This probabilistic evaluation is actually supported by Weatherbench2, so it should be easy for the authors to extend their current evaluation with probabilistic metrics (+this will give you IFS ENS baseline up to 15 days ahead for free...).\n- On top of probabilistic metrics, it would be instructive to see an analysis of the generated spectra of Stormer.\n- The following is wrong: \"it is unclear (...) how critical the multi-mesh message passing in GraphCast is to its performance\". The authors seem to have missed section \"7.3.1. Multi-mesh ablation\" in the GraphCast paper.\n- I would like to see a more transparent discussion of the exact contributions of this work and a more careful contextualization with prior work. For example, in section 3.1 it would be good to be more candid about that 1) the objective is exactly the same as for a continuous forecasting model. The only difference between the two seems to be during inference time (and the different lead times used for training); 2) the overall train+inference method is essentially the same (neural architectures aside) as for Pangu-weather (PW) but with Stormer training one model for all lead times, while PW trains one per lead time. Noting the pros and cons of each would be useful to. Even more useful would be to have a crisp ablation for Stormer, where you drop the time-conditioning and train 3 separate models for each lead time (6, 12, 24hours). This would control for the differences in architecture between Stormer and PW, giving valuable insights into the pros and cons of each; 3) The pressure-weighted loss is taken from GraphCast but this is not properly referenced in the manuscript (this is not mentioned in section 3.1.1 that introduces it but only much later); 4) Same for multi-step fine-tuning, which is very common in prior work.\n- Relatedly, I think that the paper could benefit from discussing the related work more in-depth. For example, I seem to have missed any discussion that carefully compares Stormer with other (vision) transformer for weather forecasting methods such as Pangu-Weather, ClimaX, FengWu, etc.\n- Please include results on other variables than just T2M for your ablations. Different variables might behave quite differently... Also, this is a particularly unfortunate choice for Fig. 5b because the fact that the pressure-weighted loss improves T2M RMSE is fairly trivial since the pressure-weighted loss is very strongly upweighting the influence of T2M on the total loss relative to other variables.\n- Line 292: *\"We note that Stormer achieves this improvement with no computational overhead compared to the single-interval models, as the different models share the same architecture and were trained for the same duration.\"*. Is this true? To me it seems that at inference time there does exist a computational overhead due to the larger size of the ensemble...\n\n[1] A Practical Probabilistic Benchmark for AI Weather Models (https://arxiv.org/abs/2401.15305)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very detailed and constructive feedback, and for recognizing the strong results and technical contributions of Stormer. We answer each of the reviewer's concerns below.\n\n> The paper is essentially using an ensembling technique but comparing against deterministic models.\n\nWe would like to clarify that even though Stormer uses an ensembling technique, we consider it a deterministic model, and that’s why we compare it with deterministic baselines. The reason is that while Stormer can produce multiple forecasts for a given lead time at inference, we found these forecasts not diverse enough and should not be used for uncertainty estimation. This is shown by the under-dispersion of Stormer w.r.t to the spread-skill ratio in Figure 2 of our PDF. We will add this discussion to the updated paper.\n\n> Thus, a fairer comparison would be to either 1) ensemble the deterministic ML-based baselines (e.g. through input perturbations or with lagged ensembles as in [1]) or 2) show results of Stormer without ensembling (even a \"m=1 in n\" forecast would be much fairer than the way it is now).\n\nWe agree these comparisons would provide more insights into the performance of Stormer. To do this, we have compared the non-ensemble version of Stormer with the baselines. Specifically, we performed the Pangu-style inference, where we only used the 24-hour interval forecasts to roll out into the future (i.e., 1-day=24, 2-day=24+24, 3-day=24+24+24, etc.), instead of combining different intervals.\n\nFigure 1 in our PDF shows that non-ensemble Stormer outperforms Pangu and performs competitively with Graphcast. Moreover, we note that the ensembling technique in Stormer is much cheaper and easier to use than other methods such as training multiple networks, dropout, or IC perturbations, as we only have to train a single neural network and do not need extensive hyperparameter tuning. For better efficiency, one can always use the Homogeneous version of Stormer for inference, which only requires 3 forward passes and performs competitively to the Best m in n version, as shown in Figure 3 of our PDF. We will add this result and discussion to the updated paper.\n\n> Additionally, the proper way to evaluate an ensemble weather forecast is via probabilistic metrics such as the CRPS and spread-skill ratios. This should be included to properly assess the quality of the homogenous or \"best m in n\" ensembles.\n\nTo make Stormer a probabilistic forecast system, we need to introduce more randomization to the forecasts via IC perturbations. To do this, for each combination of intervals during the Best m in n inference, we added 4 different noises sampled from a Gaussian distribution, resulting in a total number of 128 ensemble members. Figure 2  in our PDF shows the RMSE and probabilistic metrics of Stormer with different standard deviations of the noise distribution.\n\nThe result shows that IC perturbations improve the probabilistic metric significantly, but may hurt the deterministic performance at short lead times. Moreover, it is difficult to find an optimal noise level for the spread-skill ratio across different variables and lead times. We can further improve this by using a better noise distribution or variable-dependent and lead-time-dependent noise scheduling, which we defer to future works. We will add these results and discussions to the updated paper.\n\n> On top of probabilistic metrics, it would be instructive to see an analysis of the generated spectra of Stormer.\n\nWe thank the reviewer for the suggestion and we will include this in the updated version of the paper. Due to the time constraint, we will defer this experiment to later. If the reviewer thinks this is crucial to assessing the paper, we will conduct this experiment during the discussion phase.\n\n> The authors seem to have missed section \"7.3.1. Multi-mesh ablation\" in the GraphCast paper.\n\nWe thank the reviewer for pointing this out. Our main message here is questioning whether we need a specialized neural network architecture for weather forecasting, or whether a standard architecture like transformers can work equally well. In this paper, we consider the Transformer architecture as a special reference due to its low inductive bias, good scaling properties, and great performance across different data modalities (text, image, audio, etc.). We will reword this part for better clarity in the updated version.\n\n> I would like to see a more transparent discussion of the exact contributions of this work and a more careful contextualization with prior work.\n\nWe initially wanted to describe our method first before connecting it with the related work, but we agree with the reviewer that it'd be better and more transparent to mention the prior works as we discuss each component of Stormer. We will update the paper accordingly. We answer the reviewer’s specific concerns as follows: 1) This is correct, but this seemingly small difference can make a huge difference in performance, as shown by the big gap between Stormer and ClimaX, as shown in Figure 4 of our PDF, 2) They are similar, but there are some nuanced differences. As the reviewer may have noticed, we train a single model for all intervals, while Pangu trains a separate model for each lead time. During inference, Pangu uses a single combination of intervals for each lead time that minimizes the rollout step, while we use a combination of them, 3) and 4) We agree with the reviewer and will add this discussion to the appropriate part of the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlr3jnGj7T", "reviewer": "Reviewer_uUm3", "summary": "The paper proposes a transformer-based model for weather prediction. Experiments show improvement in downstream predictions.", "questions": "-  What is the computational complexity of training and inference? Are there any ablation studies regarding the compactness and inference time studies?\n- How do you counter for boundary conditions? Do the variable and patch embeddings respect boundary conditions?\n- Can the model accommodate different lead-time resolution query points (t = 1hr, 19hr, 91hr, etc.) that differ from the dataset, showcasing its generalizability and applicability in modeling weather?\n- Does the method use standard 2D ViT or a 3D transformer as the stormer block?", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The presentation of the paper is clean, and the paper is easy to read and understand.\n- Some improvements over long-term weather forecasting.", "weaknesses": "- The paper has very limited novelty and is incremental. The transformer-based architecture is the same as ClimaX and other transformer-based methods; the variable embeddings are similar to what ClimaX proposes, multi-step fine-tuning is the same as FourCastNet, and pressure-weighted loss is by GraphCast. \n- The paper misses out on a ton of related works, such as  ClimODE (https://arxiv.org/abs/2404.10024), Neural GCM (https://arxiv.org/abs/2311.07222), GenCast (https://arxiv.org/abs/2312.15796), etc all seem to be missing and comparisons to them are missing.\n- Recent works have shown that continuous time methods with physical inductive biases and hybrid ML modeling have surpassed transformer-based methods such as ClimODE and Neural GCM,  providing uncertainty and interpretability. The proposed method does not offer any benefits to any of them. \n- Although the authors utilize randomized dynamic forecasting mechanism to have stability over predictions, they still restrict the whole method to output only point estimates rather than giving uncertainties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and the appreciation of the good presentation and good performance of Stormer. We answer each of the reviewer's concerns below.\n\n> The paper has very limited novelty and is incremental.\n\nWe acknowledge that some components in Stormer are similar to those of prior works. We also discussed the relation to them in the paper: Lines 204-205 discussed the variable embedding in ClimaX, Line 298 mentioned the pressure-weighted loss in Graphcast, and Lines 100-101 mentioned multi-step finetuning in previous works. We will update the paper to bring this discussion to the methodology section for better clarity.\n\nHowever, we would like to emphasize the differences and contributions of Stormer:\n- Stormer has a similar architecture to climax, but we use adaptive layer norm for time-conditioning, which we show is important to the performance in Figure 3c. Even with a fairly similar architecture, Stormer significantly outperforms ClimaX, showing the superiority of randomized iterative forecasting vs continuous pretraining + direct finetuning.\n- We introduce randomized iterative forecasting, a paradigm not explored in previous works. This allows any architecture (GNN, transformers, etc.) to gain performance improvements w.r.t deterministic metrics with only minor computation overhead (during inference only) compared to single-interval models.\n- Unlike previous works, we carefully ablate each component in Stormer to understand their importance in obtaining a good forecast performance. Via this paper, we show that it may not be necessary for a specialized neural network architecture, and a standard model like transformers can achieve state-of-the-art performance with careful design choices. We believe this extends the current understanding of data-driven weather forecasting, and is a valuable contribution to the community.\n\n> The paper misses out on a ton of related works\n\nThank you for pointing us to these works, we will discuss them in detail in the updated paper. Both Gencast and NeuralGCM were concurrent works and used much larger compute resources to run on higher-resolution data. Moreover, our goal in this paper is not to show SOTA (even though we could potentially achieve better performance with more compute and higher resolution data), but rather to show the power of simple and scalable approaches based on the transformers architecture, and what contributes to the performance of an ML weather forecasting model.\n\nFigure 4 in our PDF compares Stormer with NeuralGCM and ClimODE. We additionally include Stormer (IC noise), a version of Stormer with initial condition perturbations. Specifically, for each combination in the “Best 32 in 128” strategy, we sample 4 different noises from a Gaussian distribution with a standard deviation of 0.1 and add to the input, resulting in a total number of 128 ensemble members. The figure shows that Stormer outperforms deterministic NeuralGCM and performs competitively with NeuralGCM ENS. With IC perturbations, the gap between Stormer and NeuralGCM is negligible. ClimODE performs significantly worse than the other methods. While ClimeODE may improve by training on higher-resolution data, we do not believe it can close this huge gap. We will add this comparison to the updated paper. \n\n> Recent works have shown that continuous time methods with physical inductive biases and hybrid ML modeling have surpassed transformer-based methods such as ClimODE and Neural GCM, providing uncertainty and interpretability. The proposed method does not offer any benefits to any of them.\n\nWe respectfully disagree. NeuralGCM, ClimODE, and Stormer are different approaches and each has its pros and cons:\n- NeuralGCM is a hybrid model that combines a differentiable dynamical core with ML components for end-to-end training. While the dynamical core allows the method to leverage powerful general circulation models, it also has various drawbacks. First, to make predictions, the dynamical core in NeuralGCM has to solve discretized dynamical equations, which are more computationally expensive than forward-passing a neural network. Second, the performance of NeuralGCM is upper-bounded by the accuracy of the fixed dynamical core, while fully learnable models like Stormer continue improving with more data. This is a desirable property, given the scaling properties we have seen in Stormer and ClimaX.\n- ClimODE introduces physical inductive biases into deep learning models to improve the interpretability of their method. However, in terms of forecasting accuracy, as Figure 4 in our PDF shows, ClimODE is quite inferior to Stormer or other state-of-the-art methods.\n\n> They still restrict the whole method to output only point estimates rather than giving uncertainties.\n\nThis work focuses on deterministic forecasting and how to achieve state-of-the-art performance in deterministic metrics with a simple but scalable architecture like a transformer. However, we note that we can make Stormer a probabilistic model with IC perturbations. Figure shows the probabilistic performance of Stormer with different noise levels (standard deviations of the Gaussian noise distribution). IC perturbations significantly improve CRPS and SSR metrics of Stormer as well as deterministic performance at long lead times, but may hurt the accuracy at short lead times. Moreover, it is difficult to find an optimal noise level for the spread-skill ratio across different variables and lead times. We can further improve this by using a better noise distribution or variable-dependent and lead-time-dependent noise scheduling, which we defer to future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xV4WaXamd", "reviewer": "Reviewer_aebu", "summary": "This paper introduces a deep learning weather prediction model called Stormer. Stormer is vision transformer type network that employs various techniques to improve the network's performance in weather forecasting applications, including a \"weather-specific embedding\" that first processes each variable separately, and a technique for producing multiple weather scenarios for each input by using the model's ability to be tuned to various lead times. By using a lower spatial resolution than competing models, Stormer is much faster to train and inference than current state of the art models, while still achieving competitive performance at short-to-medium time scales and superior performance at longer time scales.", "questions": "How do you expect that the resolution difference between the Stormer and GraphCast/PanguWeather models affects the comparisons? Are you computing the losses at the native resolution of Stormer, downsampling the other models to it? Do you expect that it would change the results if you instead perform the comparison at the native resolution of GraphCast/PanguWeather and upsample the Stormer results to it?", "limitations": "The authors do briefly discuss the limitations of their approach and possible future directions. However, I think one point that gets glossed over by the authors is that due to its low resolution Stormer is unable to resolve weather features that models such as PanguWeather and GraphCast can resolve. Thus the performance gain of lower resolution comes at a cost of worse ability to resolve more localized weather features. It's also not clear how this impacts the model's representation of weather extremes, which are often found at small scales.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper shows that high-performance weather prediction DL models do not necessarily need to use high resolution, and in this way can save a large amount of computing power while still achieving good performance. The inclusion of adaptive layer normalization to enable variable lead times within the single model is also an interesting development. Furthermore I find the strategy for using the variable time stepping to average multiple forecasts an interesting technique.\n\nThe paper is very well written and the methodology was easy to understand from the description.", "weaknesses": "I do not see serious weaknesses with the paper. However, one point that could be improved is with the randomized forecast strategy introduced in the paper. PanguWeather is also able to make weather forecasts with variable lead times (although it achieves this with separately trained models). If I'm not mistaken, PanguWeather could then also be used to implement the randomized forecast strategy. It would be interesting to see this included in the model comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very detailed and constructive feedback, and for recognizing the technical contributions and good presentation of Stormer. We answer each of the reviewer's concerns below.\n\n> PanguWeather could then also be used to implement the randomized forecast strategy. It would be interesting to see this included in the model comparisons.\n\nWe agree that Pangu-Weather is capable of performing the randomized forecast strategy with the cost of training 3 separate models. We plan to include this experiment in the updated paper. \n\nAs a fairer comparison to Pangu, we have compared the non-ensemble version of Stormer with the baselines. Specifically, we performed the Pangu-style inference, where we only used the 24-hour interval forecasts to roll out into the future (i.e., 1-day=24, 2-day=24+24, 3-day=24+24+24, etc.), instead of combining different intervals.\n\nFigure 1 in our PDF shows that non-ensemble Stormer outperforms Pangu and performs competitively with Graphcast. Moreover, we note that the ensembling technique in Stormer is much cheaper and easier to use than other methods such as training multiple networks, dropout, or IC perturbations, as we only have to train a single neural network and do not need extensive hyperparameter tuning. For better efficiency, one can always use the Homogeneous version of Stormer for inference, which only requires 3 forward passes and performs competitively to the Best m in n version, as shown in Figure 3 of our PDF.\n\n> How do you expect that the resolution difference between the Stormer and GraphCast/PanguWeather models affects the comparisons? Are you computing the losses at the native resolution of Stormer, downsampling the other models to it? Do you expect that it would change the results if you instead perform the comparison at the native resolution of GraphCast/PanguWeather and upsample the Stormer results to it?\n\nWe downsampled the forecasts of Graphcast and Pangu-Weather to 1.40625deg and compared all methods in this resolution. This is the same strategy that was used in WeatherBench 2 to compare different models at different resolutions. We expect the comparisons to change if we instead upsample the Stormer forecasts to 0.25deg. While this is technically possible, we (and Weatherbench 2) opted not to do so because upsampling introduces additional error due to the complexity of extrapolating information from lower to higher resolution. In contrast, downsampling is straightforward, effectively averaging neighboring pixels without significant loss of information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank ACs for handling our paper and reviewers for their insightful comments and constructive feedback. The suggestions by the reviewers are very helpful and have added significant insights to the paper. We have responded to each review individually, and also **submitted a PDF file containing the figures and tables for additional experiments** we conducted during rebuttal. We summarize these experiments and their results here:
- (Figure 1) Non-ensemble vs the baselines: We compared the non-ensemble version of Stormer with the baselines. Specifically, we performed the Pangu-style inference, where we only used the 24-hour interval forecasts to roll out into the future (i.e., 1-day=24, 2-day=24+24, 3-day=24+24+24, etc.), instead of combining different intervals. Figure 1 in our PDF shows that non-ensemble Stormer outperforms Pangu and performs competitively with Graphcast. Moreover, we note that the ensembling technique in Stormer is much cheaper and easier to use than other methods such as training multiple networks, dropout, or IC perturbations, as we only have to train a single neural network and do not need extensive hyperparameter tuning. For better efficiency, one can always use the Homogeneous version of Stormer for inference, which only requires 3 forward passes and performs competitively to the Best m in n version, as shown in Figure 3 of our PDF.
- (Figure 2) Probabilistic performance of Stormer with IC perturbations: To make Stormer a probabilistic forecast system, we introduce more randomization to the forecasts via IC perturbations. To do this, for each combination of intervals during the Best m in n inference, we added 4 different noises sampled from a Gaussian distribution, resulting in a total number of 128 ensemble members. Figure 2 in our PDF shows the RMSE and probabilistic metrics of Stormer with different standard deviations of the noise distribution. The result shows that IC perturbations improve the probabilistic metric significantly, but may hurt the deterministic performance at short lead times. Moreover, it is difficult to find an optimal noise level for the spread-skill ratio across different variables and lead times. We can further improve this by using a better noise distribution or variable-dependent and lead-time-dependent noise scheduling, which we defer to future works.
- (Figure 3) Homogeneous vs Best m in n inference: Figure 3 compares the two inference strategies we proposed in the paper. Homogeneous performs competitively to Best m in n and only underperforms at 13-day and 14-day lead times, while using fewer ensemble members. We recommend this strategy if efficiency is the priority.
- (Table 1) Ablation studies with more variables: We showed the performance of Stormer with and without the weighted loss for 6 additional variables. As expected, the weighted loss model achieves better accuracy for high-pressure variables while underperforming the unweighted version for low-pressure variables. We were aware of this trade-off and proposed to use the weighted loss in Stormer to focus on variables that are more important to forecasting and/or human activities.
- (Figure 4) Comparison of Stormer with additional baselines: We added 4 more baselines -- ClimaX, NeuralGCM, NeuralGCM ENS (mean) and ClimODE. Stormer significantly outperforms ClimaX, ClimODE, and NeuralGCM, while slightly underperforming NeuralGCM ENS (mean). With IC perturbations, the gap between Stormer and NeuralGCM ENS (mean) is negligible.

Meta-review: This paper is about weather forecasting, in particular the authors propose a new (vision) transformer architecture that has several advantages over the state of the art: weather-specific patch embedding, randomized dynamics forecast, and pressure-weighted loss.

The main contribution of this paper is to show that a simple transformer architecture can obtain state of the art performance with simple settings, in particular with a careful transformer setup that is specific to weather forecasting.

Reviewers were positive, in particular noting that the paper is very well written and understandable, the simple approach, the important problem setting, ablation results, and improvements to the state of the art in long term weather forecasting. During rebuttal the authors provided additional results requested by the reviewers, in particular comparison with other baselines and a fairer comparison as Stormer is basically an ensemble model, a non-ensemble Stormer still performs very well compared to the state of the art.

To improve the paper I recommend to consider the (many) reviewer comments, in particular clarifying contributions, (conceptual) comparisons of the proposed method to previous methods in the literature, and incorporating the results presented in the rebuttal phase into the main paper or appendix.

This area chair also suggest to consider uncertainty estimation, as was also mentioned by reviewers, as it is an important component of weather forecasting, and since the method is partially an ensemble, simple uncertainty estimates could be extracted from it.

Given the positive contribution to the state of the art, and good opinion from reviewers, this paper should be accepted.
Decision: Accept (poster)