Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Authors: Christopher Subich
Syed Zahid Husain
Leo Separovic
Jing Yang
Venue: ICML 2025 poster (2025)
Keywords: Loss functions
spherical harmonics
weather forecasting
tropical cyclones
fine tuning
double penalty
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: This paper proposes a simple modification to the mean squared error loss function that eliminates the problem of overly-smooth fine scales in data-driven weather forecasts.

Abstract: Recent advancements in data-driven weather forecasting models have delivered deterministic models that outperform the leading operational forecast systems based on traditional, physics-based models. However, these data-driven models are typically trained with a mean squared error loss function, which causes smoothing of fine scales through a ``double penalty'' effect.  We develop a simple, parameter-free modification to this loss function that avoids this problem by separating the loss attributable to decorrelation from the loss attributable to spectral amplitude errors.  Fine-tuning the GraphCast model with this new loss function results in sharp deterministic weather forecasts, an increase of the model's effective resolution from 1,250km to 160km, improvements to ensemble spread, and improvements to predictions of tropical cyclone strength and surface wind extremes.

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Christopher Subich
Syed Zahid Husain'
Leo Separovic' Jing Yang
Abstract
publication of these models, the field has been joined by many others, including the Artificial Intelligence Forecast- ing System (AIFS) developed by ECMWF itself (Lang et al., 2024a).
Recent advancements in data data-driven weather fore- casting models have delivered deterministic mod- els that outperform the leading operational fore- cast systems based on traditional, physics-based -based models. However, these data-driven models are typically trained with mean squared error loss function, which causes smoothing of fine scales through "double penalty" effect. We develop a simple, parameter-free modification to this loss function that avoids this problem by separating the loss attributable to decorrelation from the loss attributable to spectral amplitude errors. Fine- tuning the GraphCast model with this new loss function results in sharp deterministic weather forecasts, an increase of the model's effective res- olution from 1,250 km to 160 km, improvements to ensemble spread, and improvements to predic- tions of tropical cyclone strength and surface wind extremes.
From the standpoint of machine learning, atmospheric fore- casting is a large-scale generative problem comparable to predicting the next frame of video. As a typical example, the version of the GraphCast model deployed experimentally by the National Oceanic and Atmospheric Administration (NOAA) (Sadeghi Tabas et al., 2025; NOAA, 2024) pre- dicts the 6-hour forecast for six atmospheric variables at each of vertical levels plus five surface variables, on a 1/4" latitude/ongiiude grid, for about 86 million output degrees of freedom in aggregate. GraphCast takes two time-levels as input, so the input for this model has about 170 million degrees of freedom.
These first-generation data-driven weather models generally act as deterministic forecast systems, where each unique initial condition is mapped to a single forecast and verified against "ground truth" from a data analysis system. The ERA5 atmospheric reanalysis (Hersbach al., 2020) of ECMWF is most often used as the source of initial and verifying data for these forecast systems owing to its high quality and consistent behaviour from 1979 to present.
1. Introduction
The models developed in Weyn et al. (2020) and Keisler (2022) suggested that deep neural networks might "solve" the problem of medium-range weather forecasting with data-driven machine learning models. In 2023, the lease of GraphCast (Lam et al., 2023), FourCastNet (Kurth et al., 2023), and Pangu- Weather (Bi et al., 2023) demon- strated forecast skill that met or surpassed that of the high- resolution forecast system (IFS) of the European Centre for Medium Range Weather Forecasts (ECMWF) at lead times (forecast lengths) up to 10 days, and some com- menters (Bauer, 2024) anticipated that data-driven fore- casting would soon supplant traditional numerical weather prediction (NWP) in all operational contexts. Since the
1.1. The Problem of Forecast Smoothing
Despite their overall forecast skill, deterministic data data-driven forecast systems are universally understood to produce overly-smooth forecasts. A typical example of this be- haviour shown in figure where 3.5-day prediction of winter storm Eunice by the 13-level, 1/4" GraphCast model is too weak and overly smooth. This smoothing results in an under-prediction of localized extreme events, and it makes the model less suitable for downstream tasks such as spectral nudging (Husain et al., 2024) and data assimilation (Slivinski et al., 2025).
MMeteorological Research Division, Environment and Climate Change Canada, Dorval, Quebee, Canada. Correspondence to: Christopher Subich <christopher..ubicheee @etiiichcc..cc........
This smoothing is most-discussed in relation to the predic- tion of gridded, global weather fields, but it is still present in models that have radically different architectures. Allen et al. (2025) develops model that operates directly in obser- vation space without an underlying grid that still produces
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
60ºN
24
Ground Truth T HHRESY
directly model the physies of the atmosphere, such that the system's forecasts are always plausible atmospheric states without excessive smoothing. Turning such a system into an ensemble prediction system involves supplying it with perturbed initial conditions and possibly stochastically perturbing the model's sub-grid parameterizations (Palmer, 2001; Berner et al., 2015).
55'N
50
45
10m wind speed (m/s)
60
Control
In the machine learning space, Mahesh al. (2024) devel- ops well-calibrated large ensemble using 29 independently- trained instantiations of the Bonev et al. (2023) architecture. When combined with cintal-condition perturbations, the result was a well-calibrated large ensemble, despite each individual ensemble member suffering from the smoothness typical of deterministic data-driven forecast systems.
55
17
50"N
45
60ºN
AMSE AR12
Lagerquist & Ebert-Uphof (2022) also develops a variety of loss functions based on the same spatial methods (such as filtering and max-pooling) to verify forecasts of convective events like thunderstorms in evaluation of high-resolution, limited-area models.
55
50
10
45
1020
20"W
10"W
0"
Figure/ 1. 10 wind speed and mean level pressure for winter storm Eunice, 18 Feb 2022 at O h UTC. Top: HRES data at 1/4" (ground truth), middle: 3.5d forecast produced by GraphCast, bottom: this work. This work produces an overall sharper forecast, with a better prediction of the winter storm's strength.
NEURALGCM
NeuralGCM (Kochkov et al., 2024) is one of the few global data-driven models that has addressed the problem of smoothing even in deterministic (non-ensemble) config- urations. However, this model is difficult to compare with its peers. It has a hybrid architecture, combining a classical dynamical core with a learned network for sub-grid parame- terizations that acts independently at each vertical column, and the classical dynamical core should cause fine-scale features develop naturally. In addition, the model was trained using weighted sum of several loss functions, one of which uses MSE only on coarsened (smoothed) version of the forecast and verifying analysis while another matches the spherical harmonic power spectrum (but not phase) only at high wavenumbers (short scales). It is not clear which of these properties are necessary or sufficient to reduce the smoothing of deterministic NeuralGCM forecasts, and the use of several loss functions adds many degrees of freedom in their weighting and internal filtering.
smooth forecasts of the future, and Han et al. (2024) shows diminished forecast activity (a bulk measure related to blur- ring) at longer lead times for a local-area model despite a nominal kilometer-scale resolution.
The conventional wisdom is that this smoothing some- thing that can be fixed in the context of an ensemble forecast- ing system, which produces realizations from the space of potential future forecasts. GenCast (Price et al., 2025) and (Lang et al., 2024b) directly produce a stochas- tic 1/40 forecast given initial values and a source of random noise. SEEDS (Li et al., 2024) and Arches WeatherGen (Couairon et al., 2024) are examples of models that predict variations around an ensemble mean, using the generative step to "fill in the blanks" around smooth baseline. Lippe et al. (2023) approaches this problem from more general partial differential equation framework, and it develops diffusion method that iteratively refines finer scales.
1.2. This Work
The purpose of this work is to tackle the problem of smooth- ing in purely deterministic, data-driven setting: can we produce a sharp forecast of the atmosphere without directly modelling ensemble uncertainty? Our answer is "yes." By modifying the MSE loss function to smoothly interpolate between implituude-preseraatie
Of these examples, all but AIFS-CRPS use a diffusion tech- nique with mean squared error (MSE) used as the de- de-noising instead uses the continuous
loss function, while AIFS- ranked probability score (CRPS, Gneiting & Raftery (2007)) as its loss function to directly optimize the spread/error rela- tionship of its produced ensemble.
and classical MSE, we can efficiently fine-tune a version of the GraphCast model to fix its smoothing problem and reproduce sharp forecasts. This greatly increases the model's effective resolution, producing better predictions of tropical cyclone intensity and surface wind speed.
However, we think that the problem of generating a good ensemble is distinct from the problem of forecast sharpness and effective resolution. Traditional NWP systems try to
2

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Section 2 describes the modified loss function, its theory of operation, and the fine-tuning procedure used for this work. Section 3 presents verification results of the fine- tuned GraphCast model, and section 4 concludes with dis- cussion of the method's limitations and potential extensions. Appendix discusses the loss function in the context of maximum likelihood estimation, and appendix presents more detailed verification statistics.
that obeys Parseval's theorem. Taking this decomposition point-by-point, extending the analysis include nonzero mean, and taking the expectation over ensemble of pre- dictions gives rise to skill/spread evaluations. However, this decomposition is not possible at training time for a deter- ministic data-driven weather forecast, and instead we turn to a spherical harmonic decomposition.
Let (i.j) be the complex- valued spherical harmonic mode with total wavenumber & and zonal wavenumber I at the (i,j) grid point on laiitue//longttuee grid, normal- ized such that (Ym)) = 6km where (.)* is the complex conjugate' A scalar field (ii, defined on the latinde/longitude grid can be written in terms of spherical harmonics as:
2. Method
2.1. Smoothing Is Optimal Under Mean Squared Error
In the NWP community, model evaluation using the mean squared error is widely understood to suffer from a so-called "double penalty" (Hoffman et al., 1995; Ebert et al., 2013). Under MSE, a good forecast that correctly predicts a feature such as a storm but misses its location is penalized twice compared to perfect forecast, once for missing the storm at its correct location and again for predicting storm at incorrect location. In traditional NWP, this double penalty makes model verification more difficult, particularly when studying the impact of improvements to forecast resolution that create more opportunities for misplaced predictions.
x(i.)) = -E Σ a (k. (1 1=-
with (k. I) the corresponding spectral coefficient. For two fields x and the latitude-weighted MSE is:
MSE(r,9) = ΣΣ dA(I, v(1,5))²
When MSE is used the loss function to train data-driven model, the double penalty problem is more than annoy- ance: it encourages the model to generate unrealistically smooth predictions by reducing the amplitude of unpre- dictable scales. To show this quantitatively, consider the case of predicting a single variable. Let Y = -N(0,) 1) be the target, and let X be the imperfect prediction of that tar- get, modelled as a normal random variable with standard deviation of σx / (X2) and correlation coefficient of p B(XY)/ox where (E()) is the expectation opera- tor. Writing X in terms of correlated and an uncorrelated component gives:
Σ Σ |a,(t.1)-a,(k1101)
(3)
where the dA term is incorporated into the normalization of Y Importantly, 0g and ay are independent with respect to zonal and total wavenumber, but the double summation here now allows us to group these terms in a physically meaningful way. Grouping terms in the inner (zonal) sum together gives rise the power spectral density PSDx(z) E, (a 1)1² and coherence (oh Σ Ri (ka)k, / PSD (where R() takes the real part) scale-dependent analogs to variance and correlation respectively. Performing the appropriate substitutions:
X x +
(1)
and the corresponding expected MSE is:
MSE(I, v) = Σ PSDs(*) + PSDx
E (MSE(X, r)) = B((X - )
= =B(X²) + 1( 2 E(XY)
2 PSDD (v) Cohn,ar (4)
(2) For fixed Y. this MSE is optimized with perfect predic- tion, when σx 1 and = 1. However, if 0 < / < 1 because the process is only partially predictable, the MSE is optimized with respect to ox when x < 1, leading to an underprediction of the process's ' natural variability.
= =xx + 1 20xp.
If x is taken to be a forecast field and y is the ground-truth analysis, as in (2) this is minimized when (oh,(x,
(PSD(())PSD(())
This optimum leads to the observed smoothing in data- driven models through two factors:
Fine scales (large k, short wavelengths) are generally less predictable than coarse scales (small k. large wave- lengths), particularly at longer lead times, and
2.2. Spectral Separation of the Mean Squared Error
Predictions of global weather are high-dimensions but equations (I) and (2) can be extended to any decomposi- tion (partition of unity) of the prediction and target fields
II practice, this work takes advantage of the property that = =(Y2"" to work with only non-negative wavenumbers.
3

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
more desirable if the loss function reflected our true goal, encouraging forecasts correlate well to the ground-truth and retain realistic variation at finer scales.
Amplitude ratio Coherence
1.0
Single-st step
0.9
Fortunately, beginning with MSE written in terms of its spectral decomposition, this is a simple modification. First, we write (4) in terms of a perfectly-correaace loss (with Coh (z. y) = 1) and a residual:
0.8
0.7
75000
150000
225000
300000
MSE v) I(V [(VPSD&(*) - (PSD
0.9
0.8
2 PSDx (U) - (z, (5)
Multi-tee
0.7
Then, we seek to break the interaction between the spectral amplitudes and coherence contained in the second term of (5). One option would be to fix the role of x as a trial prediction and y as the verifying analysis and replace v /PSD (x) (V) by (y) but the symmetry of the loss function can be retained by writing:
24h
0.6
48h
72h
0.5
300000
304000
307000
311000
Batch
Figure 2. Amplitude ratio (solid) and coherence (dashed) for the spherical harmonic mode with total wavenumber 100 for tempera- ture 850hPa during the training of a 1 version of the GraphCast model with an MSE loss function. At top, values for 6h lead time during the single-step pre-training phase and bottom, values for 6h-72h during the forecast rollout (batches 100,0000311,000 incrementing one step every 1,000 batches).
AMSE(x,,) v) = Σ (VVVVSI (T) vPSD
2 max (()) (1D (oh (6)
AMSE is now an adjusted mean squared error, which can act as a drop-in replacement during model training. Like its unmodified counterpart, AMSE is zero if and only if x = y, and has the same Taylor expansion (in x) about x = y. The gradients of AMSE(2, (z, y) with respect to x (that is, minimizing AMSE) will always point in the direction of increased coherence (Coh,, y) + 1) and a correct spec- tral magnitude (PSDx (x) + PSDx even if physical limits to predictability impose a practical limit to coherence. AMSE retains the units of MSE and has a similar magnitude, but is no longer proper metric because does not satisfy the triangle inequality.
Data-driven models with conventional architectures learn to smooth fine scales (reducing the power spectral density) more quickly than they learn to predict them (increasing coherence).
This is illustrated in figure 2, which shows the amplitude ratio (square root of power spectral density ratio) and coher- ence for total wavenumber 100 (wavelength about 400 km) between predictions of the temperature field at the 850hPa level and the ground truth, for a 1 version of GraphCast dur- ing training with the curriculum of Lam et al. (2023). After rapid adjustment from initially random outputs, the ampli- tude ratio and coherence closely track each other, with an initial smoothing followed by gradual but partial sharpen- ing as the model learns to predict this scale (with increasing coherence). When training is extended autoregressively to 12 steps (72h forecasts), smoothing increases at longer lead times as the forecast length increases.
Unlike the mix of filtered and spectral loss functions used by NeuralGCM, (6) is parameter-fiee requiring no selection of cutoff scales or scaled addition of qualitatively different terms. A parameter could be added to (6) to change the relative weights of its two terms, but that was not neces- sary in this work. Appendix A contemplates extending this framework to maximum likelihood estimation.
Equation 6 is defined for a single two- wwo-dimensional variable, but GraphCast produces several outputs per gridpoint. In the 1/4² 13-level version of the model considered here, there are six variables (geopotential, temperature, specific humidity, two components of horizontal wind, and vertical wind) pro- duced at each of 13 atmospheric levels plus five variables (2-meter temperature, two components of 10-m horizontal wind, mean sea level pressure, and 6h-accumulated precipi- tation) at the surface. This work follows equation (A. 19) of Lam et al. (2023) by aggregating each variable's ' error (MSE there, AMSE here) with a per- parivali weight, level weight- ing proportional to the pressure level, and normalization
2.3. Spectrally Adjusted Mean Squared Error
This smoothing is undesirable. It makes the produced fore- casts less realistic, and it complicates model comparisons. Lam al. (2023) performs extensive verification under an "optimal blurring" model to show that the purported forecast power of GraphCast is not just an artifact of its smooth- ing, and more straightforward verification methodologies such as that of Rasp et al. (2024) may conflate the effects of optimal smoothing with forecast skill even when evaluating at reduced resolution. It would instead be far
4

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Length scale (km)
Table 1. Fine-tuning curriculum for the 1/9//3l-evell version of GraphCast trained for this study, including the peak and termi- nal learning rates (LR) the cosine annealing schedule used at each stage. The batch size was 8 throughout, and each stage had warm-up period of 64 batches.
4000
1250
444
160
100
55
1.0
Control
AMSE AR12
AMSE AR1
0.0
Length
Batches
Peak/End LR
GPU Time
step (6h)
25,000
2.5 10 10111000/// 2/.25- 10
7.7d
1.0
2 steps (12h)
2,500
2.5 10 "
2.2d
1200 p0m
4 steps (24h)
2,500
2.5-10 15..00177.--0--
4.3d
8 steps (48h)
1,250
2.5 2.5-10-977.5-00
4.6d
0.5
Amplitude rebio
12 steps (72h)
1,250
15.-10-97..-110
7.4d
Coherence
0.0
of the disparate units by veriabbialee per-level standard deviation.
1.0
2408
0.5
2.4. Fine- Fine-Tuning Methodology
0.0
We demonstrate the efficacy of this loss function using 1/4, 13-level version of GraphCast. Based on the observation above that the model tends to rapidly adjust its per-scale smoothing to match its coherence, we treat this as a fine- tuning process and begin with the "operational" checkpoint provided by Lam et al. (2024), which is publicly available under Creative Commons license.
10
32
90
250 400 720
Total wavenumber
Figure 3. Amplitude ratio (solid) and coherence (dashed) for all output variables and levels, weighted using the variable/level weights in the loss function, for the control model and this work after the 1-step training and after complete fine-tunie mne-tuning. Top: 6h lead time, middle: 120h (5d) lead time, bottom: 240h (10d) lead time. The dashed line is placed where a model would underrepre- sent the power spectral density by 25%.
Our fine-tuning methodology is summarized in table 1, and the overall approach is inspired by Subich (2024). While the baseline model checkpoint was trained over 72h (12 autoregressive steps of 6h each), in earlier testing at 1 we found it better to begin the fine-tuning with single-step forecasts and increase the forecast length in stages. Training over single steps is both faster per step and supports higher learning rates.
taining 4 NVidia A 40GiB GPUs per node, using data parallelism with the batch split across GPUs and the gradi- ents accumulated via MPI. Overall, the fine-tuning process took about 26.2 GPU-days. Lam et al. (2023) does not dis- close the total training time required to produce the model checkpoint from scratch, but other models of similar size (Bonev et al., 2023; Lang et al., 2024a; Bi et al., 2023) report training times of about GPU-year using similar hardware.
The other training hyperparameters, including Adam W (Loshchilov& & Hutter, 2019) hyperparameter settings and per-variable, per-level loss weightings were identical to those described by Lam et al. (2023).
3. Results
The 13-level GraphCast checkpoint that forms the base of our fine-tuned model was originally trained the ERA5 reanalysis from 1979-2017, then itself fine-tuned on the initial conditions used for the contemporaneous HRES (IFS) model from ECMWF over 2016-2021. We used this latter dataset and training period in our work, and it is available from Rasp et al. (2024) as the "HRES-fcO" dataset. As described in Lam al. (2023), we supplemented the HRES data with the accumulated precipitation field from the ERA5 reanalysis over the training period, since an initial conditions dataset has no accumulated precipitation by definition. The equivalent data for calendar year 2022 is also available, and we used this period for model evaluation.
The fine-tuned model is evaluated against the control (un- modified) model over calendar year 2022 using the HRES dataset for initialization and as ground truth unless other- wise specified. As reported in Lam et al. (2023) and is typical in other deterministic data-driven models, forecast performance at longer lead times improves when the model is autoregressively trained over multiple steps, and the fully- tuned model (trained over 12 forecast steps and labelled "AMSE AR12 in the figures and discussion below) is con- sidered the primary model for evaluation.
Since multi-step training also tends to cause both fine-scale smoothing and a loss of variability in ensemble settings, these respective evaluations (sections 3. and 3.2) will also include the model checkpoint created after just single-step
We fine-tuned our model on 1-2 nodes of a cluster con-
5

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
fine-tuning, denoted "AMSE ARI."
The AMSE AR1 model shows the same qualitative be- haviour but generates this "noise" more strongly, leading to reduced effective resolution of about 450 km (wavenumber 90). The forecasts produced by this version of the model are less coherent with the analysis, showing a reduced forecast skill at all scales for longer forecasts.
3.1. Effective Resolution
Conventional, physics- NWP models are widely un- derstood to have an effective resolution that is coarser than the model's native grid resolution. Limits to effective reso- lution come from the limited fidelity of spatial temporal discretization, from artificial diffusion or damping used to stabilize a model, and from sub-grid processes (such as turbulence) that must be imperfectly estimated rather than directly modelled. A model behaves unrealistically at scales finer than its effective resolution, typically providing insuffi- cient variability and too-smooth solutions.
For illustration, appendix B.2 shows amplitude spectra for select variables at various lead times, without normalizing by the spectral magnitude of the ground truth. Appendix B.5 discusses the effective resolution of the model when trained with either mean squared error or mean absolute (L1) error.
3.2. Lagged Ensemble Verification
Deterministic data-driven NWP models do not have the same underlying numerical issues that result in reduc- tions to effective resolution, but the smoothing produced by training with an MSE-based loss function acts in a very similar way. Figure 3 shows the amplitude ratio
The observation that AMSE- fine-tuning provides sharp forecasts is encouraging, but that alone is not enough to demonstrate utility. The model might have learned to match its expected variance by generating quasi-static noise that does not sufficiently depend on the surrounding flow, for example. The ideal way measure this sort of forecast skill is in an ensemble setting, where the chaotic nature of the atmosphere is accounted for by evaluating the full distribution of plausible outputs given an initial condition.
(<PSDr(=) PSD (v)) and coherence (Coha (z,y)) B)) be- tween each of the GraphCast models and the verifying analysis over calendar year 2022. To compute a combined curve despite the many per-gridpoint values predicted by the model, the statistics for each separate variable are com- bined using the same variable and level weighting used in the model's Is loss function?.
Development of a full ensemble system is well beyond the scope of this work, but Brenowitz et al. (2025) provides a procedure evaluate a deterministic model using an ensem- ble generated from timesspprrted initial conditions. The central idea of this method is that predictions initialized at different times should diverge, so several consecutively- initialized forecasts that are all valid at a shared time form an ad- hoc ensemble, without the need for an auxiliary method of defining an ensemble of initial conditions.
The control model significantly smooths fine scales even after a single 6-hour forecast step, and that smoothing in- creases with the forecast lead time. If we somewhat arbitrar- ily draw the line of effective resolution at the point where the model has lost 25% of the per- wavenumbrr energy (cor- responding to a ratio of power spectral densities of 0.75 or an amplitude ratio of v VO.75), the 5-day predictions of the control model reach that cutoff at wavenumber 32, corre- sponding to oscillations with a wavelength of about 1250 km. Small changes in the target amplitude ratio will result in small changes to the derived effective resolution.
This approach is implemented here, using forecasts initial- ized at 12-hourly intervals in 2022 and evaluated from 10 January 2022 0:00 UTC to 31 December 2022 12:00 UTC. Each set of nine consecutively initialized forecasts (span- ning four days from beginning to end) forms an ensemble, and the ensemble's 's notional lead time is that of its central member.
The models fine-tuned in this work do not show this type of fine-scale dissipation. The AMSE ARI2 model has small amount of smoothing at moderate scales, but the variability recovers again at finer scales, and dissipation-basee defi- nition of effective resolution would be extremely sensitive to the cutoff value. Instead, we observe that for longer fore- casts the model has more energy at small scales than in the ground-truth dataset, suggesting a baaee" definition of effective resolution. For long forecasts, the amplitude ratio rises above around wavenumber 250, giving an ef- fective resolution of about 160 km.
The primary evaluation metries the CRPS, ensemble root mean squared error (eRMSE), and spread/error ratio, with definitions given in appendix B.4. For an operational ensem- ble, a spread/error ratio close to is considered idcal, but that is confounded here because the members of a lagged ensemble are not statistically interchangeable. Since de- terministic data-driven NWP models are underdispersive, however, a larger spread/error ratio is generally better.
2 NNormalization of the disparate variables by standard deviation was not required here, since the amplitude ratio and coherence are already dimensionless.
Figure 4 shows the evolution of these statistics versus lead time for a selection of variables and levels, and more detailed evaluation of CRPS and cRMSE are shown in figures 11 and 12. The AMSE AR12 model shows consistent improve-
6

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
2500 (m²
(K)
(g kg
200
1.0
ANSE AR12
1.00
0.6
0.8
CRPS
0.75
0.6
50
0.50
0.4
AMSE
ORMSE (1)
400
1.25
1.5
200
1.00
1.0
1.0
0.9
0.8
Spread/error
0.9
0.8
0.8
0.7
0.7
0.6
0.6
7
5
6
5
6
Lead time (d)
Lead time (d)
Lead time (d)
Lead time (d)
Figure Lagged ensemble statistics for geopotential (z) 500hPa, temperature (1) 850hPa, specific humidity (q) at 700hPa, and 2-meter temperature (21) from left to right. The statistics are the CRPS, root mean squared error of the ensemble mean, and spread-error
ratio, from top to bottom.
ments to the CRPS while the eRMSE sees little change, indicating that the fine tuning process produces better- calibrated (more dispersive) ensemble without degrading overall predictive performance.
version produce reasonable 5-day prediction of the storm's location (within about 125 km), but the control version of GraphCast predicts an unrealistically weak storm.
More quantitatively, figure 6 shows the mean intensity and mean absolute position errors for tropical cyclones over June-19 September 2022 initializations, using the algo- rithm of Zadra al. (2014) to compare against the Interna- tional Best Track Archive for Climate Stewardship database (Knapp et al., 2010). Compared to these observations, even the HRES data is imperfect and shows weak-intensity bias. The control model has larger weak-intensity bias that increases with lead time, but the AMSE ARI2 model retains the quality of the HRES dataset. The storm location predictions between the control and AMSE ARI2 models are equivalent.
While the AMSE ARI model shows greater ensemble spread, the less skillful forecast results in a significantly reduced CRPS. However, unlike the results of Brenowitz et al. (2025), the spread/error ratio of the AR1 and AR12 models converge for most variables at longer lead times, suggesting that multi-step training in this framework does not cause a collapse of variability in an ensemble setting.
3.3. Hurricane Prediction and Extreme Weather
The effect of improved effective resolution is most strongly apparent in the prediction of local extremes, and few weather events are more extreme than tropical cyclones.
Extreme weather includes more than tropical cyclones, and appendix B.3 discusses quantile-quante predictions of sur- face wind speed and temperature, validated against station observations, Both the control model and AMSE AR12 pro- duce realistic temperature extremes, but the AMSE AR12 model provides more realistic predictions of wind-speed extremes.
Data-driven NWP models like GraphCast improve predic- tions of hurricane tracks relative to conventional NWP mod- els (see for example figure 3A of Lam al. (2023)). Since storms are guided by large-scale "steering flows" that have natural scales of thousands of kilometers, these predictions of storm position are relatively unaffected by the models' limited effective resolutions but benefit from improvements large-scale forecast skill. However, cyclones themselves comparatively small, and predictions of the storm inten- sity significantly affected by MSE-induces induced smoothing.
4. Discussion & Limitations
Using the mean squared error as a model loss function asks the model to average away unpredictable scales. In weather forecasting, the unpredictable scales are generally the smaller scales that carry information about local vari- ance, and this averaging process leads to data-driven weather forecasts that are far smoother than the grid resolution would
Figure 5 depicts this situation for Hurricane lan, the most intense Atlantic tropical cyclone of the 2022 season. Both the control version of GraphCast and the AMSE AR12

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
28'N
Wind Speed [m '1
Mean Maximum Wnnd Speed Erro
HRES
40.0
AMGE ARR2 HRES
-10
N6
-20
(3)
Mean Minmum Central Pressure Error
Pressuure [hPPs]
24
34.5
28
Control
10m wind speed (m/s)
b)
Mean Absoluto Postioon Error
N6'N
1900
29.0
800
Deterce
300
100
N4
8
28'N
AMSE AR 2
23.5
Figure 6. Predictions of tropical cyclone intensity ((a), mcan maxi- mum surface wind speed; (b) mean minimum central pressure) and mean absolute position error (c) for forecasts initialized 20 June- 19 September 2022. Orange squares show statistically significant differences between the AMSE AR12 and control predictions.
Forecast Lead Time
N6
18.0
N
85
W3
81 W
Since the AMSE loss function (6) is zero if and only if the predicted field matches the ground truth, it may be useful throughout model training rather than just during a fine- tuning pass. However, a thorough test of this proposition would require considerable computational budget, it is left for future work. Use of the AMSE loss function throughout the training process might improve the coher- ence of fine-scale prediction by allowing the model to spend more of its training time "seeing" these modes, but on the other hand the coherence-dependen smoothing encouraged by the MSE loss function (figure might act as an implicit regularization that smooths the model's gradients and speeds up training overall.
Figure 5. 10 m wind speed and mean sea level pressure for Hurri- cane lan, 28 Sept 2022 12h UTC. Top: HRES data 1/4, middle: 5d forecast produced by the control GraphCast model, bottom: the model after 12-step fine-tuning with AMSE.
suggest.
This is not property inherent data-driven NWP. The al- ternate loss function based on (6) uses spectral transform to separate the loss attributable to amplitude error from that attributable to decorrelation, encouraging the model to reproduce a realistic spectrum even if it can't 't make an ac- curate prediction. When applied to the 142h, 13-1evel version of GraphCast with an abbreviated fine-tuning process, we recover a model that has a much finer effective resolution, has improved CRPS-based verification in lagged ensemble setting, and fixes the weak intensity bias in the prediction of tropical cyclones.
4.1. Effective Resolution
The ultimate conclusion of this work is that the AMSE- based error measure improves the effective resolution of NWP weather models, but the phrase "effective resolution" must always be accompanied by the question, "effective at what?"
When fine-tuned autoregressively over multiple forecast steps, the model suffers from a small amount of smoothing at mesoscales (intermediate scales). We speculate that this because such autoregressive training has two objectives: forecasts are asked both be accurate (and thus sharp, per (6)) and to be good initial conditions for the next forecast step. This latter goal is implicit, and it is not directly affected by the loss function used in training. Future work will consider the use of a replay buffer training (like that of Chen et al. (2023)) to see if long-range forecast skill might be retained with even better prediction of amplitudes.
We chose to define an effective resolution based on smooth- ing of fine scales, since a model that simply doesn't 't rep- resent scale cannot effectively model it. However, other definitions exist in the literature, and users of these models should keep their ultimate goals in mind. For example, Kent et al. (2014) studies various discretization schemes for nu- merical partial differential equations under both diffusion (smoothing) and dispersion (wave propagation) definitions of effective resolution.
8

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
4.2. Alternative Grids
trajectory, in addition to the whole-ensemble optimization encouraged by CRPS-like ensemble training.
Passing from equation (2) to (4) makes use of Parseval's ss theorem to give exact relationship between the spatially- defined mean squared error and the equivalent in the spec- tral representation. Implementing this in a training cycle requires fast computation of spherical harmonic transforms. This is simple enough for global latitude/loggiude grids, but it might be difficult for local-area models without a regular global grid structure.
Code and Data Availability
An implementation of the AMSE error function and the code used to train GraphCast for this work are available at https: / / /github. cow/csubich//ee/ccccc////mcc t tree/ under the Apache 2.0 license. The fine-tuned checkpoints produced for this study available at https://hugh http://hhgggggccceeeeeo// wtt.co//ccsooooo co/csubich/ graphcast_ _assrattt_sssaeaaee_eee under the CC-BY-ND-SA 4.0 license, as derivative works of the DeepMind "graphcast-operations checkpoint.
In these cases, we think that the basic intuition behind (6) might still apply through other multiscale decompositions such as wavelet lifting (Sweldens & Schröder, 2000), pro- vided suitable equivalents to scale-dependent variance and correlation could be found. The multiscale decomposition critical in some form, however, since the method takes ad- vantage of the approximate independence of seale-separated modes. Without such decomposition (e.g. applying the adjustment of (6) globally, without the harmonic transform), the model might be able to "cover up" a lack of fine-scale variability by over-emphasizing coarser scales.
Acknowledgements
The authors would like to thank Charlie -ibbrdt-Pinard, Vikram Khade, and Hugo Vandenbroucke-Ment of the Canadian Centre for Meteorological and Environmental Prediction for access to the 1r-traieed version of GraphCast used to produce the results of figure 2.
4.3. Applications to Other Domains
Impact Statement
AMSE is a natural error function for weather prediction be- cause the spectral decomposition is physically meaningful and relatively stable over time. A partially incorrect but realistic prediction of weather at 2000 km scales would not significantly change the amount of energy present at 100 km scales, just its relative location. The goal of deter- ministic forecast is to be physically plausible, and correct prediction of spectral amplitudes is necessary condition for physical plausibility.
Accurate weather forecasts are a vital public service, and its benefits are disproportionately concentrated in the extremes. Accurate forecasts of extreme weather such as tropical cy- clones save lives. On one hand, this means that we should be eager to develop improvements weather forecasting systems, but on the other hand means that we should be very careful not to just "chase scores." confusing what's easy to calculate with what's truly important.
The method can be mechanically applied whenever a spec- tral decomposition is possible, but additional value only likely when a sub-aggregation of that spectrum meaning- ful. This is most obviously possible in other areas of fluid dynamics, particularly the modelling of turbulent flows. In that domain, Chakraborty et al. (2025) developed a binned spectral loss function (on a planar domain) that is reminis- cent of the amplitude-only component of (6), but it discards the phase information. We are optimistic that integrating the spectral correlation along the lines of AMSE will make such models more robust.
This work contributes to this field by introducing a way to make data-driven weather forecasting more realistic, with variability at moderate and fine scales that much closer to reality. This improves various probabilistic scores and predictions of tropical cyclone intensity, but this is not a guarantee of complete physical plausibility. In particular, we have not yet shown that these forecasts are behave "out of distribution," such as when simulating possible future climate paths.
Operational weather centres are very diligent about per- forming rigorous evaluation of models before making them operational, and we hope that this work can help ease the path towards the adoption of better- petter-perfomming. data-driven forecasting systems in the near future.
4.4. Applications Ensemble Modelling
We are particularly encouraged by the beneficial impact that AMSE-based training has on the spread of forecasts in an ensemble setting. Without any dedicated mixemble-basee training we end up with model that nonetheless produces a more realistic spread of forecasts. In future work, we hope to use this loss function as a basis for an ensemble forecast where each individual ensemble member produces a realistic
References
Allen, A., Markou, S., Tebbutt, W., Requeima, J., Bruinsma,
W. P., Andersson, T. R., Herzog, M., Lane, N. D., Chantry, M., Hosking, J. S., and Turner, R. E. End-to-end data-

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
driven weather prediction. Nature, pp. 1-8, March 2025. ISSN 1476-4687. doi: 10.1038//41186--05-088888
Gneiting, T. and Raftery, A. E. Strictly Proper Scoring Rules,
Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359-375 March 2007. ISSN 0162-1459, doi: 10. 00.11980016114550000000111100100
Bauer, P. What if? Numerical weather prediction at the crossroads. Journal of the European Meteorological So-
ciety, 1:100002, December 2024. ISSN 2950-6301. doi:
Han, T., Guo, S., Ling, F., Chen, K., Gong, J., Luo, J., Gu, J., Dai, K., Ouyang, W., and Bai, L.. FengWu-GHHR: Learn- ing the Kilometer-scale Medium-range Global Weather Forecasting, January 2024. URL
Berner, J., Fossell, K. R., Ha, S.-Y., Hacker, J. P., and Sny- der, C. Increasing the Skill of Probabilistic Forecasts: Understanding Performance Improvements from Model- Error Representations. Monthly Weather Review, 143(4): 1295-1320, April 2015. ISSN 1520-0493, 0027-0644.
org/abs 2402 00059. arXiv:2402.00055 [physics].
Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñooz-aabter,, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abel- lan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R. J., H6lm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg. F., Villaume, S., and Thépaut, J.-N. The ERA5 global reanalysis. Quarterly Journal of the Royal Me- teorological Sociery, 146(730):1999-2044) 2020. ISSN 1477-870X. doi: 10. 10.1002/qj.3803.
Bi, K. Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q. Accurate medium-range global weather forecast- ing with 3D neural networks. Nature, 619(7970):533- 538, July 2023. ISSN 1476-4687. $41586-023-06185-3
doi: 10.1038/
Bonev, B., Kurth, I., Hundt, C., Pathak, J., Baust, M., Kashinath, K., and Anandkumar, A. Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere. In Proceedings of the 40th International Confer- ence on Machine Learning, volume 202, pp. 2806-2823. PMLR, July 2023. ISSN: 2640-3499
Hoffman, R. N., Liu, Z., Louis, J.-F. and Grassoti, C. Distor- tion Representation of Forecast Errors. Monthly Weather Review, 123(9))7588-2770) September 1995. ISSN 1520-0493, 0027-0644. doi: 10. 10.1175/11200-09931995 123 123,27558DRROFE)220000
Brenowitz, N. D., Cohen, Y., Pathak, J., Mahesh, A., Bonev, B., Kurth, T., Durran, D. R., Harrington, P., and Pritchard, M. S. A Practical Probabilistic Benchmark for AI Weather Models. Geophysical Research Letters, (477))22044G.113666 2025. ISSN 1944-8007. doi: 10.1029/2244LI113333
Husain, S. Z., Separovic, L., Caron, J.-F., Aider, R., Buehner, M., Chamberland, S., Lapalme. E., McTaggart-Cowal R., Subich, C., Vaillancourt, P., Yang, J., and Zadra, A. Leveraging data-driven weather models for improving numerical weather prediction skill through large-scale spectral nudging, July 2024. URL http: / rxiv. org/ ////ab//24000 06100. arXiv:2407.0611 [physics].
Chakraborty, D., Mohan, A. T., and Maulik, R. Binned Spectral Power Loss for Improved Prediction of Chaotic Systems, February 2025. URL http: /ttp::///axxii arxiv. org/ abs/2502. 72472.44420022 arXiv:2502.00472 [cs].
Chen, K., Han, T., Gong, J., Bai, L., Ling, F., Luo, J.-J., Chen, X., Ma, L., Zhang. T., Su, R., Ci, Y., Li, B., Yang, X., and Ouyang, W. Fun Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead, April 2023. URL http: / arxiv. www./or////a...////////////// 02948. 8. arXiv:2304.0294. [physics].
Keisler, R. Forecasting Global Weather with Graph Neu- ral Networks, February 2022. URL http: /arxiv. org/ /aaabb//2200 07575. arXiv:2202.0757: [physics].
Kent, J., Whitehead, J. P., Jablonowski, C., and Rood, R. B. Determining the effective resolution of advection schemes. Part Dispersion analysis. Journal of Compu- tational Physics, 278:485 496, December 2014. ISSN 0021-9991. doi: 10. jop.o//.jp..204...111
Couairon, G., Singh, R., Charantonis, A., Lessig, C., and Monteleoni, C. ArchesWeather & Arches Weathere::::::e::: deterministic and generative model for efficient ML weather forecasting, December 2024. URL http:// arxiv. org/ abs /21./2222111222 [cs].
12971. arXiv:2412.12971
Knapp, K. R., Kruk, M. C., Levinson, D. H., Diamond, J., and Neumann, C. The International Best Track Archive for Climate Stewardship (IBTrACS). Bulletin of the American Meteorological Society, 91(3):363-376. March 2010. doi: 10. 10.117522009AAM22755..
Ebert, E., Wilson, L., Weigel, A., Mittermaier, M., Nurmi, P., Gill, P., Göber, M., Joslyn, S., Brown, B., Fowler, T., and Watkins, A. Progress and challenges in forecast cap.oooooiicappplplaattooosoooso Applications, 20(2):130-139 2013. ISSN 1469-8080 doi: 10. 1002/000......mm 1392.
verification. Meteorological
Kochkov, D.,, Yuval, J., Langmore, I., Norgaard, P., Smith, J., Mooers, G., Klöwer, M., Lottes, J., Rasp, S., Düben,
10

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
P., Hatfield, S., Battaglia, P., Sanchez-Gonzalez, A., Will- son, M., Brenner, M. P., and Hoyer, S. Neural general circulation models for weather and climate. Nature, 632 (8027):1000-1666 August 2024. ISSN 1476-4687. doi: 10. 1011088841586--24407744444
Leutbecher, M. and Palmer, T. N. Ensemble forecasting. Journal of Computational Physics, 227(7):3515-3539 March 2008. ISSN 0021-9991. doi: 10. 10.1016ff.jep.0007. 02.014.
Li, L., Carver, R., Lopez-Gomezz, I., Sha, F., and Anderson, J. Generative emulation of weather forecast ensembles with diffusion models. Science Advances, 10(13)eadk44489 March 2024. doi: 10. 20.1126/siaddv.adk4455
Kurth, T., Subramanian, S., Harrington, P., Pathak, J., Mar- dani, M., Hall, D., Miele, A., Kashinath, K., and Anand- kumar, A. FourCastNet: Accelerating Global High- Resolution Weather Forecasting Using Adaptive Fourier Neural Operators. In Proceedings of the Platform for Advanced Scientific Computing Conference, PASC '23, Pp. 1-11, New York, NY, USA, June 2023. Association for Computing Machinery. ISBN 9798400701900. doi: 10.1 00.11445/552997933554122
Lippe, P., Veeling, B., Perdikaris, P., Turner, R., and Brand- stetter, J. PDE-Refiner: Achieving Accurate Long Roll- outs with Neural PDE Solvers. Advances in Neural Infor- mation Processing Systems, 36:67398-67433, December 2023.
Lagerquist, R. and Ebert-Uphofi, I. Can We Integrate Spatial Verification Methods into Neural Network Loss Functions for Atmospheric Science? Artificial Intelligence for the Earth Systems, (4), November 2022. ISSN 2769-7525. doi: 10.1175/AIES---22--000
Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regu- larization, January 2019. URL http: : / araiiiiiivv...vi org/ abs / 1711111110 05101. arXiv:1711.05100 [cs, math].
Mahesh, A., Collins, W. Bonev, B., Brenowitz, N., Co- hen, Y., Elms, J., Harrington, P., Kashinath, K., Kurth, T., North, J., OBrien, T.. Pritchard, M., Pruitt, D., Risser, M., Subramanian, S., and Willard, J. Huge Ensem- bles Part I: Design of Ensemble Weather Forecasts us- ing Spherical Fourier Neural Operators, August 2024. ar arXiv:2408.033 00000 [physics] version:
Lam, R., Sanchez-Gonzaler A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton- Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Vinyals, O., Stott, J., Pritzel, A., Mohamed, S., and Battaglia, P. Learning skillful medium-range global De-
weather forecasting. Science, 382(6677):1416- cember 2023. doi: 10. 22.112///ci
NOAA. GraphCast with GFS input, 2024. https::/reght http:///eeggstty.ceeaataaaaaaaa opw.data..ws aws/ noaa nws noaa-nws-graaee www.:/s-ggaaccaag--pps....
URL
Lam, R., Sanzhez-Gonzalez, A., Willson, M. Wirns- berger, P., Fortunato, M., Alet, F,, Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Vinyals, O., Stott, J. Pritzel, A., Mohamed. S., and Battaglia, P. GraphCast GitHub repository, July 2024. URL bttps://githh) com/ http//gooledeeemindl...... date: 2023-07-144111:07:577.
Palmer, T. N. A nonlinear dynamical perspective on model error: A proposal for non-local stochastic-dynan parametrization in weather and climate prediction mod- els. Quarterly Journal of the Royal Meteorological So- ciety, 127 2001. ISSN 1477-870X. doi: 10. 10.10002/g4497117577222
htg.aaapppnnngggapaaaa... graphcast. original-
Lang. S., Alexe, M., Chantry, M., Dramsch, J., Pinault, F., Raoult, B., Clare, M. C. A., Lessig. C., Gerber, M., Magnusson, L., Bouallègue, Z. B., Nemesio, A. P., Dueben, D., Brown, A., Pappenberger. F,, and Rabier, F. AIFS ECMWF's data-driven forecasting system, June 2024a. URL http::tppp. / arxiv. arg/abs/2406. 01465. urXiv:240600146 [physics].
Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El- Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R., and Willson, M. Probabilistic weather forecasting with machine learning. Nature, 637 (8044)844-90, January 2025. ISSN 1476-4687. doi: 10. 10.10386641586-024-0222.
Lang, S., Alexe, M., Clare, M. C. A., Roberts, C., Ade- woyin, R., Bouallègue,, Z. B., Chantry, M.. Dramsch, J., Dueben, P. D., Hahner, S. Maciel, P., Prieto-Nemesio, A., O'Brien, C., Pinault, F., Polster, J., Raoult, B., Tietsche, S., and Leutbecher, M. AIFS-CRPS: Ensemble fore- casting using a model trained with a loss function based on the Continuous Ranked Probability Score, Decem- ber 2024b. URL http://t//// http:://atxi....t////////a arxiv. org/ hazgg/bb//244 abs/2412. 15832, arXiv:2412. [physics] version: 1.
Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russell, T., Sanchez-Gonzalez, A.. Yang, V., Carver, R. Agrawal, S., Chantry, M., Ben Bouallegue, Z., Dueben, P., Bromberg, C., Sisk, J., Barrington, L., Bell, A., and Sha, F. WeatherBench 2: A Benchmark for the Next Generation of Data-Driven Global Weather Mod- els. Journal of Advances in Modeling Earth Systems, 1610):20033SS00401,, 2024. ISSN 1942-2466. doi: 10. 10.1029/20033M5044099

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Sadeghi Tabas, S., Wang, J., Lei, W., Row, M., Zhang, Z.,, Zhu, L.,, Peng. J., and Carley, R. GFS-Powered Machine Learning Weather Prediction: A Comparative Study on Training GraphCast with NOAA's GDAS Data for Global Weather Forecasts, 2025. URL rett:e:/reeository.LLiie noaa. gov gor/viiew/ncaa/67455
Slivinski, L. C., Whitaker, S., Frolov, S., Smith, T. A., and Agarwal, N. Assimilating Observed Surface Pressure Into ML Weather Prediction Models. Geophysical Research Letters, 146)):20244L111499 2025. ISSN 1944-8007.
doi: 10. 10.100//2224L1144999
Subich, C. Efficient fine-tuning of 37-level GraphCast with the Canadian global deterministic analysis, Au- gust 2024. URL http:/ arxiv. wtt.v...rggaa////00000 14587. arXiv:2408.14588 [es].
Sweldens, W. and Schröder, P. Building your own wavelets at home. In Klees, R. and Haagmans, R. (eds.), Wavelets in the Geosciences, Lecture Notes in Earth Sciences, pp. 72-107. Springer Berlin Heidelberg, Berlin, Heidel- berg, 2000. ISBN 978-3-540-46590-4, doi: 10.1007/ BFb0011092
Tödter, and Ahrens, B. Generalization of the Igno- rance Score: Continuous Ranked Version and Its De- composition. Monthly Weather Review, 240s6):2005- 2017, June 2012. ISSN 1520-0493, 0027-0644. doi: 10.1 10.1 11-1111022661.111
Weyn, A., Durran, R., and Caruana, Improv- ing Data-Driven Global Weather Prediction Using Deep Convolutional Neural Networks
Cubed Sphere. Journal of Advances in Modeling Earth Systems, 12 (9):e2020MS0022099 2020. ISSN 1942-2466. doi: 10.1029920200MS0021099
Zadra, A., McTTaggart-Cowal R., Vaillancourt, P. A., Roch, M., Bélair, S., and Leduc, A.-M. Evaluation of Tropi- cal Cyclones in the Canadian Global Modeling System: Sensitivity to Moist Process Parameterization. Monthly Weather Review 14273):1177-1222 March 2014.
12

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
A. Relationship to Maximum Likelihood Estimation
In developing the AMSE loss function, the transformation from ordinary, gridpoint-based MSE (2) to its spectral definition with power spectral densities and coherence (5) algebraic in nature. The beneficial effect of the AMSE loss function's separation of apectral-ampltiude and decoherence terms arises because the underlying spectral decomposition physically meaningful. At fine enough scales, atmospheric dynamics are increasingly rotationally symmetric and position-invariant, with individual spectral amplitudes that look like draws from Gaussian distribution.
If we elevate this property from fortunate coincidence to a simplifying assumption, we can treat the set of modes corresponding to a particular total wavenumber as random variables and apply the machinery of ensemble verification to individual, deterministic forecasts. The goal of producing realistic forecasts despite limited predictability is conceptually similar to the goal of maxiimum-likelihoon estimation, so we consider here the effect of Kullback-Leibler (KL. divergence minimization. In the meteorology literature, the KL divergence is named the continuous ignorance score (Tödter & Ahrens, 2012), and it is sometimes used for ensemble verification.
Treat the modes corresponding to a single total wavenumber k as draw from a 2 - 1-dimensional normal random variable with mean zero and some finite standard deviation. In this interpretation, the ground-truth analysis is:
Y σy NN2 1).
The forecast is itself' taken to be normal random variable, but following the pattern of (5) it is partially correlated to Y and has its own standard deviation. Take the correlation to be and the forecast standard deviation to be dx, and:
( ( 1-111pp000 N N (0.1) oy
X -x
(8)
noting for emphasis that this definition of X depends upon Y. With the assumption that each of the per-wavenumber modes are independently drawn from this distribution, we can also treat X and Y as product of 2k 1 independent, scalar random variables, which will simplify the following algebra.
The KL divergence of the data given the forecast is then given by:
/ Pr(('') log
Pr(3') Px(v')
DEL DY
dy',
(9)
for the respective probability density functions (PDFs) Py and Px and integrating over the space of possible observations parameterized by y/. With these formulations, the PDF of Y is simple:
y2
Py(3)) = (2ma²) 1/2 exp
11/2
(10)
20
The PDF of X is more complicated because of its dependence on Y, but for any individual observation (8) becomes a shifted Gaussian, giving:
(a x -x 22x y)
Px(xly) (1 p2)) (1-')-1- exp
20 p2)
or
(1- oy )22z
(x(vvv) (1 -12)) -21)-112 exp
(1I)
20 (1 - (2)
(9) then becomes:
consl(Y) /
/ (22m0)
expp exp ( (-20
y2
[og((mo0
(1 - xX 22y - (1-
const(Y)++ +
dy
20
20 - 12)
'That is, independent complex-valued modes from 1...k with independent real and imaginary parts and single, real zero- wavenumber mode.
13

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
HRES
Control
AMSE AR1
AMSE AR12
2.5
N
0.0
(
N
N
2.5
2500
2.5
55%N
1950/h 1950
(
0.0
50
N5
2.5
2
55%N
5
45%N
2
5
5
2t (K)
0
5
45%N
-5
20"W
10
0
10"W
W
10"W
0
10"W
0
Figure 7. Visualization of high-pass filtered forecast and analysis fields for the forecast shown in
const(Y) 10 (oy x) 20 (1 -
(12)
Minimizing (12) for ox while holding σy and p fixed is complicated, but solved numerically the optimal standard deviation ratio ax is less than unity, reaching a minimum of about 0.66 near 0.4 and increasing for both lower and higher values of correlation. This is less intuitive than the o oy optimum of (6), but it still would smooth fine scales much less than the ox = p optimum of (2).
Implementing (12) as loss function would be conceptually interesting, but this seems impractical because the expression has singular behaviour near p 1, where the implied random part of the prediction collapses zero variance.
B. Supplemental Verification
B.1. Visualization
Figure 7 visualizes the high- high- wavenumber components of a sample forecast matching the winter storm Eunice prediction shown in figure The applied filter fourth-order in spherical harmonic space, with the functional form:
A
HPF(K) =
(13)
K
where & is the total wavenumber and kn = 50 the cutoff number, chosen to emphasize modes with length scales 800 km and shorter. Overall, the predictions the control and A.MSE-trained models show very similar structures, but training with (6) as the loss function enhances the high-mode variability the forecasts.
14

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Z500
1850
4700
2t
10
Control
10
10-1
10-1
10
10-2
10-2
10
10-1
10
AMSE AR1
120h
10-1
109
10-2
10
10-1
10
10a
10-1
1
240h
10-1
100
10-2
10-2
10
10
30
90
270
720
270
720
270
720
30
90
270
720
Total wavenumber
Total wavenumber
Total wavenumber
Total wavenumber
Figure 8. Amplitude spectral density for the variables of figure 4 6h, 120h, and 240h lead times.
B.2. Spectra
Figure shows the amplitude spectral density (square root of power spectral density, with units proportional to //veyele) moderate fine scales for several variables and lead times. Because of the energy cascade in the atmosphere, the spectra of most variables follow power-law distributions. Energy in the atmosphere is ultimately removed by turbulent, frictional dissipation, but practical global atmospheric model can effectively resolve these scales.
Nonetheless, the available energy per total wavenumber varies over several orders of magnitude, and even large amplitude density differences between models can appear small on the typical log-log scales of these graphs. The 2m temperature field shows very little smoothing compared to the analysis regardless of model because it is strongly affected by the local elevation, which always supplied as a constant field.
B.3. Quantile/Quantile Plots
Quantile-quantic plots show a joint cumulative density function, and we use them here to evaluate the overall realism the forecasts produced by the control and AMSE AR12 models independently of the forecast skill. In figures 9 and 10, the x-location of each point the labelled percentile of North American weather station observations for Northern Hemisphere winter and summer periods. The y-location of each point the corresponding percentile for the HRES analysis the 5-day forecasts produced by the control and AMSE AR12 models, interpolated to the station locations. For example, in the left panel of figure 9, the 98th percentile corresponds to an observed wind speed of about 11.5m/s, but the 98th percentile of the HRES analysis was about 10m/s.
The y = xr line on the quantile-uaantle plot, shown as a dashed line in each panel, suggests that the forecast and observations have the same unconditional distributions when aggregated, and departures from the diagonal line indicate systematic underprediction or overprediction of extreme values. In our case, figure 9 shows that the AMSE ARI2 model has more realistic representation of surface winds, matching the trends seen in the HRES data. The control model produces noticeably weaker winds at all percentiles, showing systematic shift in the distribution towards weaker surface winds, particularly in summer.
In contrast, figure 10 shows that the models are essentially equivalent in the distribution of 2m temperatures. As discussed in section B.2, the 2m temperature field shows little smoothing in the control model, likely due to the strong influence of elevation on the surface temperature. Improvements the forecast of 2m temperature in the AMSE AR12 model are found
15

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
14
Boreal Winter Valid Oz
Anemometer- Lavel Wind Modulus (ms '1
Boreal Surmmer Valid Oz
Anemometor Sannttr-Levas Levell Wind Modulus (ms
12
Control
AR12
099.25
HRES
D38
p88
Forecasis
Forecasis
P95
250
690.
A75
6
PSO
P50
4
p25
2
2
Observations
Observations
10
14
10
Anemometer-Leen Wind Modulus [ms")]
Anemometer- Wind Modulus [ms")]
Figure Quantile-quantile plots 10 m wind speed surface station locations North American domain. At left, Jan-30 March 2022 (boreal winter), and at right 20 June- 19 September 2022 (boreal summer). The control and AMSE AR12 points show model evaluations for 5-day forecasts. The shaded region denotes confidence interval based on the Kolmogorov-Smine Smimov test.
more in the forecast skill (see figure 11) than in the unconditional distribution of temperatures.
B.4. Details of the Lagged Ensemble Verification
Brenowitz et al. (2025) uses several metrics to evaluate the quality of the lagged ensembles. this work, we use the fair CRPS score, the ensemble root mean squared error (eRMSE), and the spread-error ratio (SER). The eRMSE statistic is derived from its squared version (ensemble mean squared error), evaluated pointwise and integrated over the grid. The SER statistic is the simple ratio of the integrated MSE and ensemble spread (unbiased estimate of variance), noting for emphasis that the ratio is taken after the grid-averaging For ensemble of Ne members (#1...N.) evaluated over Naate forecasts with verifying analysis y, the corresponding formulas are:
Naote
N. (x (1,5))++
CRPS(E, 9) Ndate
Σ
Σ dA(i,))
Ne Σ
d=1
No No
1
I (xx 21(1
(14)
2N - 1)
1/2
Náato
eRMSE(2,9)
Σ dA(i.)) v(i.5))²
and
(15)
Ndate d=1 2.j
Naose En dA(i,)) En (Zx =(i,5))²
1/2
SER(x,
(16)
Ndtte
N
Σ., dA(i, () -((.)))))((
where a(i.j) -. Σ. (i. is the ensemble mean the (i.j) gridpoint.
Figures 11 and 12 show the CRPS and eRMSE skill scores respectively of the lagged ensemble generated with AMSE AR12 compared to the lagged ensemble of the control model for the geopotential (z), temperature (t), specific humidity (q). and u-component of wind (u) at several elevations and for the mean sea level pressure (msl), 2-meter temperature (2t),
u-component of 10m wind (10u), and 6h-accumulated precipitation (tp) at the surface.
For these figures, statistical significance was determined by bootstrapping, sampling 1/3 of the total dates each sample to give an average gap between dates of 36h. The forecast skill of persistence (that is, the gain over climatological forecast by
16

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
5
Boreal Winter 122
p50.
44
Surmmer Valid Oz
Screen-Lod Lovel Temperatue (CC)
0
p25
Screen- Level Temperature ("C)
AMSE AR12
40
38
099.5
-10
P10.
600.
Forecasia
36
-15
DS
34
p66
P2.
32
P80.
-25
30
28
-30
Observations
p50.
Observations
-35
-30
-25
-20
-15
-00
-5
24
26
28
32
36
38
40
Screen Level Temperature [CC]
Level
Screen-Level Temperature [*C]
Figure in figure for 2m temperature. Low percentiles (extreme cold) are shown for the Northern Hemisphere winter, high percentiles (extreme heat) are shown for the Northern Hemisphere summer.
predicting that everything will remain constant) decays very quickly over 36h, samples so-spaced apart reasonably independent of each other.
Overall, AMSE AR12 shows CRPS skill improvements for most variables and most lead times, but total precipitation shows only small improvements at long lead times and degradation at short lead times. This is explained by the separation of modes (5) not being natural one for precipitation. Precipitation is often localized but always non-negative, and consequently its spectral decomposition does not really resemble the normally-distribuce random values that give meaning (5) and (6).
The eRMSE skill chart should be interpreted with caution. The scores of (14)-(16) ensemble, where members are statistically indistinguishable from each other and equally accurate in expectation. This is not really the case for lagged ensemble, where the shorter-duration members should be noticeably more accurate than longer-duration members and an ideal aggregation would separately weight each term. This is not done by Brenowitz et al. (2025) for simplicity and to avoid free parameters, but we believe that the early lead-time smoothing in the control model makes gqual-weighting more optimal for lagged ensemble than for the lagged ensemble of AMSE AR12.
developed for the case of an ideal
For long lead times this advantage diminishes, where the relative degradation of forecast quality much stronger between 0.5 days and 4.5 days than it is between 6 days and 10 days. In this regime, AMSE AR12 begins to show eRMSE skill over the control ensemble.
B. UNBIASED ENSEMBLE RoOT MEAN SQUARED ERROR
The eRMSE formula of (15) a biased estimator the true ensemble mean error, overestimating the error in proportion to the ensemble (sample) spread when the ensemble size is finite.
Consider N. different realizations (x;) of single variable drawn from (p, o²) when the ground-truth value 0. Applying
(15) to this gives:
Ne Σ 24 )
r² N
E (cMSE(x, 0)) E
11²
(17)
which overestimates the true ensemble mean squared error. This overestimate is more severe for small ensembles such as the lagged ensemble configuration of section 3.2, where the ensemble size cannot be easily increased.
Leutbecher & Palmer (2008) proposes correcting this overestimate by subtracting the standard error term to give an unbiased estimator the ensemble mean squared error with finite sample size. In the notation of (15), the corresponding root mean
17

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
10
10
1
250hPa
2
8
4
6
5
6
A
2
4
A
4
u
A
A
A
A
4
A
2
2
4
A
4
5
S
5
6
500h
t
A
A
2
A
3
4
2
3
2
3
3
3
3
A
A
A
A
A
O
A
A
A
2
2
A
A
3
3
A
850hPa
5
6
6
6
3
A
A
A
O
O
O
O
O
O
O
4
A
V
A
msi
2
2
2
2
2
3
B
Surface
2t
A
4
2
2
10u
V
V
O
O
O
A
A
4
4
V
V
A
A
A
A
5
7
8
Lead time (d)
Figure 11. CRPS skill score (% improvement), measured as the relative difference between the CRPS (14) of the 12-step fine-tuned model and the CRPS the control model, for selection of variables and lead times. Orange up-arrows show where the fine-tunes model performs better, blue down-arrows show where the control model performs better. Hollow arrows represent difference of less than I 1%, differences of 2% or larger are marked. Hollow circles mark values that are not statistically significant the 90% level.
squared formula becomes:
Naute
ubLeRMSE(2,.)
dA(i,)) (
Σ
xx(i, -v(i,))
Ndate
d=1 1.
k=1
1/2
Ne - 1) (an(t, -((,))) -(..)))))
-
(18)
which performs this correction pointwise on the grid before computing the spatial average and taking the square root.
Implementing this adjustment slightly improves the scores of the AMSE-tuned model compared to the control model, and the corresponding "scorecard" is shown in figure 13.
18

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
2
2
250hPa
4
2
A
A
500hPa
A
O
O
850nPa
t
A
A
A
A
A
4
O
O
A
Syrface
2t
4
10uu
8
Lead (d)
Figure 12. As figure 11, for ensemble squared error (15).
19

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
2
2
2
2
3
A
250hPa
2
8
5
5
5
O
O
50ONP
A
A
A
o
A
V
O
O
O
O
O
A
A
850nPa
t
O
A
A
B
3
V
O
A
A
msl
Surface
2t
10uu
V
tp
5
5
7
8
Lead (d)
Figure 13. As figures 11 and 12, for the unbiased ensemble root squared error (18).
20

---

Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function
Table Cumulative ranked prohability scores for the models fine-tuned in this study in the lagged ensemble configuration descrihed in section 3.2, for the "headline" variables and levels in figure of Rasp et al. (2024). Lower is better, the best score is bolded and the second- place score is italicized.
Model
z 500hPa (m3 an?s-?)
850hPa (K)
q 700hPa (g kg
u 850hPa -s
2.5d
5.0d
7.5d
2.5d
5.0d
7.5d
2.5d
5.0d
7.5d
2.5d
7.5d
Control
31.038
84.315
162.481
0,428
0.691
1.003
0.357
0.523
0.652
0.823
1.340
1.904
MSE AR12
31.285
82. 100
155.703
0.419
0.664
0,949
0.356
0,526
0.652
0.819
1.335
1.886
MAE AR12
29.969
80.621
155.361
0.410
0,654
0.947
0.340
0.499
0.624
0.811
1.313
1.859
AMSE AR1
33.720
94.703
186.202
0.422
0.721
1.078
0.354
0.558
0.721
0,863
1.485
2.115
AMSE AR12
30.565
80.469
153.267
0.418
0.653
0.935
0.347
0.510
0.634
0.832
1.341
1.882
B.5. Ablation Studies
6h
240h
1.2
1.0
0.8
0.6
0.4
AMSE AR12
MSE AR12
Amplitude ratio
0.2
MAE AR12
Coherence
0.0
32
90
250
720
10
32
250
720
10
250
720
Total wavenumber
Total wavenumber
Total wavenumber
Figure 14. As figure 3, for the comparison models section B.5. Only the model trained with the AMSE function retains sharpness to fine scales.
To ensure that the results of this study not simply artifact of increasing the model's overall training time, compare against two additional fine-tunings:
1. MSE AR12 implements the fine-ture Ane-tuning schedule of table with the unmodified mean squared error loss function, as with GraphCast's principal training.
2. MAE AR12 implements the fine-tuning schedule with a mean absolute error loss function, preserving the per-variable and per-level weightings of error.
Figure 14 shows the aggregated per- wavenumber performance of these models, and table evaluates their CRPS for selection of variables, levels, and lead times in the lagged ensemble configuration.
Both models still show excessive smoothing fine scales, but training with mean absolute error moderately improves sharpness at the medium scales (wavenumbers 32-200 for longer lead times, corresponding length scales 1250-250 250 kilometers).
The excessive smoothing of the MSE-trained model is expected from section 2.1, but that argument does not directly apply to the mean absolute error loss function. However, we can still understand this behaviour intuitively. A model that is optimal under the mean absolute eтror predicts the mean of a distribution, and at longer lead times fine scales are less predictable than coarser scales. Therefore, the prediction of the median future should be smoother than its realization.
Even the moderate improvement sharpness for the MAB-trained model results in improvements to the CRPS the lagged ensemble, shown in table
21


Reviews:
[{"review_id": "cWTVbteQeU", "reviewer": "Reviewer_9jxY", "summary": "This paper addresses a significant issue in state-of-the-art data-driven weather forecasting models: the tendency for forecasts to be overly smooth, particularly at finer scales. This smoothing is attributed to the commonly used mean squared error (MSE) loss function, which penalizes models for misplacing features (the \"double penalty\" effect) and incentivizes averaging away less predictable scales. The authors propose a novel, parameter-free loss function, the Adjusted Mean Squared Error (AMSE), based on a spherical harmonic decomposition of the MSE. AMSE aims to separate errors due to decorrelation from errors in spectral amplitude, encouraging models to maintain realistic variability even for less predictable scales.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.  This review brings up some interesting points regarding the limitations, applications, and extensions of AMSE, and our detailed response follows.\n\n> Briefly looked at the code. It's somewhat complicated the way it's structured and would suggest cleaning that up if authors wish wider adoption.\n\nThis was also mentioned by reviewer KNJY.  Our intention is to first publish the training code and model checkpoints in the spirit of an open science record, and as a result the provided AMSE code is relatively closely coupled to the GraphCast training code.  However, we are looking at testing AMSE with a different (currently unpublished) PyTorch-based model in the coming weeks, and needing a second implementation is sufficient motivation to make the design more modular and accessible.  \n\nIt likely won't be ready in time for a camera-ready version of the paper, but we thank you and KNJY for the encouragement.\n\n> The paper is well connected to prior work. https://arxiv.org/abs/2308.05732 might be a different way to consider this problem.\n\nWe thank you for the reference, and it deserves a place in our literature review.  We knew from papers such as GraphCast (Price 2025) that diffusion models seemed to provide more realistic spectra, but it's very revealing to see a work such at this that arrives at a diffusion-like approach beginning with a motivation to accurately reproduce the low-amplitude portions of the spatial spectrum.  \n\n> \"Effective resolution\" is tricky to define/understand and claiming those numbers in the abstract is somewhat iffy.\n\nThat's a valid complaint about 'effective resolution.'  The implied question is always 'effective at what?'  Our work was motivated by downstream problems like that of (Husain 2024), where the ML-based forecast can only be used to guide downstream applications over scales where it has the correct amount of energy, and our smoothing-based analysis aligns with the cutoff scale used there.\n\nThis is not the only definition of effective resolution.  For example, Kent 2014 (doi 10.1016/j.jcp.2014.01.043) calculates the dispersion relation of various advection schemes and computes both diffusion-limited (smoothing) effective resolutions and dispersion-limited (wave propagation) resolutions.  We will add some discussion of this to the 'limitations' section, to make our assumptions more apparent.\n\nWe don't know enough detail about how GraphCast-like models propagate atmospheric waves to perform a similar study.  However, we do know that the unmodified GraphCast provides reasonably accurate predictions of tropical cyclone locations even as it systematically weakens the storms, so we conjecture that the unmodified model has a diffusion-limited effective resolution that is far coarser than a dispersion-limited effective resolution.  Since AMSE-based fine tuning doesn't seem to degrade the per-wavenumber correlations of the forecast fields (fig. 3), we think we improve the smoothing-limited effective resolution without damaging anything else.\n\nIn recent days, Selz 2025 was published (10.22541/essoar.174139239.94807670/v1) as a preprint.  As a testament to the subtlety of this issue, the paper discusses relative differences in effective resolutions of AI-based weather models without quantifying those resolutions with single numbers.\n\n> Given the poor performance for precipitation, attributed to the spectral decomposition not suiting non-negative, localized fields, do you think alternative modifications or hybrid approaches within the AMSE framework (or separate loss terms) could specifically improve precipitation forecasts while retaining the benefits for other variables?\n\nFor precipitation itself, the best option is to probably look at other error measures entirely, but the idea of doing this for our paper seemed like cherry-picking.  Ultimately, precipitation is a hybrid random variable: it doesn't rain a finite fraction of the time, so there's a qualitative difference between zero precipitation and 0.1mm of precipitation.  That's most suited to double-variable predictand (chance of precipitation, expected precipitation if present), with a combination of a cross entropy loss function with something MSE-like, but this is a study that belongs elsewhere.\n\nHowever, other, smoother variables like dewpoint temperature, total column water content, or net evaporation-less-precipitation are correlated with precipitation and might be more suitable for AMSE-based loss calculations.  AIFS (Lang 2024) includes the first two of those, but we wanted to maintain strict compatibility with the control GraphCast model.", "claims_and_evidence": "Central claim is that data-driven weather models trained with standard Mean Squared Error (MSE) suffer from unrealistic smoothing due to the \"double penalty\" effect. This is a known problem (even in computer vision). The authors analyze MSE optimization properties and authors show the smoothing behavior during training. \n\nThe proposed Adjusted Mean Squared Error (AMSE) loss function, derived by spectrally separating amplitude and coherence terms, mitigates this smoothing. This is visually compared and shown as well as via spectral analysis and quantile plots of wind distributions. The evidence provided is generally clear and convincing for the main claims, particularly regarding improved sharpness, resolution, and tropical cyclone intensity. Ensemble evals are trickier and authors acknowledge that. Save with respect to variables like precipitation where spherical decomposition is less meaningful.", "methods_and_evaluation_criteria": "Methods and evaluation criteria are relatively standard and make sense for the problem. Evaluations includes a fair mix of qualitative and quantitative methods. The derivation of MSE in terms of spectral power density and coherence (Equation 4)  relies on Parseval's theorem for spherical harmonics  and appears standard.", "theoretical_claims": "I didn't do a rigorous mathematical check of the derivations or proofs. The core theoretical claim linking MSE optimization to smoothing seems sound and widely accepted in the NWP and ML communities.", "experimental_designs_or_analyses": "The experiment design and analyses were quite reasonable. Starting from publicly available GraphCast checkpoint and fine-tuning on the HRES dataset is a valid and efficient design choice to demonstrate the specific impact of the loss function change. The staged increase in forecast steps during fine-tuning is also quite standard.", "relation_to_broader_scientific_literature": "The paper is well connected to prior work. https://arxiv.org/abs/2308.05732 might be a different way to consider this problem.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- While spectral losses aren't entirely new, the specific formulation is simple and easier to apply as finetuning.\n- Addressing the smoothness issue is crucial for the operational viability and downstream use of data-driven models. \n- The paper is generally well-written and clearly explains the problem, the proposed method, and the results with supporting figures.\n\nWeaknesses:\n- Likely won't work with irregular grids.", "other_comments_or_suggestions": "\"Effective resolution\" is tricky to define/understand and claiming those numbers in the abstract is somewhat iffy.", "questions_for_authors": "Given the poor performance for precipitation, attributed to the spectral decomposition not suiting non-negative, localized fields, do you think alternative modifications or hybrid approaches within the AMSE framework (or separate loss terms) could specifically improve precipitation forecasts while retaining the benefits for other variables?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUDVMnSOcH", "reviewer": "Reviewer_KNJY", "summary": "A spherical loss variation of the MSE is introduced that breaks MSE's tendency to push ML models to converge to the mean via its double penalty. The prestented AMSE effectively conserves amplitudes in weather forecasts, as demonstrated with a carefully fine tuned GraphCast model, leading to sharper forecasts that are the base for well calibrated ensembles.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we thank the reviewer for their comments.  They accurately note limitations of this work and suggest extensions that will improve the robustness of these results in a camera-ready version of the paper. \n\nSeveral of the suggested ablation studies should be possible, but they will take long enough that results won't be available during this comment period.  We will include them as supplemental material in the full version of the paper.\n\nOur more specific responses are as follows:\n\n> A comparison to L1 as fine tuning metric would be interesting.\n\nAn L1 study can be included in the camera-ready version of the paper. We don't expect to see strong differences, since L1-optimality predicts the median rather than mean of the distribution.  The Aurora model [Bodnar 2024, arxiv:2025.13062] was trained with L1 error, and it shows similar smoothing (figures I37 and I38).\n\n> [T]he AMSE fine tuned models seem to overestimate amplitudes at long wavelengths. This is well addressed to noise that seems to be imputed in the forecasts. It would be great qualitatively see some of those noisy forecasts to somewhat understand what type of noise appears. \n\nAs a minor correction, the amplitude overestimation is at large wavenumbers and short wavelengths.  The revised paper will include an auxiliary length scale to ensure clarity with the figures. We will also show examples of this overestimation through high-pass filtered forecast fields.  This overestimation seems to be an amplification of grid imprinting [arxiv:2212.12894, section 7.5.3] that already exists in the Graphcast model.\n\n> It would therefore be interesting to see how the amplitude-preserving loss introduced here performs on other datasets, such as moving MNIST. Edit: I understand that the method is only applicable for spherical domains.\n\nSpherical geometry is helpful, but a similar derivation is possible on the plane.  The core idea is that AMSE groups together modes that differ only by grid rotation (eq. 3 $\\to$ 4), and on the sphere this nicely divides modes into equivalence classes based on total wavenumber.  On the (hyper)plane, a similar construction is possible that groups together modes with similar-enough total wavenumbers ($\\sqrt{k^2 + l^2 + m^2}$).  The recently-announced Chakraborty 2025 [arxiv:2502.00472] does this to develop its own modified error measure, but unlike AMSE Chakraborty only considers the amplitude (energy) component.  Overall, we're optimistic that AMSE might apply to a wide variety of fluid/turublence problems, but we don't want to over-claim in the paper without firm proof.\n\nApplication of AMSE to moving MNIST or other image/video problems is less straightforward.  High-frequency modes in images or video correspond to edges, so it's not obvious that the Fourier modes can be lumped together as equivalent.  A similar problem arises in weather with precipitation, where positivity (i.e. no negative precipitation) breaks the independence of spherical harmonic modes; AMSE slightly degrades the Graphcast precipitation forecast (figures 11 and 12).\n\n> Also, how does the introduced loss perform when used not only for fine tuning, but for an entire model training.\n\nThis is a question we've considered, but training at a fine enough scale to show the smoothing is not currently practical.  In theory, AMSE should be applicable throughout the training process, but in practice it might result in learning problems.  We know that MSE-based training smooths/sharpens during learning (fig. 2), so the smoothing might force the model to learn to predict large scales first, acting as implicit regularization that speeds learning.  Discussion of this would be added to the 'limitations' section of a camera-ready paper.\n\nLow-resolution studies like Karlbauer aren't great test-beds for AMSE comparisons because the atmosphere is more predictable at large scales. For example, fig. 3 shows that for short lead times there is barely any smoothing noticeable at wavenumber 32, corresponding to the 5.625° resolution of Karlbauer's examples.\n\n> I am skeptic about the comparison of a fine tuned vs not fine tuned model. Is the improved sharpness attributable to the new loss, or just a result of careful fine tuning? I am concerned that the deliberate fine tuning curriculum might also be a source of improved sharpness.\n\nWe will add a fine-tuned control model for the camera-ready version.  The process here was based on [Subich 2024, arxiv:2408.14587], and there the author showed that fine-tuning to 12-steps changed smoothing only slightly (figure 12).  Since smoothing is optimal under MSE, we should expect smoothing out of any MSE-trained model and be surprised by any exceptions.\n\n> A standalone pip install amse or alike would be great for applicability.\n\nPlease see our response to 9jxY for discussion of this.\n\n> Some older deep learning weather prediction approaches could be cited as well\n\nAgreed, also Keisler 2022 as the foundational GNN weather prediction model.", "claims_and_evidence": "Looks good. I only have one major concern, as detailed in point 4 under Experimental Designs Or Analyses. If this can be addressed, I'm happy to raise my score of this work. I like this work and the presented analyses a lot and hope it will be polished further for publication.", "methods_and_evaluation_criteria": "1. L1 could also help to prevent regression to the mean and would be even simpler. A comparison to L1 as fine tuning metric would be interesting.", "theoretical_claims": "Not carefully checked.", "experimental_designs_or_analyses": "1. Figure 3 and 7 demonstrate that the amplitude is nicely preserved, that's exciting! But the AMSE fine tuned models seem to overestimate amplitudes at long wavelengths. This is well addressed to noise that seems to be imputed in the forecasts. It would be great qualitatively see some of those noisy forecasts to somewhat understand what type of noise appears. The AMSE AR1 forecasts seem well suited for such a qualitative analysis and I'd be curious to see it.\n2. Regression to the mean is a prominent problem in spatiotemporal forecasting tasks. It would therefore be interesting to see how the amplitude-preserving loss introduced here performs on other datasets, such as [moving MNIST](https://paperswithcode.com/dataset/moving-mnist). Edit: I understand that the method is only applicable for spherical domains.\n3. Also, how does the introduced loss perform when used not only for fine tuning, but for an entire model training. I'm wondering whether the model converges in a similar way (or worse/better). Such an ablation could also be performed on few variable prediction only with a small model to reduce the computational burden, e.g., [this study](https://arxiv.org/abs/2407.14129).\n4. I am skeptic about the comparison of a fine tuned vs not fine tuned model. Is the improved sharpness attributable to the new loss, or just a result of careful fine tuning? I am concerned that the deliberate fine tuning curriculum might also be a source of improved sharpness.", "relation_to_broader_scientific_literature": "Some older deep learning weather prediction approaches could be cited as well, such as Dale Durran's and Jonathan Weyn's pioneering works with CNNs, e.g., this work:\n\nhttps://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020MS002109", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "_originality_: The introduced loss component is novel and well motivated. It is also well justified with amplitude analyses.\n\n_significance_: Fixing the double penalty in spatiotemporal forecasting tasks marks a solid contribution to deep learning weather prediction.\n\n_clarity_: The manuscript is well structured and organized. Arguments are clearly motivated and approachable.", "other_comments_or_suggestions": "- Line 28, right: ERA5 dates back to 1949 by now.\n- Figure 4: A quick description for the metrics would be good. That is, CRPS and eRMSE are best when small, whereas spread/error supposedly (?) should be at 1.0. Edit: found it in line 378, left. But might be good to have this in the figure caption too to prevent the case that I ran into.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EyjUcWpt38", "reviewer": "Reviewer_Udrq", "summary": "This paper introduces a simple, parameter-free modification to the loss function, separating decorrelation loss from spectral amplitude errors. Fine-tuning the GraphCast model with this new loss function yields sharper deterministic forecasts, increases effective resolution from 1,250 km to 160 km, improves ensemble spread, and enhances predictions of tropical cyclone strength and surface wind extremes. This advancement addresses a key limitation in data-driven models, significantly improving forecast precision and detail.\n\n## Update after rebuttal\nWhile I recognize the authors’ efforts to address my concerns, I believe the current version of the paper still lacks the necessary quality and impact to meet the ICML acceptance standards.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.  While we disagree with some of the reviewer's conclusions (see below), the fact that they raise these issues demonstrates that some sections of the paper should be rephrased in a camera-ready version.  We also think that we can add further extreme-weather verification in a camera-ready version of the paper.  Our specific rebuttal follows:\n\n> While Figure 5 and 6 show improvements in specific cases [...] these are individual examples and lack broader statistical validation.\n\nFigure 5 is indeed a specific case study (hurricane Ian), but it is included for illustration and qualitative understanding only.  Figure 6 is a systematic evaluation of the model performance with respect to all tropical cyclones for June–September 2022, and it is statistically robust.  Tropical cyclones are ideal for systematic evaluation such as this because they are rigorously defined and independently observed, and no other observation record comes close for model verification of compact, extreme events.\n\n> Figure 4 reveals no significant RMSE improvement over the original GraphCast, raising doubts about the method's overall effectiveness. \n\nFirst, Figure 4 (comparable to figure 2 of Brenowitz 2024) shows RMSE of the (lagged) ensemble mean, not RMSE of individual forecasts.  We will clarify the text to make this difference more obvious.\n\nSecond, we should not expect large differences to the RMSE of the ensemble mean.  AMSE-based training improves forecast variability, but that variability largely 'averages out' over a large ensemble. That being said, mean-RMSE is a biased estimator (see Leutbecher 2008, doi 10.1016/j.jcp.2007.02.014) which we nonetheless use to match Brenowitz.  \n\nIf we used the unbiased estimator of Leutbecher, we would expect to see our results show modest improvements in mean-RMSE over the control Graphcast model.  We will add a discussion of this to the supplementary material of the revised paper.\n\nOverall, we consider CRPS to be the preferred metric since it directly incorporates the ensemble spread.  By this metric, our results show consistent improvements.\n\n> [T]he authors should provide statistical evaluations across more extreme weather events\n\nWe reiterate that figure 6 is such a systematic evaluation for tropical cyclones, but we would welcome any specific suggestion of neglected evaluations.  This is a very challenging problem, particularly for models that do not fully resolve convection (thunderstorms); see for example Marsigli et al. 2021 (doi 10.5194/nhess-21-1297-2021) and Ebert 2008 (doi 10.1002/met.25) for a general overview.\n\nWe may be able to provide more detailed verification with the data on hand.  Would you consider systematic evaluation of min- and max-pooled observations a useful addition? \n\n> Essential References Not Discussed: \n\nWe will certainly add brief discussion of [2] and [3] to the literature review section.  [2] demonstrates that a dramatically different training approach (learning directly from observational data) still produces a forecast model that blurs predictions, and [3] demonstrates that very high resolution training is insufficient to solve the problem.  \n\nWe request clarification about why [1] is considered 'essential', since it's a proof-of-concept study at very low resolution (5.625°) that cannot effectively demonstrate blurring or the lack of it.\n\n>  The work builds on GraphCast by fine-tuning it with a new loss function, which, while effective, does not introduce a fundamentally new approach.\n\nWhile this work uses GraphCast as the basis for implementation, the problem of forecast blurring is a general one, seen in essentially every deterministic ML-based forecast system.  We argue that AMSE is broadly applicable, and please see the response to reviewer KNJY about applicability to other fluid dynamics problems.\n\n>  The ensemble forecasting approach averages predictions from consecutive initialization times, which, although used in prior literature, is not a standard ensemble method.\n\nFirst, the ensemble approach does not _average_ predictions from different lead times, it treats the different initializations as distinct ensemble members.  It's uncertain if this is just a typo or whether there's a deeper misunderstanding of the approach.\n\nSecond, we think that the lagged ensemble approach is the best one available for fundamentally deterministic models such as GraphCast.  A \"true\" ensemble of initial conditions involves either non-public data (such as the ensemble data assimilation members from ECMWF/IFS) or an extreme number of free parameters to create a 'balanced' initial perturbation.  \n\nWe agree with Brenowitz 2024 that while the lagged ensemble makes for a poor forecast system, it provides the fairest comparison between different systems because it isolates the properties of the ML model from the circumstances of its initialization.", "claims_and_evidence": "The claim that AMSE improves extreme weather predictions is not fully supported. While Figure 5 and 6 show improvements in specific cases (e.g., tropical cyclones), these are individual examples and lack broader statistical validation. Additionally, Figure 4 reveals no significant RMSE improvement over the original GraphCast, raising doubts about the method's overall effectiveness. To strengthen the claim, the authors should provide statistical evaluations across more extreme weather events and clarify the relationship between AMSE's benefits and traditional metrics like RMSE.", "methods_and_evaluation_criteria": "The methods seem to make sense, but the evaluation is not sufficient", "theoretical_claims": "Yes, they are correct.", "experimental_designs_or_analyses": "I have reviewed all sections of the results, and the experimental design and evaluation are sound and valid. The authors' experiments are strictly aligned with the setup used in GraphCast, ensuring a fair comparison.", "relation_to_broader_scientific_literature": "The paper addresses the smoothing effect in data-driven weather forecasting caused by traditional MSE loss functions, a well-known issue in the field. By introducing the AMSE loss function, which separates decorrelation loss from spectral amplitude errors, the work aligns with efforts to improve fine-scale variability and extreme weather predictions. While it enhances effective resolution and ensemble spread, the authors acknowledge limitations in ensuring physical plausibility for out-of-distribution scenarios. This contribution advances the integration of data-driven models into operational forecasting, bridging gaps between machine learning and traditional physics-based approaches.", "essential_references_not_discussed": "[1] Verma, Y., Heinonen, M., & Garg, V. (2024). ClimODE: Climate and weather forecasting with physics-informed neural ODEs. arXiv preprint arXiv:2404.10024.\n\n[2] Vaughan, A., Markou, S., Tebbutt, W., Requeima, J., Bruinsma, W. P., Andersson, T. R., ... & Turner, R. E. (2024). Aardvark weather: end-to-end data-driven weather forecasting. arXiv preprint arXiv:2404.00411.\n\n[3] Han, Tao, et al. \"Fengwu-ghr: Learning the kilometer-scale medium-range global weather forecasting.\" arXiv preprint arXiv:2402.00059 (2024).", "other_strengths_and_weaknesses": "### Strengths:\nThe proposed AMSE loss function significantly improves fine-scale variability and extreme weather predictions, addressing a critical limitation in data-driven forecasting. This has direct relevance for operational weather centers and public safety.\n\nThe paper is well-written, with clear explanations of the methodology, experiments, and results.\n\n### Weaknesses:\nLimited Novelty: The work builds on GraphCast by fine-tuning it with a new loss function, which, while effective, does not introduce a fundamentally new approach. This limits the perceived originality of the contribution.\n\nNon-Standard Ensemble Method: The ensemble forecasting approach averages predictions from consecutive initialization times, which, although used in prior literature, is not a standard ensemble method. This could raise questions about the robustness and generalizability of the ensemble results.", "other_comments_or_suggestions": "See above", "questions_for_authors": "Addressing these weaknesses, particularly by enhancing novelty and adopting more standard ensemble techniques, would strengthen the paper's overall contribution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles a significant and well-known issue in data-driven weather forecasting: the tendency of models trained with MSE to produce overly smooth forecasts, particularly losing detail at finer scales due to the "double penalty" effect. The authors propose a novel, parameter-free loss function, the Adjusted Mean Squared Error (AMSE), derived from a spherical harmonic decomposition. AMSE aims to disentangle loss components related to decorrelation and spectral amplitude errors, thereby encouraging models to retain realistic variability. The authors demonstrate their method by fine-tuning the GraphCast model, showing improvements in forecast sharpness, effective resolution, ensemble spread characteristics, and the prediction of extreme events like tropical cyclone intensity and surface winds.
The overall reviewer assessment strongly supports acceptance. The paper addresses a highly relevant and impactful problem in a large-scale ML application domain. The proposed AMSE loss function is a methodologically sound and clever approach specifically designed to tackle the identified weakness of MSE. The empirical results convincingly demonstrate significant improvements in forecast sharpness and the representation of important extreme weather phenomena (tropical cyclones), which are key areas where current data-driven models falter.
Decision: Accept (poster)