ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs
Authors: Yogesh Verma
Markus Heinonen
Vikas Garg
Venue: ICLR 2024 oral (2024)
Keywords: neural ODE
time-series forecasting
climate prediction
physics-informed ML
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce a novel climate and weather modeling approach, inspired by physics, using ODEs that capture underlying inductive biases and allow for uncertainty quantification in predictions.

Abstract: Climate and weather prediction traditionally relies on complex numerical simulations of atmospheric physics. Deep learning approaches, such as transformers, have recently challenged the simulation paradigm with complex network forecasts. However, they often act as data-driven black-box models that neglect the underlying physics and lack uncertainty quantification. We address these limitations with ClimODE, a  spatiotemporal continuous-time process that implements a key principle of advection from statistical mechanics, namely, weather changes due to a spatial movement of quantities over time. ClimODE models precise weather evolution with value-conserving dynamics, learning global weather transport as a neural flow, which also enables estimating the uncertainty in predictions. Our approach outperforms existing data-driven methods in global and regional forecasting with an order of magnitude smaller parameterization, establishing a new state of the art.

Published as a conference paper ICLR 2024
CLIMODE: CLIMATE AND WEATHER FORECASTING WITH PHYSICS-INFORMED NEURAL ODES
Yogesh Verma, Markus Heinonen
Vikas Garg
Department of Computer Science
Yai Ltd and Aalto University vgargocsail wwt.mmteeee...
Aalto University, Finland
(yogesh. verma, markus. (eeinonen)@aaltee
ABSTRACT
Climate and weather prediction traditionally relies on complex numerical simulations of atmospheric physics. Deep learning approaches, such as transformers, have recently chal- lenged the simulation paradigm with complex network forecasts. However, they often act as data-driven black-box models that neglect the underlying physics and lack un- certainty quantification. We address these limitations with ClimODE, a spatiotemporal continuous-time process that implements a key principle of advection from statistical me- chanies, namely, weather changes due to spatial movement of quantities over time. ClimODE models precise weather evolution with value-conserving dynamics, learning global weather transport as neural flow, which also enables estimating the uncertainty in predictions. Our approach outperforms existing data-driven methods global and re- gional forecasting with order of magnitude smaller parameterization, establishing state of the art.
INTRODUCTION
slate-of-the-arl climate and weather prediction relies on high-precision numerical simulation of complex atmospheric physics (Phillips, 1956; Satoh, 2004; Lynch, 2008). While accurate to medium timescales, they are computationally intensive and largely proprietary (NOAA, 2023; ECMWF, 2023).
There is a long history of 'free-form' neural networks challenging the mechanistic simulation paradigm (Kuligowski & Barros, 1998; Baboo & Shereef, 2010), and recently deep learning has demonstrated signif- icant successes (Nguyen et al., 2023). These methods range from one-shot GANs (Ravuri al., 2021) autoregressive transformers (Pathak et al., 2022; Nguyen et al., 2023; Bi et al., 2023) and multi-scale GNNs (Lam et al., 2022). Zhang et al. (2023) combines autoregression with chysics-insspied transport flow.
In statistical mechanics, weather can be described as flxx, spatial movement of quantities over time, governed by the partial differential continuity equation (Broomé & Ridenour, 2014)
transport
compression
du d
Vu
uV
(I)
advection
time evolution
where u(x, t) is quantity (e.g. temperature) evolving over space x Ω and time t E driven by flow's velocity v(x, t) e and sources s(x,t) (see Figure 1). The advection moves and redistributes existing weather 'mass' spatially, while sources add or remove quantities. Crucially, the dynamics need be continuous-time, and modeling them with autoregressive 'jumps' violates the conservation of mass and incurs approximation errors.

---

Published as conference paper ICLR 2024
(a) State
(b) Neural Velocity
(c) Divergence V
(d) Advection ú
(e) Transport Vo
(1) Compression uV Vo
4471
601
1.07
1207
007
5012
1207E
1807
1007
3207W
60'W
0011
10072
3007
3807
320PW
50'W
0012
320ne
3207
Figure 1: Weather as a quantity-preservinn by a neural flow velocity (b), whose divergence is the flow's compressibility (c). The flow translates into state change by advection (d), which combine quantity's transport (e) and compression (f).
advection system. A quantity (eg. temperature) (a) is moved
We introduce a climate model that implements continuous-time, second-order neural continuity equation with simple yet powerful inductive biases that ensure - by definition - value-conserving dynamics with more stable long-horizon forecasts. We show computationally practical method to solve the continuity equation over entire Earth as a system of neural ODEs. We learn the flow v as a neural network with only a few million parameters that uses both global attention and local convolutions. Furthermore, we address source variations via a probabilistic emission model that quantifies prediction uncertainties. Empirical evidence underscores ClimODE's ability attain state-of-the-arr global and regional weather forecasts.
1.1 CONTRIBUTIONS
We propose to learn continuous-time PDE model, grounded on physics, for climate and weather modeling and uncertainty quantification. In particular,
we propose ClimODE, continuous- neural advection PDE climate and weather model, and derive its ODE system tailored to numerical weather prediction.
we introduce a flow velocity network that integrates local convolutions, long-range attention the ambient space, and Gaussian emission network for predicting uncertainties and source variations.
empirically, ClimODE achieves state-of-the-ar global and regional forecasting performance.
Our physics-inspieed model enables efficient training from scratch on single GPU and comes with an open-source Py Torch implementation on GitHub.
RELATED WORKS
Numerical climate and weather models. Current models encompass numerical weather prediction (NWP) for short-term weather forecasts and climate models for long-term climate predictions. The cutting- edge approach in climate modeling involves earth system models (ESM) (Hurrell et al., 2013), which inte- grate simulations of physics of the atmosphere, cryosphere, land, and ocean processes. While successful,
'ttpps: / /ttp:/gggtttttt/////g/ wwtp.//alloo.../eee........
2

---

Published as a conference paper ICLR 2024
Table Overview of current deep learning methods for weather forecasting.
Method
Value-greserving
Explicit PeriodicittySccaooaally
Uncertainty
Contimaous-inng
Parameters (M)
FourCastNet
N/A
Potbak et al. (2022)
GraphCast
37
Lam et (2022)
Weagh-Weather
256
Bi et al. (2023)
ClimaX
107
Nguyen el al. (2023)
NowcastNet
N/A
7hang al. (2023)
ClimODB
2.8
this work
they exhibit sensitivity to initial conditions, structural discrepancies across models (Balaji et al., 2022), re- gional variability, and high computational demands.
Deep learning for forecasting. Deep learning has emerged as compelling alternative to NWPs, focusing on global forecasting tasks. Rasp al. (2020) employed pre-training techniques using ResNet (He al., 2016) for effective medium-range weather prediction, Weyn et al. (2021) harnessed a large ensemble of deep-learning models for sub-seasonal forecasts, whereas Ravuri et (2021) used deep generative models of radar for precipitation nowcasting and GraphCast (Lam et al., 2022; Keisler, 2022) utilized a graph neural network-based approach for weather forecasting. Additionally, recent state-of-the-art neural forecasting models of ClimaX (Nguyen et al., 2023), FourCastNet (Pathak et al., 2022), and Pangu-Weathen (Bi et al., 2023) are predominantly built upon data-driven backbones such as Vision Transformer (ViT) (Dosovitskiy et al., 2021), UNet (Ronneberger et al., 2015), and autoencoders. However, these models overlook the fundamental physical dynamics and do not offer uncertainty estimates for their predictions.
Neural ODEs. Neural ODEs propose learning the time derivatives as neural networks (Chen al., 2018; Massaroli et al., 2020), with multiple extensions to adding physics-based constraints (Greydanus et al., 2019; Cranmer al., 2020; Brandstetter al., 2023; Choi al., 2023). The physics-insping embed mechanistic understanding in neural ODEs (Raissi et al., 2019; Cuomo et al.. 2022), while multiple lines of works attempt to uncover interpretable differential forms (Brunton al., 2016; Fronk & Petzold, 2023). Neural PDEs warrant solving the system through spatial discretization (Poli et al., 2019; Iakovlev et al., 2021) or functional representation (Li al., 2021). Machine learning has also been used to enhance fluid dynamics models (Li et al., 2021; Lu et al., 2021; Kochkov et al., 2021). The above methods are predominantly applied to only small, non-climate systems.
networks (PINNs)
3 NEURAL TRANSPORT MODEL
Notation. Throughout the paper x denotes spatial gradients, = du time derivatives, inner dr product, and V . = ir(Vv) divergence. We color equations purely for cosmetic clarity.
3.1 ADVECTION EQUATION
We model weather as a spatiotemporal process uxx, t) = (uz (x, t).... uK ux(x, t)) E R of K quantities un (x, R over continuous time R and latitude-longinude locations (h, w) E [-90", 90%]
[- 1180 180%] C R.2. We assume the process follows an advection partial differential equation
ix(xx,
Vx (x,
Vug (x,
U. (x, (x(
2
transport
compression
where quantity change du (x, t) is caused by the flow, whose velocity VA (x t) E Ω transports and concen- trates air mass (see Figure 2). The equation (2) describes closed system, where value uy is moved around
3

---

Published as a conference paper ICLR 2024
(a) Perpendicular, incompressive -u O, uV-we wp-==0
(b) Upward, compressive
(c) Downward, spreading
v. v 0. v-c
v 0. up-v>0
Figure 2: Conceptual illustration of continuity equation pointwise temperature change i(xo, () -V. Vu- - uV v. (a) A perpendicular flow (green) to the gradient (blue to red) moves equally hot air causing change at Xy. (b) Cool air moves upwards, decreasing pointwise temperature, while air concen- tration at Xy accumulates additional temperature. (c) Hot air moves downwards increasing temperature at Xg, while air dispersal decreases it.
but never lost or added. While a realistic assumption on average, we will introduce an emission source model Section 3.7. The closed system assumption forces the simulated trajectories us (x(x, 1) to vulue-preserving manifold
f un (x, t)dx const,
Vt, k.
(3)
This is strong inductive bias that prevents long-horizon forecast collapses (see Appendix H for details.)
3.2 FLOW VELOCITY
Next, we need way to model the flow velocity v(x, L) (See Figure 1b). Earlier works have remarked that second-order bias improves the performance of neural ODEs significantly (Yildiz et al., 2019; Gruver et al., 2022). Similarly, we propose second-order flow by parameterizing the change of velocity with a neural network fo.
(x () = So (u(0), Vu((), v(),
(4) :x E 22 E x its gradients Vu(t) E x
function of the current state u
the current velocity v(t) = xv, f) x E 2} E R2 and spatiotemporal embeddings
R xx These inputs denote global frames (e.g., Figure 1) at time discretized to a resolution (H, W)
with a total of 5K quantity channels and embedding channels.
3.3 2ND-ORDER PDE AS A SYSTEM OF FIRST-ORDER ODES
We utilize the method of lines (MOL), discretizing the PDE into grid of location-specifec ODEs (Schiesser, 2012; lakovlev et al., 2021). Additionally, a second-order differential equation can be transformed into a pair of first-order differential equations (Kreyszig, 2020; Yildiz et al., 2019). Combining these techniques yields a system of first-order ODEs (t),(t), (t)( of quantities at locations x;:
/
dr d= 1 ( [ + [.
{ (-V- (ue(T)vv (u(T)) Vu(T), (T), (T), )).
dr, (5)
where 7 E R is an integration time, and where we apply equations (2) and (4). Backpropagation ODEs is compatible with standard autodiff, while also admitting tractable adjoint form (LeCun et al., 1988; Chen et al., 2018; 8; Metz et al., 2021). The forward solution u(t) can be accurately approximated with numerical solvers such as Runge-Kutta (Runge, 1895) with low computational cost.
4

---

Published as conference paper ICLR 2024
(a) CLimODE
(b) Advection ODE
Convalution
Y++1
Me+1:7
Advection
Emicaion Model
Attention
Velacity
Advection = V (x,y)
M,1
Figure 3: Whole prediction pipeline for ClimODE.
3.4 MODELING LOCAL AND GLOBAL EFFECTS
PDEs link acceleration (x, () solely the current state and its gradient at the location and time t, ruling out long-range connections. However, long-range interactions naturally arise as information prop- agates over time across substantial distances. For example, Atlantic weather conditions influence future weather patterns in Europe and Africa, complicating the covariance relationships between these regions. Therefore, propose hybrid network to account for both local transport and global effects,
fo (t),,, Vu((), v(t) = -eonny (t)), (u v(),, 1))
+Y fatt (u(),, Vu (), v(1), 2b)
(6)
convolution network
attention network
Local Convolutions To capture local effects, we employ local convolution network, denoted fconv- This network is parameterized using ResNets with 3x3 convolution layers, enabling it to aggregate weather information up distance of 'pixels' away from the location x, where corresponds to the network's depth. Additional parameterization details can be found in Appendix C.
Attention Convolutional Network We include an attention convolutional network sart which captures global information by considering states across the entire Earth, enabling long-distance connections. This attention network is structured around KQV dot product, with Key, Query, and Value parameterized with CNNs. Further elaboration is provided in Appendix C.2 and is learnable hyper-parameter.
3.5
SPATIOTEMPORAL EMBEDDING
Day and Season
We encode daily and seasonal periodicity of time t with trigonometric time embeddings
2Tl 2xl
(i() sin 2rrt, cos 2mt, sin cos
365
365
Location
We encode latitude h and longitude with trigonometric and ccherical-positioon encodings (xx) [[sin, cos} (h, w), sin(h) cos(w), (i)))]
(
Joint time-location embedding We create joint location-time embedding by combining position and time encodings (v(t) x (x)), capturing the cyclical patterns of day and season across different locations on the map. Additionally, we incorporate constant spatial and time features, with (h)) and ((w) representing 2D latitude and longitude maps, and Ism and oro denoting static variables in the data,
(x, t) (t), (x), vb(t) x (xx), (())),
((c)
[b),,), (ww), Ism, oro].
(9)
These spatiotemporal features additional input channels to the neural networks (See Appendix B).
5

---

Published as a conference paper ICLR 2024
3.6 INITIAL VELOCITY INFERENCE
The neural transport model necessitates initial velocity estimate, Vx(X, 40), to start the ODE solution (5). traditional dynamic systems, estimating velocity poses a challenging inverse problem, often requiring encoders in earlier neural ODEs (Chen et al., 2018; Yildiz et al., 2019; Rubanova et al., 2019; De Brouwer et al., 2019). In contrast, the continuity Equation (2) establishes an identity, ú + V (uv) = 0, allowing us solve directly for the missing velocity, v, when observing the state We optimize the initial velocity for location x, time t and quantity k with penalised least-squares
|| + (t) Vun (1) + ( va va oI + allvx(00ll))
VR (t) arg min
(10)
va(()
where V is numerical spatial derivative, and oo())) is numerical approximation from the past states u(t < fo)- We include Gaussian prior ((vec v& K) with Gaussian RBF kernel Kij rb, xj) that results spatially smooth initial velocities with smoothing coefficient a. See Appendix D.5 for details.
3.7 SYSTEM SOURCES AND UNCERTAINTY ESTIMATION
The model described so far has two limitations: (i) the system is deterministic and thus has no uncertainty, and (ii) the system is closed and does not allow value loss gain (eg. during day-night cycle). We tackle both issues with an emission outputting a bias (4k (x(x, t) and variance of (x, t) of ux(xx t) as a Gaussian,
obs 1/
xx 1) ~ N un (x, () + pa (x, 1), of (xxx)) 0),
MAN (x, ox, = gu (u(x, ()
(11)
The variances of represent the uncertainty of the climate estimate, while the mean /4 represents value gain bias. For instance, the μ can model the fluctuations in temperature during the day-night eycle. This can be regarded emission model, accounting for the total alcatoric and epistemic variance.
3.8 LOSS
We assume a full-earth dataset (yI..... of total of N timepoints of observed frames y; R xx times 44. We assume the data is organized into dense and regular spatial grid (H, W). common data modality. We minimize the negative log-likelihood of the observations yi.
(1gg.V (v.l((y))l())) (yslu((s)) + (((t), o ((t)) log.N (a (1))).))))))
C(0; D)
(12)
NKHW
where we also add Gaussian prior for the variances with a hypervariance A, to prevent variance explosion during training. We decay the \ using cosine annealing during training to remove its effects and arrive at a maximum likelihood estimate. Further details are provided in Appendix
4 EXPERIMENTS
Tasks We assess ClimODE's forecasting capabilities by predicting the future state u,+ based on the initial state u, for lead times ranging from At 6 to 36 hours both global and regional weather prediction, and monthly average states for climate forecasting. Our evaluation encompasses global, regional and climate forecasting, discussed in Sections 4.1, 4.2 and 4.3, focusing on key meteorological variables.
Data. We use the preprocessed 5.625 resolution and hour increment ERAS dataset from WeatherBench (Rasp et al., 2020) in all experiments. We consider K 5 quantities from the ERA5 dataset: ground
6

---

Published as conference paper ICLR 2024
2
v10
U10
12m
1000
RMSE
750
500
230
01
1.0
0.9
0.7
0.4
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
NODE
ClimaX
ClimODE
*
Figure 4: RMSE(L) and ACC(1) comparison with baselines. ClimODE outperforms competitive neural methods across different metrics and variables. For more details, Table 6.
temperature (t 2mm,,, atmospheric temperature (t),, geopotential (z), and ground wind vector (ul0 O, v1 and normalize the variables to [0, 1] via min-max scaling. Notably, both z and t hold standard importance as verification variables in medium-range Numerical Weather Prediction (NWP) models, while t22 and (ul O, v1 0) directly pertain to human activities. We use ten years training data (2006-15), the validation data 2016 validation, and two years 2017-18 as testing data. More details can be found in Appendix B.
Metrics. We assess benchmarks using latihude-weighted RMSE and Anomaly Correlation Coefficient (ACC) following the de-normalization of predictions.
Eup. Bee Mekw
RMSE
Σ
HW Σ
(h)) )2 ACC
(13)
N
v Ve,n, a(h))a Een. a(t) thw
where a<h) = cos( /1)// 1 Σ cos(h') is the latitude weight and ÿ = C and ā u - C' are averaged against empirical mean C E, Vihw- More detail in Appendix C.3.
Competing methods. Our method is benchmarked against exclusively open-source counterparts. We com- pare primarily against ClimaX (Nguyen al., 2023), a state-of-the-ar Transformer method trained on same dataset, FourCastNet (FCN) (Pathak et al., 2022), large-scale model based on adaptive fourier neural operators and against a Neural ODE. We were unable to compare with PanguWeather (Bi et al., 2023) and GraphCast (Lam al., 2022) due to unavailability of their code during the review period. We ensure fairness by retraining all methods from scratch using identical data and variables without pre-training.
Gold-standard benchmark. We also compare to Integrated Forecasting System IFS (ECMWF, 2023). of the most advanced global physics simulation model, often known simply the 'European model'. Despite its high computational demands, various machine learning techniques have shown superior perfor- mance the IFS, evidenced (Ben Bouallegue al., 2024), particularly when leveraging multitude variables and exploiting correlations among them, our study focuses solely on a limited subset of these variables, with IFS serving as the gold standard. More details can be found in Appendix D.
7

---

Published as conference paper ICLR 2024
Table 2: RMSE(() comparison with baselines for regional forecasting. ClimODE outperforms other com- peting methods in m2m, and achieves competitive performance u10, v10 across all regions.
North-America
South-America
Australia
Value
Hours
NODE
ClimaX
ClimODE
NODE
ClimaX
ClimODE
NODE
ClimaX
ClimODE
232.8
273.4
134,5
10.6
225. 60
205 40
107.7
20.2
251
190.2
103.8 = 14,6
329.5
225.0
17.3
365.6
220.15
169.4
29.6
344.8
184.7
170.7 21.0
7
667.2
543.0
307.7 + 25.4
551.9
269.
237.8
32.2
539.9
222.2
211.1 + 33.6
494.8
32.3
660.3
301.81
292.0
38.9
632.7
324.9
308.2 = 30.6
1.96
+ 0.06
1.38
0.97
0.13
1.37
1.19
1,05 0.12
12
3.34
L86
1.81 = 0.13
2.18
1.62
1.25 + 0.18
2.18
1.30
1.20 = 0.16
2.75
1,79
2.68
1.39
0.21
24
5.39
2.27
2.23 = 0.18
3.41
1.97
1,65
+ 0.26
3.32
1.92
1,63
0.24
2.63
L.75
1.61 = 0.2
2.12
L.85
1.33 + 0.26
1.88
1.57
0.80 = 0.13
1.57
12m
3.53
2.27
1.96 = 0.33
2.60
2.15
0.98 + 0.17
3.51
1.72
1,23 0.24
1,93
2.15 = 0.20
2.56
2.23
1.17 +
2.46
2.15
0.25 1.25
1.54
1.27
1.25
1.91
1.40
1,35 0.17
12
2.91
2.24
2.01 + 0.20
2.74
1.57
1,49
+ 0.23
2.86
1.77
1.78 + 0.21
3,24
2.17 =
183
+
3,44
2.03
1,96
3.96
3.14
2.34 + 0.32
3.77
2,04
2,08
+ 0.35
3.91
2.64
2.33
0.33
2.16
L83
1.67 = 0.23
2.29
1.31
1.30
+ 0.21
2.38
1.47
1,44 = 0.20
v10
+
+
3.60
1.79
3.96
3.52
2.31 + 0.37
4.16
1.90
2.07
+ 0.31
4.31
2.33
2.23
0.23
4.57
3.39
2.50
1 0.41
4.76
2.14
2.43
0.34
4.88
2.58
2.53
0.32
4.1 GLOBAL WEATHER FORECASTING
We assess ClimODE's performance global forecasting, encompassing the prediction of crucial meteo- rological variables described above. Figure 4 and Table 6 demonstrate ClimODE's superior performance across all metrics and variables over other neural baselines, while falling short against the gold-standard IFS, as expected. Fig. 10 reports CRPS (Continuous Ranked Probability Score) over the predictions. These findings indicate the effectiveness of incorporating an underlying physical framework for weather modeling.
4.2 REGIONAL WEATHER FORECASTING
We assess ClimODE's performance in regional forecasting, constrained the bounding boxes of North America, South America, and Australia, representing diverse Earth regions. Table 2 reveals noteworthy out- comes. ClimODE has superior predictive capabilities forecasting ground temperature (t 2m), atmospheric temperature (t), and geopotential (z). It also maintains competitive performance in modeling ground wind vectors (u10 and v)0) across these varied regions. This underscores ClimODE's proficiency in effectively modeling regional weather dynamics.
4.3 CLIMATE FORECASTING: MONTHLY AVERAGE FORECASTING
To demonstrate the versatility of our method, assess its performance in climate forecasting. Climate forecasting entails predicting the average weather conditions over a defined period. In our evaluation, we focus monthly forecasts, predicting the average values of key metcorological variables over one-month durations. We maintained consistency by utiliz the same ERA5 dataset and variables employed previous experiments, and trained the model with same hyperparameters. Our comparative analysis with FourCastNet on latitude-weighted RMSE and ACC illustrated in Figure 10. Notably, ClimODE demonstrates signifi- cantly improved monthly predictions compared to FourCastNet showing efficacy in climate forecasting.
8

---

Published as conference paper ICLR 2024
v1.0
u10
12m
1.00
0.100
RMSE
0.03
0.75
0.075
0.02
0.50
0.030
0.25
0.025
0.01
12
18
24
0.00
12
10
24
12
10
24
0.000
12
10
26
12
18
24
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
NODE
NODE+AdV
AOVV+Ad
CliMODE
Figure 5: Effect of Individual Components: The importance of individual model components. An ablation showing how iteratively enhancing the vanilla neural ODE (blue) with advection form (orange), global attention (green), and emission (red), improves performance of ClimODE. The advection component brings about the most accuracy improvements, while attention turns out be least important.
ABLATION STUDIES
12m
Effect of emission model Figure shows model predic- tions uxx, t) of ground temperature (t2mm) for a specific location while also including emission bias μ(x.t) and variance o² xx (). Remarkably, the model captures diur-
0.63
Ooservable Values (Normalized)
0.80
0.10
0.75
nal variations and effectively estimates variance. Figure 8 highlights bias and variance on global scale. Posi- tive bias is evident around the Pacific ocean, correspond- ing daytime, while negative bias prevails around Europe and Africa, signifying nighttime. The uncertainties indi- cate confident ocean estimation, with northern regions be- ing challenging.
3 Time points
Figure 6: Effect of bias: 22 observed and predicted values showcasing the effect bias.
Effect of individual components We analyze the contri- butions of various model components to its performance.
Figure 5 delineates the impact of components: NODE is free-form second-order neural ODE, Adv corre- sponds to the advection ODE form, At adds the attention in addition to convolutions, and C1 LmmDEE adds also the emission component. All components bring performance improvements, with the advection and emission model having the largest, and attention the least effect. More details are in Appendix E.
6 CONCLUSION AND FUTURE WORK
We present ClimODE, a novel climate and weather modeling approach implementing weather continu- ity. ClimODE precisely forecasts global and regional weather and also provides uncertainty quantification. While our methodology grounded scientific principles,
essential to acknowledge its inherent lim- itations when applied climate and weather predictions in the context of climate change. The historical record attests to the dynamic nature of Earth's climate, yet remains uncertain whether ClimODE can re- liably forecast weather patterns amidst the profound and unpredictable climate changes anticipated in the coming decades. Addressing this formidable challenge and also extending our method on newly curated global datasets (Rasp al., 2023) represents compelling avenue for future research.

---

Published as conference paper ICLR 2024
ACKNOWLEDGEMENTS
We thank the researchers at ECMWF for their open data sharing and maintenance of the ERA5 dataset, without which this work would not have been possible. We acknowledge CSC Center for Science, Finland, for providing generous computational resources. This work has been supported by the Research Council of Finland under the HEALED project (grant 13342077).
REFERENCES
Santhosh Bahoo and Kadar Shereef. An efficient weather forecasting system using artificial neural network. International journal of environmental science and development, 1(4):321, 2010.
V Balaji, Fleur Couvreux, Julie Deshayes, Jacques Gautrais, Frédéric Hourdin, and Catherine Rio. Are general circulation models obsolete? Proceedings of the National Academy of Sciences, 119(47), 2022.
Zied Ben Bouallegue, Mariana CA Clare, Linus Magnusson, Estibaliz Gascon, Michael Maier-Gerber, Mar- tin Janouvek, Mark Rodwell, Florian Pinault, Jesper S Dramsch, Simon TK Lang, et al. The rise of data-driven weather forecasting: A first statistical assessment of machine learning- based weather fore- casts in operational-like context. Bulletin of the American Meteorological Society, 2024.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619:533-538, 2023.
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh Gupta. Clifford neural layers for PDE modeling. In ICLR, 2023.
Sofia Broomé and Jonathan Ridenour. A PDE perspective climate modeling. Technical report, Depart- ment of mathematics, Royal Institute of Technology, Stockholm, 2014.
Steven Brunton, Joshua Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15): 9932-3937, 2016.
Ricky Т. Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
Hwangyong Choi, Jeongwhan Choi, Jeehyun Hwang, Kookjin Lee, Dongeun Lee, and Noseong Park. Cli- mate modeling with neural Divection-diffusiie equation. Knowledge and Information Systems, 65(6): 2403-2422 2407, 2023.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv, 2020.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed neural networks: Where we and what's 's next. Journal of Scientific Computing, 92(3):88, 2022.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yes Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-obserye time series. In NeurlPS, 2019.
Alcxey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
10

---

Published as conference paper ICLR 2024
ECMWF. IFS Documentation CY48RI. ECMWF, 2023.
Colby Fronk and Linda Petzold. Interpretable polynomial neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(4), 2023.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. NeurIPS, 2019.
Nate Gruver, Marc Finzi, Samuel Stanton, and Andrew Gordon Wilson. Deconstructing the inductive biases of Hamiltonian neural networks. In ICLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on Computer Vision and Pattern Recognition, 2016.
James Hurrell, Marika Holland, Peter Gent, Steven Ghan, Jennifer Kay, Paul Kushner, J-F Lamarque, William Large, Lawrence, Keith Lindsay, et al. The community earth system model: framework for collaborative research. Bulletin of the American Meteorological Society, (119))1339-1600 2013.
Valerii lakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning contimuous-time PDEs from sparse data with graph neural networks. In ICLR, 2021.
Ryan Keisler. Forecasting global weather with graph neural networks. arXiv preprint arXiv:2202.07575. 2022.
Dmitrii Kochkov, Jamie Smith, Ayya Alieva, Qing Wang, Michael Brenner, and Stephan Hoyer. Machine serming-acceleratee computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
Erwin Kreyszig. Advanced engineering mathematics. Wiley, 10th edition, 2020.
Robert Kuligowski and Ana Barros. Localized precipitation forecasts from numerical weather prediction model using artificial neural networks. Weather and forecasting. 13(4): 194-1204, 1998.
Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexander Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton-Rosen, et al. GraphCast: Learning skillful medium-range global weather forecasting. arXiv, 2022.
Yann LeCun, Touresky, Hinton, and T Sejnowski. theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume I, pp. 21-28. San Mateo, CA. USA, 1988.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR, 2021.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-22) 2021.
Peter Lynch. The origins of computer weather prediction and climate modeling. Journal of computational physics, 227(7):3431-3444 2008.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural ODEs. NeurIPS, 2020.
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman, Gradients are not all you need. arXiv, 2021.
11

---

Published as a conference paper ICLR 2024
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. ClimaX: foundation model for weather and climate. In ICML, 2023.
NOAA. The global forecasting system. Technical report, National Oceanic and Atmospheric Admin- istration, 2023. URL, emc amc..cep........................ noaa gov gww.//cc/pagg/////////// httpaagee//eerrraaaaaae foreccsst Systtess//ts/ts/ssss gfs/documentation. php.
Adam Paszke, Sam Gross, Francisco Massa, Adam Leter, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli et al. FourCastNet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv, 2022.
Norman Phillips. The general circulation of the atmosphere: A numerical experiment. Quarterly Journal the Royal Meteorological Society, (((55)):133-144 1956.
Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations, arXiv, 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Stephan Rasp, Peter Duehen, Sebastian Scher, Jonathan Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: benchmark data set for data-driven weather forecasting. Journal of Advances in Model- ing Earth Systems, 12(11), 2020.
Stephan Rasp, Stephan Hoyer. Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Vivian Yang, Rob Carver, Shreya Agrawal, et Weatherbench 2: A benchmark for the next generation of data-driven global weather models. arXiv preprint arXiv:2308. 15560, 2023.
Suman Ravuri, Karel Lenc. Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsi- mons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597:672-677 677, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 2015.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeularly-sampled time series. NeurIPS, 2019.
Carl Runge. Über die numerische auflösung von differentialgleichunge. Mathematische Annalen, 46(2): 167-178, 1895.
Masaki Satoh. Atmopperic circulation dynamics and circulation models. Springer, 2004.
William Schiesser. The numerical method of lines: integration of partial differential equations. Elsevier, 2012.
Jonathan Weyn, Dale Durran, Rich Caruana, and Nathaniel Cresswell-Clayy Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models. Journal of Advances in Modeling Earth Systems, 13(7), 2021.
12

---

Published as a conference paper ICLR 2024
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. NeurlPS, 2019.
Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, pp. 1-7, 1-7,2022 2023.
13

---

Published as conference paper ICLR 2024
A
ETHICAL STATEMENT
Deep learning surrogate models have the potential to revolutionize weather and climate modeling by pro- viding efficient alternatives to computationally intensive simulations. These advancements hold promise for applications such nowcasting, extreme event predictions, and enhanced climate projections, offering potential benefits like reduced carbon emissions and improved disaster preparedness while deepening our understanding of our planet.
B DATA
We trained our model using the preprocessed version of ERA5 from WeatherBench (Rasp et al., 2020). It is standard benchmark data and evaluation framework for comparing data-driven weather forecasting models. WeatherBeench regridded the original ERA5 at 0.25 to three lower resolutions: 5.625., 2.8125,, and 1.40625 We utilize the 5.625 resolution dataset for our method and all other competing methods. See https: https:///ooflluaaen/// fuencee ecmwf. int documentat at on for more details on the raw ERA5 data and Table summarizes the variables used.
/CKB/////////// (CKB/ERA5$ /RA55 3A data+
Table 3: ECMWF data variables in dataset. Static variables are time-independent Single represents surface-level variables, and Atmospheric represents time-varying atmospheric properties at chosen altitudes.
Type
Variable name
Abbrev.
ECMWF ID
Levels
Static
Land-sea mask
Ism
172
Static
Orography
Single
2 metre temperature
t2m
167
Single
10 metre U wind component
u10
165
Single
10 metre wind component
v10
166
Atmospheric
Geopotential
129
500
Atmospheric
Temperature
850
B. SPHERICAL GEOMETRY
We model the data in 2D latitude-longitud grid Ω2 but take the earth geometry into account by considering circular convolutions at the horizontal borders (international date line), and reflective convolutions at the vertical boundaries (north and south poles). We limit the data to latitudes #88 to avoid the grid rows collapsing to the poles at +900.
C IMPLEMENTATION DETAILS
C.1 MODEL-HYPERPARAMETUN
C.2 ATTENTION CONVOLUTIONAL NETWORK
We include an attention convolutional network fatt which captures global information by considering states across the entire Earth, enabling the modeling of long-distance connections. This attention network is struc- tured around Key-Query- Qulue-Value dot product attention, with Key, Query, and Value maps parameterized as convolutional neural networks as,
14

---

Published as conference paper ICLR 2024
Table 4: Default hyperparameters for the emission model
Hyperparameter
Meaning
Value
Padding size
Padding size of each convolution layer
Padding type
Padding mode of each convolution layer Kernel size of each convolution layer Stride of each convolution layer
X: Circular, Y: Reflection
Kernel size Stride
3
Residual blocks
Number of residual blocks
[3,2,2] [128, 64, out channels] 0.1
Hidden dimension
Number of output channels of each residual block
Dropout
Dropout rate
Table 5: Default hyperparameters for the convolution network Sconv
Hyperparameter
Meaning
Value
Padding size
Padding size of each convolution layer
Padding type
Padding mode of each convolution layer Kernel size of each convolution layer Stride each convolution layer
X: Circular, Y: Reflection
Kernel size
3
Stride
Residual blocks
Number of residual blocks
[5,3.2]
Hidden dimension Dropout
Number of output channels of each residual block
[128, 64, out channels] 0.1
Dropout rate
Key (K), Value (V): Key and Value maps are parameterized 2-layer convolutional neural net- works with stride=2 and CK.v as the latent embedding size. Based on the stride, this embeds every 4th pixel into key, value latent vector of size Ck.v. We collect all embeddings into tensor.
Query (Q): Query map is parametrized 2-layer convolutional neural networks with stride=1 and Co as the latent embedding size. This incorporates somewhat local information embeds into Co latent vector. We collect all embeddings into one tensor.
We compute the attention maps via dot- produrtdu maps as,
3 softmax(QK TV
(14)
We consider post-attention map for the attention coefficients l-layer convolutional network with x 1 filter size to map the latent vectors into output channels.
C.3 METRICS
We assess benchmarks using latitude- waiihtee-eigghted RMSE and Anomaly Correlation Coefficient (ACC) follow- ing the de-normalization of predictions.
N
H
w Σ
Exu Veh) uekw
RMSE
(h)(w MeRw)?, ACC
(15)
HW
v Ei,n, a(h) Er,., af
where a<h) chs( # Σ" cos(h') is the latitude weight and = y and ā C' are averaged against empirical mean C E, Vthu- The anomaly correlation coefficient (ACC) gauges model's
15

---

Published as conference paper ICLR 2024
ability predict deviations from normal conditions. Higher ACC values signify better prediction accuracy, while lower values indicate poorer performance. It's a vital tool in meteorology and climate science for evaluating a model's skill in capturing unusual weather climate events, aiding forecasting system as- sessments. Latitude-weighter RMSE measures the accuracy of a model's predictions while considering the Earth's curvature. The weightage by latitude accounts for the changing area represented by grid cells different latitudes, ensuring that error in climate spatial data are appropriately assessed. Lower latitude- weighted RMSE values indicate better model performance capturing spatial or climate patterns.
D TRAINING DETAILS
D.1 DATA NORMALIZATION
We utilize 6-hourly forecasting data points from the ERA5 dataset and considered 5 quantities from the ERA5 dataset: ground temperature (t 2m), atmospheric temperature (t), geopotential (z), and ground wind vector (u10, v10) and normalize the variables to [0, 1] via min-max scaling. We use ten years training data (2006-15), 2016 validation data, and 2017-18 as testing data. There are 1460 data points per year and 2048 spatial points.
D.2 DATA BATCHING
In our experiments, we utilize K quantities (See Appendix B) and spatial discretization of the earth to resolution (H, W) = (32, 64) resulting in a total of 3KWH = 30720 scalar ODEs. This can seem daunting. but they all share the same differential function fe. that is. the time evolution at Tokyo and New York follows the same rules. The system can then be batched into single image stack(u(() v(t;; s/] of size (3K + C, H, W), which is input fo(.) TR3K +CxHxW pass.
x33KHXX and can be solved in one forward
(1) BB(K+)))XX
advection So
B
(16)
We batch the data points wrt to years, giving the batch of shape (N x B x (3K + C) x H x W). where B is the batch size and N here denotes the number years. We used batch-size B to train our model.
D.3 OPTIMIZATION
We used Anniin--nnnelln scheduler for the learning rate and also for the variance weight Å, for L2 norm shown in Fig. loss in Eq. 12. We trained our model 300 epochs, and the scheduler variation is shown below.
D.4 SOFTWARE AND HARDWARE
The model is implemented PyTorch (Paszke et al., 2019) utilizing torchdiffeq (Chen et al., 2018) to manage our data and model training. We use euler our ODE-solver that solves the dynamical system forward with time resolution of hour. The whole model training and inference is conducted on a single 32GB NVIDIA V100 device.
/ttp////////////////ooo///////// CosineAmeealingle inn.. html
16

---

Published as conference paper ICLR 2024
0,0005
0.0004
leamiigg rate
0.0006
0.0002
0.0004
0.0001
0.0002
0.0000
0
50
100
150
200
250
300
50
100
150
200
250
300
Epocns
Coochs
Figure 7: Learning rate and X schedule wrt epochs.
D.5 INITIAL VELOCITY INFERENCE
The neural transport model necessitates an initial velocity estimate, Vu (x, to), to initiate the ODE system (5). We estimate the missing velocity directly, v, as a preprocessing step, for location x, time and quantity to match the advection equation by penalized least-squares, where ú is approximated by examining previous states u(t to) to obtain a numerical estimate of the change at to.
[i.(())) + (t) Ven (t) + un (1) va o|| - allvx(0lkk)
v& (t) arg min va(()
(17)
where (, } are numerical derivatives over time space. We compute (t) by utilizing torchcubiespline" fit {ug(( 2), ug(l (1-(411) 1), un (4) get smooth derivative approximation. The spatial gradients V are calculated using torch. gradient function of Torch. We additionally place Gaussian zero-mean prior (vee 0, K) with Gaussian RBF kernel K,, (x,, xj) that results in spatially smooth initial velocities with smoothing coefficient a. The distance for the rb, xj) is computed as the euclidean norm between x; and x,. This is optimized separately for each location of the initial time ty. We use Adam optimizer with a learning rate of for 200 epochs. To get a balance between smoothing, local and global pattern we set the smoothing coefficient a 10-7
ABLATION STUDY COMPONENTS
We conducted an extensive analysis evaluate the individual contributions each model component its overall performance, as illustrated in Fig. 5. We delineate the impact of different components as,
NODE: A basic second-order neural differential equation as, here fronv is parametrized by ResNet with the same set of parameters shown in Table 5,
ig (x, (x, t)
(18)
vx = fconv ((t). Vu ((t)
(19)
NODE+Adv:
This combines the second-order neural differential equation with the advection com-
ponent, where foonv is parametrized by ResNet with the same set of parameters shown in Table 5,
úg (x, t) VA (x, t) Vuk (x, ux(x, t) (x(
(20)
v. vx(x,t) Sconv [ ()) Vu (1) (1),
(21)
'tttps: /ttp//gttttttt//////////tt http:s///stttte.e.//errrrre/e//r
17

---

Published as conference paper ICLR 2024
NODE+Adv+Au: This the NODE+Adv with the attention convolutional network model both local and global effects, where fconv,att is parametrized by ResNet with the same set of parameters shown in Table 5 and Section C.2,
dg (xc, t) (vx(x,t t)- (x,(x,t) Ux (x(xx tjV (x(
(22)
(x,,,t) t) Sconv (u()) (t), Vu v(t), u) + Wfat u(t). Vu((), v(),
(23)
ClimODE encompasses all the previous components with the emission model component, including the bias and variance components. The NODE, NODE +Adv, NODE +dv++++++++ Att is trained by minimizing the MSE be- tween predicted and truth observation as they output point prediction and do not estimate uncertainty in the prediction. We employ unweighted RMSE as our evaluation metric compare these methods. Our findings reveal discernible hierarchy of performance improvement by incorporating each component, underscoring the vital role played by each facet enhancing the model's downstream performance.
The Fig. 8 showcases the effect of emission model in modeling diurnal cycle and also predicting uncetainty.
(a) Bias u(x, t)
(b) Std o(x t)
0.29
0.048
0.38
46'N
0.00
0"
0.024
0.016
00.00
-0.09
60'W
120/19
6CTM
Figure 8: Effect of emission model: Global bias and standard deviation maps at 12:00 AM UTC. The bias explains day-night cycle (a), while uncertainty is highest land, and in north (b).
F RESULTS SUMMARY
G LONGER HORIZON PREDICTIONS
Table 7 showcases the comparison of our method with ClimaX for for 72 hours (3 days) and 144 hours (6 days) lead time on latitude weighted RMSE and ACC metrics. observe that the temperature and potential (t.12m,z) relatively stable over longer forecasts, while the wind direction (u10,v10) becomes unreliable over a long time, which is an expected result. ClimaX also remarkably stable over long predictions but has lower performance. We see that method achieve better performance compared ClimaX for longer horizon predictions.
H VALIDITY OF MASS CONSERVATION
To empirically study this, we analyzed how our current model retains the mass-conservation assumption and computed the integrals Ix.6 J (x(x, t) over time and quantities. We discovered that the value constant time up to -1
18

---

Published as conference paper ICLR 2024
Table 6:
Latitude weighted RMSE(L) and ACC(() comparison with baselines on global forecasting on
ERA5 dataset.
RMSF(L)
ACC(1)
Variable
Lead-Time (hours)
NODE
ClimaX
FCN
IFS
ClimODE
NODE
ClimaX
FCN
IFS
ClimODE
6
300.64
247.5
149.4
26,9
102.9 =9.3
0.96
0.97
0.99
1.00
0.99
12
460.23
265.3
217.8
(N/A)
134.8 + 12.3
0.88
0.96
0.99
(N/A)
0.99
627.65
319.8
(N/A)
162.7
0.95
(NIA)
24
877.82
364.9
333,0
51.0
193.4 = 16.3
0.70
0,93
0.99
1,00
0.98
36
1028.20
455.0
449.0
(N/A)
259.6 + 22.3
0.55
0.89
0.99
(N/A)
0.96
6
1.82
1.64
1.18
0.69
1.16 = 0.06
0.94
0.94
0.99
0.99
0.97
2.32
1.77
(N/A)
(N/A)
18
2.93
1,93
1.65
(N/A)
1.47
0.16
0.77
0.92
0.99
(NIA)
0.96
24
3.35
2.17
1,83
0.87
1.55 = 0.18
0.72
0.90
0.99
0.99
0.95
2.49
(N/A)
0.26
(N/A)
2.02
0.97
0.82
12m
12
3.16
2.26
L.48
(NIA)
1.45 = 0.10
0.68
0.90
0.99
(NIA)
0.96
18
3,45
2.45
1.61
(N/A)
1.43 0.09
0.69
0.88
0.99
(N/A)
0.96
2.37
1.02
0.09
0.79
36
4.17
2.87
1.90
(N/A)
1.70 0.15
0.49
0.83
0.99
(N/A)
0.94
6
2.3
1.58
1.47
0.80
1.41
0.07
0.85
0.92
0.95
0.98
0.91
3.13
1.96
(N/A)
0.09
0.70
0.88
0.93
(NIA)
0.89
u10
18
3.41
2.24
2.05
(N/A)
0.58
0.84
0,91
(NIA)
0.88
24
4.1
2.49
2.33
1.11
2.01
0.10
0,50
0.80
0.89
0.97
0.87
36
4.68
2.98
(N/A)
0.18
0.35
0.69
0.85
(N/A)
0.83
6
2.58
1.60
1.54
0.94
0.08
0.81
0.92
0.94
0.98
0,92
12
3.19
1.97
1.81
(N/A)
0.61
0.88
0,91
(N/A)
0.89
v10
18
3.58
2.26
2.11
(N/A)
1.96 + 0.16
0.46
0.83
0.86
(NIA)
0.88
24
4.07
2.48
2.39
1.33
2.04
0.10
0.35
0.80
0.83
0.97
0.86
36
4.52
2.98
(N/A)
0.24
0.29
0.69
0.75
(NIA)
0.83
Table 7: Longer lead time predictions: Latitude weighted RMSE(L) and ACC(+) for longer lead times in global forecasting using the ERA5 dataset, in comparison with ClimaX.
RMSE(L)
ACC(1)
Variable
Lead-Time (hours)
ClimaX
ClimODE
ClimaX
ClimODE
72
687.0
478.7 248.3
0.73
0.88 +0004
801.9
783.6 +37.3
0.58
0.61 10.13
3.17
2.58 10.16
0.76
0.85 10.06
144
3.97
3.62 +0.21
0.69
0.77 +0.16
72
2,87
2.75 +0.49
0.83
0.85 +0. 14
t2m
144
3.38
3.30 10.23
0.83
0.79 +0.25
72
3.70
3.19=0018
0.45
0.66 +0.04
144
4.24
4,02 10.12
0.30
0.35 40.08
3.80
3,30 #0.22
0.39
0.63
v10
144
4.42
4.24=0.10
0.25
0.32 +0.11
I CRPS (CONTINUOUS RANKED PROBABILITY SCORE) AND CLIMATE FORECASTING
We further assessed model using CRPS (Continuous Ranked Probability Score), depicted in Figure 10. This analysis highlights our model's proficiency in capturing the underlying dynamics, evident in its accurate prediction of both mean and variance.
19

---

Published as conference paper ICLR 2024
v10
U10
12m
0.0010
Tu +0
0,3000
-0.0005
0.0010
Time
Time
Time
Time
Time
Figure 9: Validity of the mass conservation assumption of the ODE.
To showcase the effectiveness of our model climate forecasting, we predicted average values over a one- month duration for key meteorological variables sourced from the ERA5 dataset: ground temperature (t2m), atmospheric temperature (t:), geopotential (z). and ground wind vector (u1 0, v1 Employing identical pata-preprocessinn steps, normalization, and model hyperparameters detailed previous experiments, Figure 10 illustrates the performance of ClimODE compared to FourCastNet in climate forecasting, Particu- larly noteworthy is our method's superior performance over FourCastNet at longer lead times, underscoring the multi-facete efficacy of our approach.
t2m
u10
v10
1200
RMSE
1000
830
2
2
Lead Time (Month)
Lead Time (Month)
Lead Time (Month)
Lead Time (Month)
Lead Time (Month)
0.020
CRP5
8.01%
0.005
16
24
18 Lead Time (hr)
12
12 Lead Time (hr)
12
24
10
21
Lead Time (hr)
Lead Time (hr)
Lead Time (hr)
ClimODE
Figure 10: CRPS and Monthly Forecasting: RMSE(L) comparison with FourCastNet (FCN) for monthly forecasting and CRPS scores for ClimODE.
J CORRELATION PLOTS
To demonstrate the emerging couplings of quantities (ie. wind, temperature, pressure potential), below plot the emission model urred (x, 6) E TR5 pairwise densities averaged over space x and time t. These effectively capture the correlations between quantities in the simulated weather states. These show that temperatures (1,22m) and potential (z) are highly correlated and bimodal; the horizontal and vertical wind direction are independent (u10,v10); and there is little dependency between the two groups. These plots indicate that the emission model is highly aligned with data and does not indicate any immediate biases or skews. These results are averaged over space and time, and spatially local variations are still possible. The mean f plots show that means match data well. The standard deviation plots show some bimodality of predictions with either no or moderate uncertainty.
20

---

Published as conference paper ICLR 2024
0.8
0.6
0.4
0.2
0.8
0.4
1.0
0.5
0.2
0.8
0.4
0.3
0.7
0.6
0.5
0.2
Figure 11: Pairwise correlation among the predicted variables by the model.
21

---

Published as conference paper ICLR 2024
12m
010
Uppoa
Upua
Unx
v10
Uped
Mice
Figure 12: Correlation between Upred and Utrue for different observables, showing the efficacy of our model to predict the observables accurately.
Uose
v10
Houe
Myue
Figure 13: Correlation hetween / and Utrue for different observables.
22

---

Published as a conference paper ICLR 2024
Mlur
Urse
v10
Figure 14: Correlation between a and Utruee for different observables.
23


Reviews:
[{"review_id": "SVh6838jQy", "reviewer": "Reviewer_F1k5", "summary": "This paper introduces ClimODE, a neural ODE combining a convolutional local mechanism and a global attention mechanism to predict one timestep of weather evolution. It uses ERA5 data from WeatherBench 1 and compares it to ClimaX model and against a standard Neural ODE.", "questions": "1. What do you mean by one-shot GAN referring to Ravuri et al., 2021? Is there any pre-training involved\n2. There is Weatherbench 2 available now, maybe to recent to be included in this submission, but should be mentioned in future work: https://arxiv.org/abs/2308.15560, https://sites.research.google/weatherbench/\n3. Are any of the competing methods such as GraphCast, PanguWeather, FourCastNet that now available?\n4. Is the comparison against ClimaX fair? The main idea of ClimaX is to use pre-training, but you state you use all methods without pertaining\n5. The statement \"IFS is still far ahead of any deep learning method\" doesn't really hold anymore, e.g.: https://arxiv.org/pdf/2307.10128.pdf\n6. Kreislers GNN work should be mentioned here as well: https://arxiv.org/pdf/2202.07575.pdf\n7. This work could be useful too: https://arxiv.org/pdf/2304.04664.pdf", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality\n- First NeuralODE work applied to this problem\n- Extending Neural ODEs to be more effective for the specific climate problem\n\nQuality\n- High-quality writing, figures\n- Using relevant, real-world data for evaluation\n\nClarity\n- Written well and understandable\n- Detailed explanations\n\nSignificance\n- Satisfies the need for ML climate models to be value-conserving and have a probabilistic forecast\n- Contributes towards faster climate and weather modeling, potentially using less computational resources", "weaknesses": "- A couple of Figures (e.g Figure 5) would benefit from a longer caption/description\n- Not comparing against state-of-the-art methods, such as GraphCast\n- It should be compared against pre-trained ClimaX or other methods that don't require pre-training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ansHBkgdok", "reviewer": "Reviewer_K2uQ", "summary": "This paper proposes ClimODE, a novel climate modeling approach that leverages physics-based constraints. It represents climate dynamics as a continuous-time advection process governed by partial differential equations (PDEs). The PDEs are discretized into ODEs using the method of lines, with the velocity field modeled by a neural network integrating convolutions and attention. Gaussian emission models estimate the prediction uncertainties and source variations. Empirically, ClimODE outperforms existing data-driven methods in global and regional forecasting tasks, highlighting the efficacy of continuous-time physical constraints.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Physical Prior as the Foundation:** \n   The paper ingeniously grounds its methodology in the continuity equation, a well-established physical prior. This not only lends an elegant formulation but also ensures an interpretable and efficient model. The fair integration of this physical prior into the deep model showcases an exemplary fusion of the first principles with modern DL techniques.\n\n2. **Gaussian Emission Model:** \n   The adoption of the Gaussian as an emission model is both impressive and reasonable. It aptly addresses uncertainties and unknown sources in climate forecasting, offering a reasonable approach to handling the inherent unpredictability of the domain.\n\n3. **Experimentation and Insight:** \n   The experimental setup and investigation presented in the work are both rigorous and enlightening. The thoroughness of the research provides valuable insights and sets an easy-to-access benchmark for future endeavors in the field.\n\nAlso, the presentation is clear and easy to follow.", "weaknesses": "1. For the model of FLOW VELOCITY (section 3.2), as we already know $\\dot{\\mathbf{v}}_k(\\mathrm{x}, t) = \\ddot{u}_k(\\mathrm{x}, t)$,   why we still need to parameterize it? Is it because the computation of $\\ddot{u}_k(\\mathrm{x}, t)$ is too costly? There are also many methods that could approximate $\\ddot{u}_k(\\mathrm{x}, t)$ based on $\\dot{u}_k(\\mathrm{x}, t)$. More discussion or claims are encouraged. \n\n2. The model treats the advection PDE as independent for each task or quantity. However, in the intricate tapestry of climate dynamics, various quantities are interdependent. For instance, wind patterns can influence temperature fluctuations.  The paper doesn't clearly illustrate how the proposed method addresses these inherent inter-correlations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i9e6vF7N6b", "reviewer": "Reviewer_THXC", "summary": "The authors present ClimODE, a new method for weather forecasting that provides uncertainty esimates, benchmarked on ERA5 data from WeatherBench. This method is based on first principles from physics, namely the *continuity equation* which mathematically formulates that in a system, quantities are conserved (modulo any sources/sinks). Neural ODEs are used to model the continuity PDE and the flow velocity is parametrized using a neural net that has two components: A CNN that models local interactions and an attention mechanism that model long-range interactions. Furthermore, an emission model is added on top that serves to estimate first and second order moments of the true underlying solution (bias and uncertainty). \n\nClimODE greatly outperforms the baseline NODE which doesn't take into account the physical principles of advection nor does it correct its solution by the proposed emisison models, moreover, ClimODE also outperform ClimaX, one of the recent state-of-the-art weather forecasting models on the regridded 5.625 ° ERA5 data.\n\nFinally, strong ablation study results are shown that indicate the important role of taking into account the underlying physical principles as well as the emission model.", "questions": "- Table 1 says that NowCastNet doesn't provide uncertainty estimation, but it's a generative model which can provide such estimates and in general approximate the true underlying distribution.\n- In section 3.2, it's unclear why $\\mathbf{\\dot{v}}_k(\\mathbf{x}, t) = \\ddot{u}_k(x,t)$, especially when $\\ddot{u}_k(x,t)$ is not a vector.\n- In section 3.6, how is $\\tilde{\\dot{u}}(t_0)$ numerically apprixmated from past states?\n- Why not use different time-resolutions for solving the ODE and assessing their effect? Same goes for the ODE-solver. Given that you state that Runge-Kutta can be used with a low computational cost, it would add more quality to the paper overall if you include it as well.\n- How long does it take to train?\n- Lacks training details for ClimaX as well as the training runtimes and number of GPUs used for ClimaX.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Proposed method that is based on first principles from physics and provides interpretability of the solution: i.e. explicit transport, compression and flow velocity terms.\n- Provides efficient uncertainty estimation by learning the bias and the standard deviation, unlike previous methods that have to rely on ensembling which is computationally intensive.\n- Simple model, but still manages to outperform ClimaX which would make this method not only a strong baseline for future work but also a method to quickly iterate on and improve.\n- Strong ablation studies demonstrating the design choices made by ClimODE over NODE as well as the effect of the emission model.", "weaknesses": "- \"Closed system assumption implies value-preserving manifold\" is unclear and needs more justification. While the conclusion that: $$\\mathbb{E}_x[u_k(\\mathbf{x},t)]=\\text{const.}$$\n\nsounds intuitive, it's not justified rigourously. The expectation is taken with respect to which density? I would expect the quantity to be preserved would be $\\int_{\\Omega} u_k(\\mathbf{x},t)dV$, since that's the total quantity over the whole \"volume\" you're considering which should be preserved (e.g. when $u_k=\\rho$, the integral over the volume becomes the mass). In any case, since this is a Machine Learning submission, it would be better if this part is thoroughly justified. \n\n- Benchmark only up to 36 hours while state-of-the-art methods reported results for up to 10 days ahead. The paper would be better if it reported results for at least five or seven days ahead.\n- Authors mention that other deep learning methods lacked open-source code, while that's true for PanguWeather (who only provide a pseudo-code \"implementation\"), it's not for the others:\n    - FourCastNet: https://github.com/NVlabs/FourCastNet\n    - GraphCast: https://github.com/google-deepmind/graphcast\n\n Given that GraphCast seem to outperform ClimaX, it would have been good to compare against it as well and also FourCastNet since it's one of the first papers to perform weather forecasting on such a scale.\n- Authors claim that the vaue-preserving manifold (that emanates from the closed system assumption), presents a strong inductive bias for long-term forecasts, yet, using Euler scheme to solve the ODE is known to be unstable and it'll especially not conserve the quantity we want preserved, so that inductive bias is no longer enforced. It would be better to include that limitation in the paper and acknowledge that while it is a strong inductive bias, it's hard to enforce in practice. This is further seen from Table 2 which shows that the error does increase dramatically with lead time and that suggests that ClimODE is better at inferring the true physics but not in mitigating the error propagation in long-term forecasts.\n- While Figure 6 shows qualitatively the soundness of the predicted bias and variance, there's no quantitative approach that evaluates the quality of the bias and variance output by the model. A metric like CRPS (Continuous Ranked Probability Score) can showcase that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5MM810kb5", "reviewer": "Reviewer_MKLC", "summary": "ClimODE aims to improve climate prediction by integrating physical principles into a neural ordinary differential equation (ODE) framework. Unlike traditional black-box models, ClimODE incorporates the concept of advection from statistical mechanics, ensuring that the model considers the spatial movement of weather quantities over time. The model employs neural ODEs to model the weather evolution with value-conserving dynamics. The architecture includes components for local convolutions and global attention, allowing it to capture both local and global weather influences. ClimODE also addresses uncertainty in predictions and source variations through a probabilistic emission model. This feature allows the model to quantify prediction uncertainties and adapt to various source variations. ClimODE is reported to outperform existing data-driven methods in global and regional forecasting.", "questions": "If the several ideas behind ClimODE is effective as suggested by the ablation study, I would really be interested to see its performance in a large model compared to Pangu or FourcastNet, for example. Those benchmarks are exceptional since they excel NWP results in many cases and have been tested/used even for recent disaster predictions. It is just like the LLM that, if it becomes powerful enough, everyone will be amazed and will use it. If the ultimate goal of this paper is to propose a strong SOTA model against existing models, it is strongly recommended to compare with those large models and consider adapting to this line in the future research. \n\nSince the paper mentions efficiency, I wonder how does the model perform with more parameters. Would the performance increase? Or, it is more like a nice small model to save us from excessive computational burden?\n\nAre there any reasons why ClimODE performs worse than ClimaX in t2m? \n\nHow does the model perform with long-term prediction, say, for 7 days or 14 days? The authors mention the emission source model as a strong inductive bias that prevents long-horizon forecast collapses and it seems to work based on the ablation study, so I wonder that.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Physics-Informed Modeling: ClimODE incorporates physical principles, ensuring that the model aligns with our first-principle understanding of the meteorological dynamics. ClimODE follows the partial differential continuity equation and solves a latent equation in such a form, and this makes both the physical insights and model training easier to understand and follow. Because of the use of Neural ODE, the prediction would be continuous in spacetime. I like the cute and nice idea.\n\n2. Model Design: The use of both local convolutions and global attention allows the model to capture a wide range of influences on weather patterns, enhancing its predictive capabilities with both locality inductive bias and long-term interaction capability. The model’s ability to quantify uncertainty and/or model source variation is an additional advantage, making its forecasts more reliable and comprehensive (like a latent force). The spatio-temporal embedding also gives the model extra power to reflect the geographical differences with spatial-temporal variation, which makes a lot of sense.\n\n3. Performance and Efficiency: ClimODE achieves superior performance over some existing deep learning weather forecasting models, with an order of magnitude smaller parameterization, making it computationally efficient. \n\n4. Presentation: Very clear description of initial condition modeling, PDE to ODE modeling, Advection and Flow Velocity modeling, etc.", "weaknesses": "1. Limited Performance and Comparison: The proposed model, although surpasses vanilla Neural ODE and ClimaX, still cannot compete with IFS (ECMWF NWP), let alone bigger models such as FourCastNet and Pangu-Weather that have already reported better results than NWP. There lacks enough comparison to other models in general, such as Adaptive Fourier Neural Operator (practiced by FourcastNet), previous weather forecasting methods like [1, 2], even models for similar tasks such as spatio-temporal traffic forecasting or video forecasting, etc. Comparison to Neural ODE is more of an ablation study and ClimaX is a weak baseline marginally better than ResNet. Comparing only with ClimaX is definitely not convincing enough. \n\n2. Limited Physical Complexity: The partial differential continuity equation is indeed a fundamental concept in fluid dynamics as it describes how quantities such as mass, moisture, or energy are transported and conserved in the atmosphere over time and space. What makes it less predictive in the real world is that this equation comes with assumptions such as the homogeneity and isotropy of the fluid. In cases where these assumptions are violated, the equation might not hold perfectly. I am not certain whether relying too much on this physical equation would be optimal for modeling the complex dynamics of weather, especially in finer-granular resolution with more weather factors and potentially larger noises. \n\n3. Limited Physical Understanding: In all, the partial differential continuity equation is a general-form conservation equation rather than any specific equations explaining each weather factor. Other than that, because neural ODE is essentially still a black-box model due to using neural network, it remains hard to know, for example, the relationships and the interactions between different weather variables. \n\n[1] Yan Han, Lihua Mi, Lian Shen, CS Cai, Yuchen Liu, Kai Li, and Guoji Xu. A short-term wind speed prediction method utilizing novel hybrid deep learning algorithms to correct numerical weather forecasting. Applied Energy, 312:118777, 2022.\n[2] Xiaoying Yang, Shuai Yang, Mou Leong Tan, Hengyang Pan, Hongliang Zhang, Guoqing Wang, Ruimin He, and Zimeng Wang. Correcting the bias of daily satellite precipitation estimates in tropical regions using deep neural network. Journal of Hydrology, 608:127656, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)