Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes Dynamics

Authors: matthias  karlbauer
danielle c maddix
abdul fatir ansari
boran  han
gaurav  gupta
yuyang  wang
andrew  stuart
michael w mahoney
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-07 22:57:02

Abstract: There has been remarkable progress in the development of Deep Learning Weather Prediction (DLWP) models, so much so that they are poised to become competitive with traditional numerical weather prediction (NWP) models. Indeed, a wide number of DLWP architectures---based on various backbones, including U-Net, Transformer, Graph Neural Network (GNN), or Fourier Neural Operator (FNO)---have demonstrated their potential at forecasting atmospheric states. However, due to differences in training protocols, data choices (resolution, selected prognostic variables, or additional forcing inputs), and forecast horizons, it still remains unclear which of these methods and architectures is most suitable for weather forecasting. Here, we back up and provide a detailed empirical analysis, under controlled conditions, comparing and contrasting the most prominent backbones used in DLWP models. This is done by predicting two-dimensional incompressible Navier-Stokes dynamics with different numbers of parameters and Reynolds number values. In terms of accuracy, memory consumption, and runtime, our results illustrate various tradeoffs, and they show favorable performance of FNO, in comparison with Transformer, U-Net, and GNN backbones.

Main paper: 
 introduction: 
 Deep Learning Weather Prediction (DLWP) models have recently evolved to form a promising and competitive alternative to numerical weather prediction (NWP) models (Bauer et al., 2015) . Early attempts (Scher & Messori, 2018; Weyn et al., 2019) designed U-Net models (Ronneberger et al., 2015) on a cylinder mesh, learning to predict air pressure and temperature dynamics on a coarse global resolution of 5.625 • . More recently, Pathak et al. (2022) designed FourCastNet on the basis of the Fourier Neural Operator (FNO) (Li et al., 2020b) , moving to the native resolution of 0.25 • of the ERA5 reanalysis dataset (Hersbach et al., 2020) , which amounts to 721 × 1440 data points covering the globe. The same dataset finds application in the Vision Transformer (ViT) (Dosovitskiy et al., 2020) based Pangu-Weather model (Bi et al., 2023) and the state-of-the-art message-passing Graph Neural Network (GNN) (Battaglia et al., 2018; Fortunato et al., 2022) based GraphCast model (Lam et al., 2022) .In a comparison of state-of-the-art DLWP models, Rasp et al. (2023) find that GraphCast generates the most accurate weather forecasts on lead-times up to ten days. GraphCast was trained on 221 variables from ERA5 (substantially more than Pangu-Weather and FourCastNet, which were trained on 67 and 24 prognostic variables, respectively), but the root of GraphCast's improved performance remains entangled in details of the architecture type, choice of prognostic variables, and training protocol. Here, we seek to elucidate the effect of the model architecture, i.e., GNN, Transformer, U-Net, or FNO, and design an empirical evaluation that thoroughly compares DLWP backbone architectures under controlled conditions while freezing data choice and training protocol. To this end, we train and evaluate various architectures on two-dimensional Navier-Stokes simulations, while controlling the number of parameters to generate cost-performance tradeoff curves. We motivate our data choice of using the Navier-Stokes equations since they find application in NWP where they are used to describe parts of the atmosphere's dynamics.With this analysis, we seek to motivate a model that has the greatest potential in addressing downsides of current DLWP models, e.g., stability of long roll-outs for climate lengthscales, uncertainty quantification, and physically meaningful predictions. Our aim is to help the community find and agree on a suitable backbone for DLWP models. 
related work and methods: 
 We compare six model classes that form the basis for state-of-the-art DLWP models. We provide a brief overview of these methods in the following. (See Appendix A.1 for details and Table 2 in this appendix for how we modify them to vary the number of parameters.) As a naïve baseline and upper bound for our error comparison, we implement Persistence, which predicts the last observed state. Starting with early deep learning (DL) methods, we include convolutional long short-term memory (ConvLSTM) (Shi et al., 2015) , which combines spatial and temporal information processing by replacing the scalar computations of LSTM gates (Hochreiter & Schmidhuber, 1997) with convolution operations. ConvLSTM is one of the first DL models for precipitation nowcasting and other spatiotemporal forecasting tasks, and it finds applications in Google's MetNet1 and Met-Net2 (Sønderby et al., 2020; Espeholt et al., 2022) . Among early DL methods, we also benchmark U-Net, which is one of the most prominent and versatile DL architectures, originally designed for biomedical image segmentation (Ronneberger et al., 2015) and forms the backbone of many DLWP models (Weyn et al., 2019; 2020; 2021; Karlbauer et al., 2023; Lopez-Gomez et al., 2023) .We also include two more recent architecture backbones based on Transformer and GNN, which power current state-of-the-art DLWP models. The Transformer architecture (Vaswani et al., 2017) has found success with image processing (Dosovitskiy et al., 2020) and has been applied to weather forecasting, by viewing the atmospheric state as a sequence of three-dimensional images (Gao et al., 2022) . Pangu-Weather (Bi et al., 2023, by Huawei) and FuXi (Chen et al., 2023) , use the SwinTransformer backbone (Liu et al., 2021) , whereas Microsoft refines the Transformer into ClimaX, a model for weather and climate related downstream tasks (Nguyen et al., 2023) . Multi-Scale MeshGraphNet (MS MeshGraphNet) (Fortunato et al., 2022) extends MeshGraphNet (Pfaff et al., 2020 )-a message-passing GNN processing unstructured meshes-to operate on multiple grids with different resolutions. MS MeshGraphNet forms the basis of GraphCast (Lam et al., 2022) using a hierarchy of icosahedral meshes on the sphere.Lastly, we benchmark architectures based on FNO (Li et al., 2020b) . FNO is a type of operator learning method (Li et al., 2020a; Lu et al., 2021; Gupta et al., 2021) that learns a function-to-function mapping by combining pointwise operations in physical space and in the wavenumber/frequency domain. In contrast to the aforementioned architectures, it is a discretization invariant operator method. While FNO can be applied to higher resolutions than it was trained on, it may not be able to predict processes that unfold on smaller scales than observed during training (Krishnapriyan et al., 2023) . These uncaptured small-scale processes can be important in turbulence modeling. We implement both a two-and a three-dimensional variant of FNO, as specified in Appendix A.1. We also experiment with TFNO, using a Tucker-based tensor decomposition (Tucker, 1966; Kolda & Bader, 2009) , which is more parameter efficient. FNO serves as the basis for LBL's and NVIDIA's FourCastNet series (Pathak et al., 2022; Bonev et al., 2023; Kurth et al., 2023) , which we also include in our comparison. 1Table 1: RMSE scores for experiment 1, reported for each model under different number of parameters. Errors reported in italic correspond to models that had to be retrained with gradient clipping (by norm) due to stability issues. With OOM and sat, we denote models that ran out of GPU memory and saturated, respectively. Saturated means that we did not further increase the parameters because the performance already saturated over smaller parameter ranges. Best results shown in bold.  
experiments and results: 
 We conduct three series of experiments to explore the ability of the architectures (see Section 2) to predict the two-dimensional incompressible Navier-Stokes dynamics in a periodic domain. Our data is discretized on a two-dimensional 64 × 64 grid, and we design the experiments to test two levels of difficulties by generating less and more turbulent data, with Reynolds Numbers Re = 1 × 10 3 (experiment 1) and Re = 1 × 10 4 (experiments 2 and 3), respectively. For experiments 1 and 2, we generate 1 k samples, while experiment 3 repeats experiment 2 with an increased number, 10 k, of samples. Our experiments are designed to test: (1) easier vs. harder problems, with the modification in Re, and (2) the effect of the data size. For comparability, the initial condition and forcing of the data generation process are chosen to be identical with those in Li et al. (2020b) ; Gupta et al. (2021) .(See Appendix A.2.) Also, following Li et al. (2020b) , the models receive a context history of h = 10 input frames, on basis of which they autoregressively generate the remaining 40 (experiment 1) or 20 (experiments 2 and 3) frames.2 Concretely, we apply a rolling window when generating autoregressive forecasts, by feeding the most recent h frames as input and predicting the next single frame, i.e., ŷt+1 = φ θ (x t-h,...,t ), where ŷt+1 denotes the prediction of the next frame generated by model φ with trainable parameters θ, and x t-h,...,t denotes the most recent h frames provided as input concatenated along the channel dimension. The three-dimensional (T)FNO models make an exception to the autoregressive rolling window approach, by receiving the first h frames x 0:h as input to directly generate a prediction ŷh+1:T of the entire remaining sequence in a single step. See Appendix A.3 for our training protocol featuring hyperparameters, learning rate scheduling, and weight updates.Experiment 1: Reynolds Number Re = 1 × 10 3 , 1 k samples. In this experiment, we generate less turbulent dynamics with Reynolds Number Re = 1×10 3 and a sequence length of T = 50. The quantitative root mean squared error (RMSE) metric, reported in Table 1 and Figure 1 (left) shows that TFNO2D performs best, followed by TFNO3D, SwinTransformer, FNO3D, ConvLSTM, U-Net, FourCastNet, and MS MeshGraphNet. (See the qualitative results in Figure 2 in Appendix B.1 with the same findings.) All models outperform the naïve Persistence baseline, which always predicts the last observed state, i.e., ŷt = x h . This principally indicates a successful training of all models. We observe substantial differences in the error saturation when increasing the number of parameters, which supports the ordering of architectures seen in Figure 2 . Concretely, with an error of 1 × 10 -2 , MS MeshGraphNet does not reach the accuracy level of the other models. Beyond 500 k parameters, the model hits the memory constraint and also does not con- 4 (right) in Appendix B.1). Similarly, SwinTransformer generates comparably accurate predictions, reaching an error of 7 × 10 -3 , before quickly running out of memory when going beyond 2 M parameters. U-Net and FourCastNet express a similar behavior, saturating at the 1 M parameter configuration and reaching error levels of 1.2 × 10 -2 and 1.5 × 10 -2 , respectively. In FNO3D and TFNO3D, we observe a two-staged saturation, where the models first converge to a poor error regime of 1 × 10 -1 , albeit approaching a remarkably smaller RMSE of 9 × 10 -3 and 6 × 10 -3 , respectively, when increasing the number of layers from 1 at #params ≤ 2 M to 2, 4, 8, and 16 to obtain the respective larger parameter counts. 4 Instead, when fixing the numbers of layers at l = 4 and varying the number of channels in TFNO3D L4, we observe better performance compared to the single-layer TFNO3D L1-16 in the low-parameter regime (until 2 M parameters), albeit not competitive with other models. To additionally explore the effect of the number of layers vs. channels in TFNO3D, we vary the number of parameters either by increasing the layers over l ∈ [1, 2, 4, 8, 16], while fixing the number of channels at c = 32 in TFNO3D L1-16, or by increasing the number of channels over c ∈ [2, 8, 11, 16, 22, 32] while fixing the number of layers at l = 4 in TFNO3D 4L. Consistent with Li et al. (2020b) , we observe the performance saturating at four layers. Finally, the autoregressive TFNO2D performs remarkably well across all parameter ranges-converging at an unparalleled RMSE score of 4 × 10 -3 -while, at the same time, constituting a reasonable tradeoff between memory consumption and runtime complexity (see Figure 4 in Appendix B.1). From this we conclude that, at least for periodic fluid flow simulation, FNO2D marks a promising choice, suggesting its application to real-world weather forecasting scenarios.Experiment 2: Reynolds Number Re = 1 × 10 4 , 1 k samples. In this experiment, to stress test the consistency of the model order found in experiment 1, we generate more turbulent data by increasing the Reynolds Number Re by an order of magnitude and reducing the simulation time and sequence length to T = 30 timesteps. To reduce our search over various architectures, we focus here on stable models only by removing those that depend on gradient clipping. We make the same observations as in experiment 1, confirming TFNO2D as the most accurate model, followed by SwinTransformer, TFNO3D, and U-Net on this harder task. See Figure 1 (right) for quantitative results and Figure 6 in Appendix B.2 for qualitative results.Experiment 3: Reynolds Number Re = 1 × 10 4 , 10 k samples. Since in experiment 2, the three-dimensional TFNO models with #params ≥ 8 M start to show a slight tendency to overfit (not shown), we repeat the experiment, increasing the number of training samples by an order of magnitude to 10 k. Figure 5 , Figure 7 and Table 4 in Appendix B.2 show that the same findings hold on experiment 3, which has a larger number of 10 k samples. 
discussion: 
 In this work, we obtain insights into which DLWP models are more suitable for weather forecasting by devising controlled experiments, which are useful more generally, to compare these models. In particular, we fix the input data and training protocol and vary the architecture and number of parameters. In this limited setup on the synthetic two-dimensional periodic incompressible Navier-Stokes dataset, we find that TFNO2D performs the best at predicting the dynamics, followed by TFNO3D, SwinTransformer, FNO3D, ConvLSTM, U-Net, FourCastNet, and MS MeshGraphNet. Importantly, throughout our experiments, we see that all benchmarked models eventually saturate at an individual error regime, indicating that scaling laws could not be determined for the models and task at hand. We thus identify a value for further research in developing an architecture that scales with increased parameters without converging at a specific error level. The spherical Fourier Neural Operator SFNO (Bonev et al., 2023 ) marks a reasonable starting point for further exploration, ranging from increasing the number of prognostic variables, to providing more context to the models by means of forcing fields, to incorporating physical knowledge (Li et al., 2022; Hansen et al., 2023) and moving beyond deterministic forecasts (Cachay et al., 2023; Gao et al., 2023; Price et al., 2023) .A limitation of our work is the sole consideration of synthetic data. We seek to focus our future endeavors on repeating the analysis presented here on WeatherBench (Rasp et al., 2020) to verify our claims and suggestions on real-world weather data that is represented on the sphere. In addition, although we enable circular padding in the compared architectures, the periodic nature of the data we use may favor the inductive bias of FNO. The periodicity exhibited in our data matches that of weather dynamics on global scale. Throughout our experiments, we perform only single model runs. When experimenting with different seeds, we do not encounter severe variations, which is also supported by the consistent RMSE vs. parameter plots. We note the poor results for MS MeshGraphNet, which we attribute to the underlying graph and to our training protocol. Seeing that GraphCast has been trained very carefully on 221 prognostic variables, its lead over other DLWP may be dominated by the number of variables and by the training protocol, and less to the actual architecture. This hypothesis, however, needs further investigations in future work such as on WeatherBench.perform ten steps of teacher forcing before switching to closed loop to autoregressively unroll a prediction into the future.We implement a five-layer encoder-decoder architecture with avgpool and transposed convolution operations for down and up-sampling, respectively. On each layer, we employ two consecutive convolutions with ReLU activations (Fukushima, 1975) and apply the same parameters described above in the encoder for ConvLSTM. See Table 2 for the numbers of channels hyperparameter setting.SwinTransformer Enabling circular padding and setting patch size p = 2, we benchmark the shifted window transformer (Liu et al., 2021) by varying the number of channels, heads, layers, and blocks, as detailed in Table 2 , while keeping remaining parameters at their defaults.We formulate a periodically connected graph to apply Multi-Scale Mesh-GraphNet (MS MeshGraphNet) with two stages, featuring 1-hop and 2-hop neighborhoods, and follow Fortunato et al. ( 2022) by encoding the distance and angle to neighbors in the edges. We employ four processor and two node/edge encoding and decoding layers and set hidden dim = 32 for processor, node encoder, and edge encoder, unless overridden (see Table 2 ).FNO We compare three variants of FNO: Two three-dimensional formulations, which process the temporal and both spatial dimensions simultaneously to generate a three-dimensional output of shape [T, H, W ] in one call, and a two-dimensional version, which only operates on the spatial dimensions of the input and autoregressively unrolls a prediction into the future. While fixing the lifting and projection channels at 256, we vary the number of Fourier modes, channel depth, and number of layers according to Table 2 .FourCastNet We choose a patch size of p = 4, fix num blocks = 4, enable periodic padding in both spatial dimensions, and keep the remaining parameters at their default values while varying the number of layers and channels as specified in Table 2 .We provide additional information about the data generation process in Table 3 , which we keep as close as possible to that reported in Li et al. (2020b) and Gupta et al. (2021) .In the experiments, we use the Adam optimizer with learning rate η = 1 × 10 -3 (except for MS MeshGraphNet, which only converged with a smaller learning rate of η = 1 × 10 -4 ) and cosine learning rate scheduling to train all models with a batch size of B = 4, effectively realizing 125 k weight update steps, relating to 500 and 50 epochs, respectively, for 1 k and 10 k samples. 5 For the training objective and loss function, we choose the mean squared error (MSE) between the model outputs and respective ground truth frames, that is L = MSE(ŷ h+1:T , y h+1:T ). Note that, to stabilize training, we have to employ gradient clipping (by norm) for selected models, indicated by italic numbers in tables and triangle markers in figures.In this section, we provide additional empirical results for the three experiments.B.1 RESULTS FROM EXPERIMENT 1: REYNOLDS NUMBER Re = 1 × 10 3Figure 2 illustrates the initial and end conditions along with the respective predictions of all models. Qualitatively, we find there exist parameter settings for all models to successfully unroll a plausible prediction of the Navier-Stokes dynamics over 40 frames into the future, as showcased by the last predicted frame, i.e., ŷt=T (see the third and fifth row of Figure 2 ). When computing the difference between the prediction and ground truth, i.e., d = ŷ -y, we observe clear variations in the accuracy of the model outputs, denoted by the saturation of the difference plots in the second and fourth row of Figure 2 . Interestingly, this difference plot also reveals artifacts in the outputs of selected models: SwinTransformer and FourCastNet generate undesired patterns that resemble their windowing and patching mechanisms, whereas the 2-hop neighborhood, which was chosen as the resolution of the coarser grid, is baked into the output of MS MeshGraphNet. According to the (Pfaff et al., 2020) .Next, we study the effect of the underlying graph in GNNs. Observing the poor behavior of MS MeshGraphNet in Figure 2 , we investigate the effect of three different periodic graph designs to represent the neighborhoods in the GNN. First, the 4-stencil graph connects each node's perpendicular four direct neighbors (i.e., north, east, south, and west) in a standard square Cartesian mesh. Second, the 8-stencil graph adds the direct diagonal neighbors to the 4-stencil graph. Third, the Delaunay graph connects all nodes in the graph by means of triangles, resulting in a hybrid of the 4-stencil and 8-stencil graph, where only some diagonal edges are added. To simplify the problem, we conduct this analysis on the single-scale MeshGraphNet (Pfaff et al., 2020) instead of using the hierarchical MS MeshGraphNet (Fortunato et al., 2022) Interestingly, as indicated in Figure 3 , the results favor the Delaunay graph over the 8-and 4-stencil graphs, respectively. Apparently, the increased connectedness is beneficial for the task. At the same time, though, the irregularity introduced by the Delaunay triangulation potentially forces the model to develop more informative codes for the edges to represent direction and distance of neighbors more meaningfully.Lastly, Figure 4 compares the RMSE, memory consumption and computational cost in seconds per epoch as a function of the number of parameters. We see that TFNO2D L4 performs the best in terms of the RMSE and also scales well with respect to memory and runtime. Table 4 shows the quantitative error scores of all the experiments (for an easier comparability). We see that the same trend occurs across all three experiments with TFNO2D performing the best. Figure 5 illustrates the similar trends of these RMSE results from experiments 2 and 3. Figure 6 and Figure 7 provide the qualitative visualizations for experiments 2 and 3, respectively. Figure 5 (right) and Figure 7 for experiment 3 show that, while all models consistently improve their scores due to the larger training set, the results from experiments 1-2 still hold. That is, when comparing the convergence levels in Figure 5 (right) and Table 4 , we see that all models saturate at lower error regimes, while the ordering of the model performance from experiment 1 remains unchanged. Also qualitatively, the models benefit from the increase of training samples in experiment 3, since, visually, the yellow vortex at this higher Reynolds Number is resolved more accurately when the models are trained on more data, as illustrated in Figure 6 and Figure 7 .Due to the rectangular nature of our data, we consider the original FourCastNet implementation based onGuibas et al. (2021) instead of the newer Spherical Fourier Neural Operator (SFNO)(Bonev et al.,2023), which works with spherical data and is promising for weather prediction on the sphere.Larger Reynolds Numbers lead to more turbulent dynamics that are harder to predict. Thus,Li et al. (2020b) selects T = 50 and T =for Re = 1e3 and Re = 1e4, respectively. We follow this convention.Experiments are performed on an AWS g5.12xlarge instance featuring four NVIDIA A10G GPUs with 23 GB RAM each. We used single GPU training throughout all our experiments.We observe a similar behavior (not shown) when experimenting with the number of blocks vs. layers in SwinTransformer, suggesting to prioritise more layers per block over more blocks with less layers.With an exception for MS MeshGraphNet, which only supports a batch size of B = 1, resulting in 500 k weight update steps. 
 
 Appendix:  


Decision: Thanks to reviewers for their careful and detailed comments. I also agree that this work could be considered as one preliminary step towards evaluating DLWP models. I agree though that using simplified benchmarks authors should not make strong conclusion without having enough data points and for that I encourage them to revisit and revise their conclusions appropriately based on the points addressed by reviewers. Also, reporting and or at least adding a short discussion on runtime and memory consumption of different methods is a great point by reviewer K6Hb, as it plays a role in the applicability of methods for real weather prediction problem. Authors are expected to address all the comments by reviewers in their final version and under that condition, I vote accept for the publication of this study as poster.

Reviews: review: title: title: Review of “Comparing And Contrasting Deep Learning Weather Prediction Backbones On Navier-Stokes Dynamics”
main_review: main_review: This work compares a range of state-of the art machine learning methods used for the emulation of weather prediction systems. This is a rapidly growing field with several different groups producing highly skilled models, all built using different core architectures. This paper’s novelty is directly comparing the skill of each architecture on a single system using similar training setups for each model to reduce the factors contributing to skill. The paper looks at three configurations of the system and training data to compare results. The topic of the paper is interesting as a comparison like this has yet to be performed on data from a climate system.

However, several flaws with the idealized testbed and training/testing methodology bring into question the usefulness of the paper’s results. 

Major Concerns
1)	The two cases of homogenous isotropic forced 2D turbulence selected for the paper lack the multi-scale complexity present in true atmospheric flows. The flows selected are borderline turbulent, in addition to lacking other important characteristics such as rotation or mean flows. 
a.	In experiment 1, with Reynold number of 1000, the flow looks to be close to laminar, with a single pair of vortices dominating the domain and no visible small scales. Given that the initial condition for the simulation appears to be from the spin up of a simulation, it is also possible the run has not had sufficient time to develop turbulence.  
b.	In experiment 2, the flow does appear to be turbulent, however it still is dominated by a small number of large vortices.
2)	Choices the authors make while training the model may impact each architecture differently and differ to the implementations of the corresponding methods by the groups who have shown success with them.
a.	An example, providing the model with the previous 10 time-steps of history is not a common practice in weather predictions and may have both positive or negative effects on model prediction. These effects also may be a function of architecture. 
3)	Ultimately the paper draws conclusions outside of the scope of the work done, investing more time into understanding the key features of climate/weather emulation and choosing a more suitable testbed (e.g. higher Reynolds numbers, including rotation, or even dynamics across multiple depths/layers as in quasi geostrophic models) would make this an important contribution to the field.
Minor Concerns	
1)	The persistence RMSE is computed against a snapshot of undeveloped flow, this does not give a good baseline for the performance of the models as the system has not yet reached a steady state. In fully developed flow (which is the case for the atmosphere), the large vortices would persist longer and likely would give the system a longer decorrelation timescale and meaningfully change the values of RMSE. 
a.	The use of persistence as an initial condition is not the most representative of problems in the atmosphere as learning how flow will develop is not the same as how it will evolve.
2)	Be careful with language in the paper, some statements are not correct.
a.	“the atmosphere is modeled by means of incompressible Navier-Stokes equations” – the atmosphere is not considered to be an incompressible flow, typically it is assumed to be an ideal gas in the dynamical core of climate models. Furthermore, the actual atmosphere reflects dynamics including moisture and radiation, which are not components of the Navier-Stokes equations.
title: title: The Fourier Neural Operator shows favorable performance in predicting atmospheric states under controlled conditions compared to other Deep Learning Weather Prediction models.
main_review: main_review: The work discusses the recent advancements in Deep Learning Weather Prediction (DLWP) models, highlighting their potential to rival traditional numerical weather prediction (NWP) models. Various DLWP architectures, including U-Net, Transformer, Graph Neural Network (GNN), and Fourier Neural Operator (FNO), have shown promise in forecasting atmospheric states. However, due to differences in training protocols, data choices, and forecast horizons, it remains unclear which method and architecture are most suitable for weather forecasting. The study conducts a detailed empirical analysis comparing these architectures under controlled conditions, focusing on predicting two-dimensional incompressible Navier-Stokes dynamics. Results indicate favorable performance of the Fourier Neural Operator (FNO) backbone in terms of accuracy, memory consumption, and runtime when compared to Transformer, U-Net, and GNN backbones.
 
In general, the present study is of interest for the DL-based weather forecasting community, but also in particular for benchmarking for the machine learning community. The aim of the paper is clearly defined in the manuscript. The text is technically well-written and follows a clear structure and logic. The theoretical concepts are concisely presented in the main text, with further details described in the appendices. The figures and the experiments are in general sound, and the results support the conclusions. While these conclusions are based on synthetic data as a first step, and it is clear limitation of this work, it is fair to point to future work to confirm or challenge these results based on real-world weather data sets.
title: title: Benchmarking Deep Learning Weather Prediction Models
main_review: main_review: Summary:
The authors set out to benchmark various Deep Learning Weather Prediction (DLWP) models on a two-dimensional Navier Stokes problem, which they claim is a suitable problem for evaluating models of this kind. 

Strengths and Weaknesses:

[+] The paper is well-motivated and clearly presented. It provides a clear description of the problem and the proposed study.

[+] The authors benchmark a wide variety of DLWP models, including not only different models but also different variations of models such as two- and three-dimensional TFNOs. The inclusion of a baseline (Persistence) and a multiple of parameters for each is very comprehensive. 

[-] The claim that the chosen test problem is suitable for identifying a backbone for DLWP is let down by the author's own comments. Take, for example, footnote 1: "Due to the rectangular nature of our data, we consider the original FourCastNet implementation based on Guibas et al. (2021) instead of the newer Spherical Fourier Neural Operator (SFNO) ". For suitable benchmarking, it does not seem this is a sufficient problem; it would be better to go to 3D and on global scales (e.g., Figures 2, 6, & 7 in the appendix show small-scale features unlike the problems solved in something like the original FourCastNet paper) in addition to including a wider variety of problems. While this is an okay first step, drawing conclusions from such a small problem set does not seem prudent. 

[+] The supplementary material is beneficial and includes many important and helpful details. However, the call for papers states: "Reviewers are not obliged to read supplementary materials when reviewing the paper." Some information in the supplementary material is necessary to corroborate the main text claims (see next comment). 

[-] It would be nice to see training and inference times in addition to accuracy. The authors make the statement: "In terms of accuracy, memory consumption, and runtime, our results illustrate various tradeoffs, and they show favorable performance of FNO, in
comparison with Transformer, U-Net, and GNN backbones." in the abstract. However, I do not see any reporting of memory consumption or runtime in the main text, and inference time comparisons are also vital for real-time predictions. The authors do state when certain models run out of memory, but that is not comparative; it is only a binary statement. 

Conclusion: In terms of models, the study is comprehensive; however, in terms of problems, it is not. Given this is a workshop paper, that is to be expected in only 4 pages, however, the conclusions derived from such a limited study have to be taken with a grain of salt. I think benchmarking lends itself better to a longer format venue, so comparisons and conclusions can be fair. Still, there are insights that can be made here, such as studying memory and saturation limitations on an individual model level; therefore, I am recommending a weak acceptance and would like to see this expanded upon in a full-length article.