CLIFFORD NEURAL OPERATORS ON ATMOSPHERIC DATA INFLUENCED PARTIAL DIFFERENTIAL EQUATIONS

Authors: sujit  roy
wei ji leong
rajat  shinde
christopher e phillips
ankur  kumar
manil  maskey
rahul  ramachandran
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-08 18:21:24

Abstract: Mathematical representations of the atmosphere are key to forecasting and research tasks across Earth science. Numerically solving the underlying partial differential equations (PDEs) of the atmosphere, however, can be difficult and computationally expensive with numerous trade-offs between computing efficiency and accuracy. Utilizing neural networks to learn approximations of the PDE solutions from the data can help us model complex phenomena more efficiently than traditional numerical schemes. Here, we have applied Clifford algebra-based neural operators for predicting atmospheric variables. Clifford Fourier neural operators are used with two different backbone architectures, ResNet and UNet, on custom data of U10, V10 and surface pressure as well as U500, V500 and Z500. 
Clifford Fourier neural operators, coupled with ResNet and UNet architectures, are applied to a key reanalysis dataset. Model performance is initially strong, but we observe increasing errors, resulting in the model becoming highly unstable.

Main paper: 
 introduction: 
 Mathematical representations and specifically partial differential equations (PDEs) are central to modeling scientific and engineering phenomena, often requiring computationally intensive numerical methods like finite differences for solutions Quarteroni & Valli (2008) . However, the high computational demand, especially in understanding and modelling atmosphere dynamics, has led to a growing interest in data-driven machine learning approachesKarniadakis et al. (2021) . These methods offer promising solutions for efficiently approximating complex PDE systems in various applications, where traditional numerical solvers are often slow and inefficient Li et al. (2020a; 2021) ; Brandstetter et al. (2022) ; Kochkov et al. (2021) ; Raonić et al. (2023) ; Pathak et al. (2022) ; Evans (2022); Lu et al. (2021) . Traditional solvers like finite element and finite difference methods balance resolution against speed. Higher resolution increases accuracy but slows computation, especially for complex PDEs requiring fine discretization. In contrast, data-driven methods learn from data, potentially speeding up the process significantly Raissi et al. (2019) ; Kochkov et al. (2021) ; Li et al. (2020a) . However, standard neural networks, limited to finite-dimensional spaces, are bound to specific discretizations, a drawback for practical use. Thus, the development of mesh-invariant neural networks is essential, as seen in approaches like finite-dimensional operators. To overcome the inherent limitations of traditional neural networks, there has been the development of an innovative deep-learning framework called neural operators. This is fundamentally designed to create direct mappings between function spaces in bounded domains Li et al. (2020a) ; Raonić et al. (2023); Sergeant-Perthuis et al. (2022) ; Li et al. (2020b) . In this manuscript, we detail the development of a Clifford algebra-based neural operator for the solving of the PDEs that underlie atmospheric dynamics. 
datasets: 
 The Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2) was used for training the new operator (Gelaro et al., 2017) . Six variables were selected: U10 (zonal 10 m wind), V10 (meridional 10 m wind), SLP (surface pressure adjusted to sea-level), U500 (500 hPa zonal wind), V500 (500 hPa meridional wind), and Z500 (geopotential height of the 500 hPa level). We selected the data with 3 hours of temporal resolution. For training, we used data from 2021 and for testing, we used 2022. The adjusted pressure was selected to mitigate the influence of terrain on the pressure field. Higher elevation areas naturally have lower pressure that mask the effects of low-pressure weather systems on the wind field. Additionally, at their respective levels (near-surface and 500 hPa), horizontal wind and pressure/geopotential height are strongly coupled by a system of PDEs (Lambaerts et al., 2011) . Surface pressure and Z500 both describe the pressure tendencies of the atmosphere, which are crucial to the development and evolution of weather systems, while those systems in turn modify the pressure field (Rostami et al., 2024; Barnes et al., 2022; Jiménez-Esteve & Domeisen, 2022; Spiridonov et al., 2021; Chen et al., 2021; Ning et al., 2021) .The near-surface level was selected due to its immediate relevance to human activities and the 500 hPa layer for its role in large-scale dynamics. The middle atmosphere is less impacted by surface and boundary layer processes such as friction and contains crucial information about vertical transport and steering of synoptic-scale weather systems (Beare, 2007; Jonassen et al., 2020) . Forecasters often analyze patterns at 500 hPa to identify the presence of troughs, ridges, and other features that can influence surface weather conditions, illustrating its importance for weather prediction. While the formulation herein is not a complete description of the atmosphere due to a lack of thermodynamic components such as heat and moisture, the selected variables do provide an otherwise complete depiction of key atmospheric processes. Thus, they serve as a useful test bed for developing weather prediction models, with similar being used to test new NWP formulations (Galewsky et al., 2016; Côté & Staniforth, 1990) . 
methods: 
 When constructing the neural operator framework, we utilized the concept of Clifford algebra over R, the space of all real numbers. The dataset used thus has two vector components, i.e. U 10 &U 500 and V 10 &V 500 and one scalar component i.e. surface pressure (SP) and Z 500 . Two designs are tested, one using U 10 , V 10 , and SLP and the other U 500 , V 500 , and Z 500 , In the first case, we considered two basis vectors e 1 and e 2 and considered sp as scalar. In the second case, we provided zero as a scalar because the model did not train properly when Z 500 was used. Further, we performed Clifford Fourier convolution to the dual pair of the function given by f 0 = f 0 (x) + f 12 (x)i 2 and f 1 = f 1 (x) + f 2 (x)i. The Clifford Fourier transform (Cl 0,2 ) for multivector valued functions R 2 → G 2 and vectors x, ξ ∈ R 2 is given by eq 1 Brandstetter et al. (2022) .f (ξ) = F{f }(ξ) = 1 2π R2 f (x)e -2πi2⟨x,ξ⟩ dx, ∀ξ ∈ R 2 (1)where f (x) and f (ξ) represents multivector fields in the spatial and the frequency domain and i 2 = e 1 e 2 . We are going to use the concepts of 4D Clifford algebra (Cl 0,2 ) with vector space G 2 spanned by 1, e 1 , e 2 , e 1 e 2 . Utilizing the same definition we can rewrite eq 1 as 2. 2023), where we used quaternion kernel (g = [-1, -1]). Table 1 , shows the architecture of the Clifford convolution with U-Net as backbone. The model was trained by providing a 1-time step input and 1-time step output. We modified the loss L to be mean squared error along with 10 times the change in mean surface pressure (SP i ), as shown in eq 3.EQUATIONEQUATIONWe used auto-regressive forecasting to create the final evaluation when we provided the model with an initial condition of t = 0 and then the output generated by the model as input again. 
results and discussion: 
 Figure 5 shows the prediction, ground truth and difference between prediction and ground truth using U-Net (a) and ResNet (b) architectures respectively for U10, V10 and SLP dataset. We observed in the case of ResNet backbone the model error became excessive after around 30 timesteps. In U-NET, we observed the error to grow large around 60 timesteps ahead. This is not surprising that the U-NET-based backbone was In figure 5a , there appeared to be two main regions where errors were generated. First, are the mid-to-high latitudes. At timestep 30, regions near the equator had wind speed errors within approximately 3-5 m/s. The exception was near certain regions of activity such as a tropical disturbance in the central Atlantic. At higher latitudes, however, errors approached or surpassed 10 m/s. In these regions, Rosby waves and midlatitude cyclones appeared and were artificially intensified in the model. A potential source of this error was a discrepancy in the height/pressure distribution, the lack of friction near the surface, or the model not properly learning the relationship between pressure distribution and the wind field. In later timesteps, however, the flow in sub-tropical and tropical regions demonstrated the presence of wavelike features that led to large errors as well. In particular, west of the Americas waves appeared to diffract and experience super-positioning. We think it likely that errors in the pressure field in the tropics produced these fictitious waves. This is because the tropics are barotropic with weak thermodynamic gradients, and thus, disturbances quickly produce gravity waves that transmit their energy to their surroundings. If the AI forecasts a fictitious region of high or low pressure, that feature would produce these waves, which in turn would add energy to the model atmosphere. Consequently, this results in an exponential increase in errors in the tropics. In future, we would like to add more variables for the complexity. Additionally, we would use more training samples and do a longer autoregressive loss.  
 
 Appendix:  


Decision: Both reviewers unanimously agree on acceptance. However, I strongly command the author to address those concerns in the camera ready version.

Reviews: review: title: title: This paper explores incorporating Clifford neural operators for Earth forecasting task. The proposed method and its application are novel. However, the empirical studies are not enough to demonstrate its advantages.
main_review: main_review: This paper explores incorporating Clifford neural operators into two types of backbones ResNet and UNet, for Earth forecasting task on MERRA-2 dataset.

Pros
1. The visualization on model design is clear and easy to follow.
2. The visualization on performance comparison helps indentify the areas where the model fail to predict accurately and help analyze the potential causes.

Weaknesses
1. Some expressions are not clear enough. E.g., there lacks descriptions on the notations in Eq(1)(2).
2. The performance drops significantly when the forecasting horizon gets longer. Besides the potential causes discussed in Section 4, the single-input-single-output formulation is also a possible cause that leads to distribution mismatch and accumulated errors.
3. The major contribution is the usage of Clifford neural operator and its application to Earth forecasting. However, there is no evidence that it really results in performance gain. Ablation studies on Clifford neural operator, e.g., comparisons to the original ResNet and UNet, could be helpful.
title: title: CLIFFORD NEURAL OPERATORS ON ATMOSPHERIC DATA INFLUENCED PARTIAL DIFFERENTIAL EQUATIONS
main_review: main_review: This work presents an innovative approach to atmospheric modeling by leveraging Clifford algebra-based neural operators within ResNet and UNet architectures. The application of these methods to predict atmospheric variables, including U10, V10, surface pressure, U500, V500, and Z500, from a key reanalysis dataset, is a significant contribution to the field of Earth science. The use of neural networks to approximate the solutions of partial differential equations (PDEs) is a promising direction, potentially offering a more computationally efficient alternative to traditional numerical schemes.
However, the issue of increasing errors and resultant model instability author had observed warrants careful examination. This instability could be attributed to several factors inherent in the modeling approach or data handling processes.

In conclusion, this study represents a meaningful step forward in the application of advanced neural network architectures for atmospheric modeling. Addressing the observed instability requires a multifaceted approach, examining data preprocessing, model architecture, regularization, and validation strategies. Further research in this direction could significantly enhance our capability to model complex atmospheric phenomena efficiently and accurately.