Operator Learning with Neural Fields: Tackling PDEs on General Geometries
Authors: Louis Serrano
Lise Le Boudec
Armand Kassaï Koupaï
Thomas X Wang
Yuan Yin
Jean-Noël Vittaut
patrick gallinari
Venue: NeurIPS 2023 poster (2023)
Keywords: PDEs
Physics
Operator Learning
Deep Learning
Spatiotemporal
TL;DR: We propose a new method for Operator Learning that can handle general geometry with free-form data.

Abstract: Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neural networks are constrained to discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely on some form of discretization. In order to alleviate such limitations, we present CORAL, a new method that leverages coordinate-based networks for solving PDEs on general geometries. CORAL is designed to remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends to diverse problem domains, including PDE solving, spatio-temporal forecasting, and inverse problems like geometric design. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-art models.

Operator Learning with Neural Fields: Tackling PDEs on General Geometries
Louis Serrano', Lise Le Boudec" Armand Kassaï Koupai" Thomas Wang', Yuan Yin', Jean-Noël Vittaut?, Patrick Gallinari'.
Sorbonne Université, CNRS, ISIR, 75005 Paris, France
2 Sorbonne Université, CNRS, LIP6, 75005 Paris, France
Criteo AI Lab, Paris, France
lise. www.udebooo.ooooo rmand, kassai, thomas www.tt.wwagg.aaaee.oo...o jean-noel vittaut, taatttuut,aticckkk ga aall1 narr w
Abstract
Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neu- ral networks are constrained discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely form of discretization. In order alleviate such limitations, we present CORAL, a new method that leverages coordinate-basee networks for solving PDEs on general geometries. CORAL is designed remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends diverse problem domains, including PDE solving, spatio-temporal forecasting, and geometry aware inference. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-au models.
Introduction
Modeling physics dynamics entails learning mappings between function spaces, a crucial step in formulating and solving partial differential equations (PDEs). the classical approach, PDEs are derived from first principles, and differential operators are utilized to map vector fields across the variables involved in the problem. To solve these equations, numerical methods like finite elements, finite volumes, or spectral techniques are employed, requiring the discretization of spatial and temporal components the differential operators (Morton & Mayers, 2005; Olver, 2014).
Building on successes in computer vision and natural language processing (Krizhevsky et al., 2017; He al., 2016; Dosovitskiy et al., 2021; Vaswani al., 2017). deep learning models have recently gained attention in physical modeling. They have been applied to various scenarios, such as solving PDEs (Cai et al., 2021), forecasting spatio-temporal dynamics (de Bézenac al., 2019), and addressing inverse problems (Allen et al., 2022). Initially, neural network architectures with spatial inductive biases like ConvNets (Long et al., 2018; Ibrahim al., 2022) for regular grids GNNs (Pfaff et al., 2021; Brandstetter et al., 2022b) for irregular meshes were explored. However, these models are limited to specific mesh points and face challenges generalizing to new topologies. The recent trend of neural operators addresses these limitations by modeling mappings between functions, which can seen as infinite-dimensional vectors. Popular models like DeepONet (Lu et al., 2022) and Fourier Neural Operators (FNO) (Li et al., 2022b) have been applied in various domains. However, they still have design rigidity, relying on fixed grids during training and inference, which limits their use in real-world applications involving irregular sampling grids
geometries. A variant of
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

FNO tailored for more general geometries is presented in (Li et al., 2022a), but it focuses on design tasks.
To overcome these limitations, there is a need for flexible approaches that can handle diverse geometries, metric spaces, irregular sampling grids, and sparse measurements. We introduce CORAL, a COordinate- model for opeRAtor Learning that addresses these challenges by leveraging implicit neural representations (INR). CORAL encodes functions into compact, low-dimensional latent spaces and infers mappings between function representations in the latent space. Unlike competing models that are often specific,, CORAL is highly flexible and applicable to various problem domains. We showcase its versatility in PDE solving, spatio-temporal dynamics forecasting, and design problems.
Our contributions are summarized as follows:
CORAL can learn mappings between functions sampled on an irregular mesh and maintains consistent performance when applied to new grids not seen during training. This characteristic makes it well-suited for solving problems in domains characterized by complex geometries or non-uniform grids.
We highlight the versatility of CORAL by applying it to a range of representative physical modeling tasks, such initial value problems (IVP), geometry-aware inference, dynamics modeling, and forecasting. Through extensive experiments on diverse datasets, we consistently demonstrate its state-of-the-art performance across various geometries, including convex and non-convex domains, as well as planar and spherical surfaces. This distinguishes CORAL from alternative models that are often confined to specific tasks.
CORAL is fast, Functions are represented using compact latent code in CORAL, capturing the essential information necessary for different inference tasks in condensed format. This enables fast inference within the compact representation space, whereas alternative methods often operate directly within a dimhes--imeeniinall representation of the function space.
2 Related Work
Mesh-based networks for physics. The initial attempts learn physical dynamics primarily centered around convolutional neural networks (CNNs) and graph neural networks (GNNs). Both leverage discrete convolutions extract relevant information from given node's neighborhood Hamilton (2020). CNNs expect inputs and outputs to be on regular grid. Their adaptation to irregular data through interpolation (Chae al., 2021) is limited to simple meshes. GNNs work on irregular meshes (Hamilton et al., 2017; Veličković et al., 2018; 88; Pfaff et al., 2021) and have been used c.g. for dynamics modeling (Brandstetter al., 2022b) or design optimization (Allen et al., 2022). They typically select nearest neighbors within small radius, which can introduce biases towards the type of meshes seen during training. In Section 4.2, we show that this bias can hinder their ability to generalize to meshes with different node locations or levels of sparsity. Additionally, they require significantly more memory resources than plain CNNs to store nodes' neighborhoods, which limits their deployment for complex meshes.
Operator learning. Operator learning is burgeoning field in deep learning for physics that focuses on learning mappings between mininte-dimensionall functions. Two prominent approaches are DeepONet (Lu et al., 2021) and Fourier Neural Operator (FNO; Li et al., 2021). DeepONet can query any coordinate in the domain for a value of the output function. However, the input function must be observed on a set of predefined locations, requiring the same observation grid for all observations, for training and testing. FNO is an instance of neural operators (Kovachki et al., 2021), family of approaches that integrate kernels over the spatial domain. Since this operation can be expensive, FNO addresses the problem by employing the fast Fourier transform (FFT) to transform the inputs into the spectral domain. As a consequence it cannot be used with irregular grids. Li al. (2022a) introduce an FNO extension to handle more flexible geometries, but it is tailored for design problems. To summarize, despite promising for several applications, current operator approaches still face limitations to extrapolate to new geometries; they do not adapt to changing observation grids or are limited to fixed observation locations. Recently, Li et al. (2023); Hao et al. (2023) explored transformer- based architectures an alternative approach.

---

Initial Value Problem
Monsete
(a) Cylinder
Dynamics modeling
Viriilly
My Mipl
(b) Navier-Stokes
(c) Shallow Water
Geometry-aware inference
Moch nunter
X PT
Elasticity
NACA-Ealer
Illustration the problem classes addressed in this work: Initial Value Problem (IVP) (a), dynamic forecasting (b and and geometry-awer -awre inference (d and e).
Figure I:
Spatial INRs. Spatial INRs class of coordinate- based neural networks that model data as realization of implicit function of a spatial location x E +> fa(x)) (Tancik et al., 2020; Sitzmann et al., 2020b; Fathony et al., 2021; Lindell et al., 2022). An INR be queried at any location, but encodes only one data sample or function. Previous works use meta-learning (Tancik al., 2021; Sitzmann et al., 2020a), auto-encoders (Chen & Zhang, 2019; Mescheder et al., 2019), modulation (Park et al., 2019; Dupont et al., 2022) to address this limitation by enabling an INR to decode various functions using per-sample parameters. INRs have started to gain traction in physics, where they have been successfully applied to spatio-temporal forecasting (Yin et al., 2022) and reduced-order modeling (Chen al., 2022). The former work is probably the closest designed for forecasting and cannot handle the range of tasks that CORAL can address. Moreover, its computational cost is significantly higher than CORAL's, which limits its application in real-world problems. The work by Chen et al. (2022) aims to inform the INR with known PDEs, similar to PINNs, whereas our approach is entirely data-driven and without physical prior.
but is
The CORAL Framework
In this section, present the CORAL framework, novel approach that employs encode-process: decode structure to achieve the mapping between continuous functions. We first introduce the model and then the training procedure.
3.1 Problem Description
Let Ω C. Id be a bounded open set of spatial coordinates. We assume the existence of mapping G* from one infinite-dimensional space A C (2( Rdd.) to another ii L Rd.), ). such that for any observed pairs (a,, u;) A x U1 a probability measure supported on and Vn the pushforward measure v. by g". We seek to approximate this operator by an i.i.d. collection of point-wise evaluations of input-output functions through a highly flexible formulation that can be adapted to multiple tasks. In this work, we target three different tasks examples: solving an initial value problem, i.e. mapping the initial condition Mp - + uax i = 0) to the solution predefined time ur = +> u(x, T), modeling the dynamics of physical system over time (u, U(+5t) over given forecasting horizon or
(a:)). We have ~ Us ~ where va

---

prediction based on geometric configuration. At training time, we have access to fer pairs of input and output functions (a,, u4) M evaluated over a free-form spatial grid X. We denote alx; (a(x) ) and u|,x; = (u(æ: ex; the vectors of the function values over the sample grid. In the context of the initial value and geometry-aware problems, every sample observed on specific grid 23. For dynamics modeling, we use unique grid Xer for all the examples to train the model and another grid Lte for testing.
obs.
Input lunction
output space
output function
(a))
(u(A)
de
x
X
x
encode inputs
predicted output values on query grid
e
&x
Zg
forecasted code
Figure 2: Inference for CORAL. First, the model embeds the input function without constraints on the locations of the observed sensors into input latent code žu. then infers the output latent code zu and finally predicts the output value ú(x) for any query coordinate Q. For the grid X, we use the vector notation x (a(x))xEX, 0 x (ū(x))xxx.
3.2 Model
CORAL makes use of two modulated INRs, fo..d. and fo..... . for respectively representing the input and output functions of an operator. While 0g and 6, denote shared INR parameters that contribute in representing all functions a; and u,, the modulation parameters tha and Φu, are specific to each function a, and u4 Given input/output INR functions representation, CORAL then learns a mapping between latent representations inferred from the two INRs' modulation spaces. The latent representations Zaz Zus are low dimensional embeddings, capturing within compact code information from the INRs' parameters. They are used as inputs to hypernetworks ha and hy to compute the modulation parameters do, = ha (Za,) and d.,- = hu(z,,). The weights of the input and output hypernetworks are respectively denoted wa and w..
CORAL proceeds in three steps: encode, to project the input data into the latent space; process, to perform transformations in the latent space: and decode. to project the code back to the output function space. First, the imput function is encoded into the small input latent code za using spatial encoder ea A + R.d. Next, a parameterized model fs : Rdd TRd used to infer an output latent code. Depending on the target task, Je can be as simple as plain MIP or more complex as for example a neural ÓDE solver (as detailed later). Finally, the processed latent code decoded into a spatial function using decoder &. Rd= + U. The resulting CORAL operator then writes as Ea G o ea, as shown in Figure The three steps are detailed below.
Encode Given an input function a; and a learned shared parameter 0a. the encoding process provides code 2a; (a.). This code is computed by solving inverse problem through procedure known auto-decoding, which proceeds as follows. We want to compress into a compact code Za, the information required for reconstructing the original field a; through the input INR, i.c.:
VI E X,, foa. (x) (x))) x (I(z) with da, = ha (Z..). See Figure 3a in Appendix B for details.
The approximate solution to this inverse problem is computed as the solution e, (a,)) = (K) of gradient descent optimization:
Zas 2
(k+1) -
a (e) Cp.. (S. (2), a); with ( ha (x( for 0 < (1) An.
where is the inner loop learning rate, K the number of inner steps, and L,,, (v. w) E (1(x)) (z)))2 for a measure μ4 over Q. Note that in practice, μ, is defined through the observation grid

---

Nr [.,,(+)===ez) (-) where 6x(-) ) is the Dirac measure. Since we can query the INRs anywhere within the domain, we can hence freely encode functions without mesh constraints. This is the essential part of the architecture that enables us to feed data defined on different grids to the model. We show the encoding flow in Appendix B, Figure
Process Once we obtain Za,, we can infer the latent output code Z.; (2a,), ). For simplification, we consider first that Gh implemented through an MLP with parameters vp. For dynamics modeling, in Section 4.2, we will detail why and how to make use of Neural ODE solver for De-
Decode We decode sns with the output hypernetwork hy and modulated INR and denote &u the mapping that associates to code Sn, the function ',......" where कu, = hr,(2,)) Since ő... is an INR, i.e function of spatial coordinates, it can be freely queried at any point within the domain. We thus have Vx E Q2 ú = Eu (2.,, (z) = fe.. (x). See Figure 3b in Appendix B for details.
During training, we will need to learn to reconstruct the input and output functions and ui. This requires training a mapping associating an input code to the corresponding input function &a Rd + A and a mapping associating function to its code in the output space eu l + R even though they are not used during inference.
3.3 Practical implementation: decoding by INR Modulation
We choose SIREN (Sitzmann al., 2020b) state-of-the-arr coordinate-based network as the INR backbone of our framework. SIREN is a neural network that uses sine activations with specific initialization scheme (Appendix B).
fe(x) WL ..pee.....ooooo oz() Py, with (m)) sin b:))
(2)
where T/ = J and (7))21 are the hidden activations throughout the network. wo E T hyperparameter that controls the frequency bandwidth of the network, W and b are the network weights and biases. We implement shift modulations (Perez et al., 2018) to have small modulation space and reduce the computational cost of the overall architecture. This yields the modulated SIREN:
(x) =
o (z())) + br, with o; (m)) = sin sin( Way, + b, + &.)) (3)
with shared parameters 0 = (W,, by) 2 and example associated modulations Φ (6:) We
compute the modulations & from with a linear hypernetwork i.e. for 0 < i < L -
V1= + C;. The weights V, and constitute the parameters of the hypernetwork w = (V,, cz i=
This implementation similar to that of Dupont al. (2022), which use a modulated SIREN for representing their modalities.
3.4 Training
We implement a two-step training procedure that first learns the modulated INR parameters, before training the forecast model fø- It is very stable and much faster than end-to-end training while providing similar performance: once the input and output INRs have been fitted, the training of gø is performed in the small dimensional modulated INR z-code space. Formally, the optimization problem is defined as:
arg min Ea,uors 2n (E)(a)) - (u)l
s.t. G. = arg min Ear,, L(Ea ea (a), a)
(4)
tn
and é. arg min Eu LlEa eu cu(u),
Note that functions (ew, S..) and (eaz are parameterized respectively by the weights (0n,, w.) and (0.; Wa), of the INRs and of the hypernetworks. In Equation (4), we used the (ew) & (ea, Ea) description for clarity, but as they are functions of (0.., w..) & (0a, w.), optimization is tackled on the latter parameters. We outline the training pipeline in Appendix B, Figure 5. During training,
5

---

constrain eu2 to take only few steps of gradient descent to facilitate the processor task. This regularization prevents the architecture from memorizing the training set into the individual codes and facilitates the auto-decoding optimization process for new inputs. In order to obtain network that is capable of quickly encoding new physical inputs, we employ a second-order meta-learning training algorithm based on CAVIA (Zintgraf et al., 2019). Compared to first-order scheme such as Reptile (Nichol et al.. 2018), the outer loop pack-propagates the gradient through the K inner steps, consuming more memory we need to compute gradients of gradients but yielding higher reconstruction results with the modulated SIREN. We experimentally found that using 3 inner-steps for training, or testing, was sufficient to obtain very low reconstruction errors for most applications.
4 Experiments
To demonstrate the versatility of our model, we conducted experiments on three distinct tasks (Figure 1): (i) solving an initial value problem (Section 4. (ii) modeling the dynamics of physical system (Section 4.2), and (iii) learning to infer the steady state of a system based on the domain geometry (Section 4.3) plus an associated design problem in Appendix D. Since each task corresponds to different scenario, we utilized task-specific datasets and employed different baselines for each task. This approach was necessary because existing baselines typically focus on specific tasks and do not cover the full range of problems addressed in our study, unlike CORAL. We provide below an introduction to the datasets, evaluation protocols, and baselines for each task setting. All experiments were conducted on a single GPU: NVIDIA RTX A5000 with 25 Go. Code will be made available.
4.1 Initial Value Problem
An IVP is specified by initial condition (here the input function providing the state variables at t = 0) and a target function figuring the state variables value at a given time T'. Solving an IVP is a direct application of the CORAL framework introduced in Section 3.2.
Datasets We benchmark our model on two problems with non-convex domains proposed in Pfaff et al. (2021). In both cases, the fluid evolves in a domain - which includes an obstacle - that is more densely discretized near the boundary conditions (BC). The boundary conditions are provided by the mesh definition, and the models are trained on multiple obstacles and evaluated at test time on similar but different obstacles. Cylinder simulates the flow of water around cylinder on a fixed 2D Eulerian mesh, and is characteristic of incompressible fluids. For each node j we have access to the node position x!(5), the momentum w(((6)) and the pressure p(x(()). We seek to learn Airfoil simulates the aerodynamics around the cross-section of an airfoil wing, and is an important use-case for compressible fluids. In this dataset,
the mapping (z, Wo (x). Po (o(z))
+ (wr(x), pr(x))
have in addition for each node the fluid density p(xx()),, ), and we seek to learn the mapping (z, wo (x), (o(z), Po(x)) + (wT (z), pr (x)), pr pr))))xxX. For both datasets, each example is associated a mesh and the meshes are different for each example. For Airfoil the average number of nodes per mesh is 5233 and for Cylinder 1885.
Evaluation protocols Training is performed using all the mesh points associated to an example. For testing we evaluate the following two settings. Full, we validate that the trained model generalizes well to new examples using all the mesh location points of these examples. Sparse We assess the capability of our model to generalize on sparse meshes: randomly selecting 20% of its nodes. We use a train, validation, test split of 1000 / 100 / 100 samples for all the evaluations.
the original input mesh is down-sampled by
Baselines We compare our model to + NodeMLP, a FeedForward Neural Network that ignores the node neighbors and only learns a local mapping GraphSAGE (Hamilton et al., 2017), a popular GNN architecture that uses SAGE convolutions MP-PDE (Brandstetter et al., 2022b), a message passing GNN that builds on (Pfaff et al., 2021) for solving PDEs.
Results. We show in Table 1 the performance on the test sets for the two datasets and for both evaluation settings. Overall, CORAL is on par with the best models for this task. For the Full setting, is best on Cylinder and second on Airfoil behind MP-PDE. However, for the spurse protocol, can infer the values on the full mesh with the lowest error compared to all other models. Note that this

---

second setting is more challenging for Cylinder than for Airfoil given their respective average mesh size. This suggests that the interpolation of the model outputs more robust on the Airfoil dataset, and explains why the performance of NodeMIPP remains stable between the two settings. While MP-PDE is close to CORAL in the sparse setting, GraphSAGE fails to generalize, obtaining worse predictions than the local model. This is because the model aggregates neighborhood information regardless of the distance between nodes, while MP-PDE does consider node distance and difference between features.
Table 1: Initial Value Problem Test results. MSE on normalized data.
Model
Cylinder
Airfoil
Full
Sparse
Full
Sparse
NodeMI.P
1.48e-1 + 2.00e-3
2.29e-1
3.06e-3
2.88e-1 + 1.08e-2
2.83e-1 + 2. 22-3
GraphSAGE
7,40e-2 + 2.22e-3
2.66e-1 1 + 5.03e-3
2.47e-1 = 7.23e-3
5.55e-1 + 5.54e-2
MP-PDE
8,72e-2 + 4.65e-3
1.84e-1 4.58c-3
1.97e-1 + 1.34e-2
3.07e-1 + 2.56e-2
CORAL
7.03e-2 + 5.96e-3
1.70e-1 2.53e-2
2.40e-1
= 4.336--
2.43e-1 + 4.14e-3
4.2 Dynamics Modeling
For the IVP problem, in section 4.1, the objective was to infer directly the state of the system at given time T given an initial condition (IC). We can extend this idea model the dynamics of physical system over time, so as to forecast state values over given horizon. We have developed an autoregressive approach operating on the latent code space for this problem. Let us denote (uo. u8t,... .... ( a target sequence of observed functions of size I, + 1. Our objective will be to predict the functions Ugst; k 1,... L, starting from an initial condition No. For that we will encode 20 = e(uo), then predict sequentially the latent codes 28st, k = 1, ..... L using the processor in an auto regressive manner, and decode the successive values to get the predicted usst, = I,... L at the successive time steps.
4.2.1 Implementation with Neural ODE
The autoregressive processor implemented by Neural ODE solver operating in the latent z-code space. Compared to the plain MLP implementation used for the IVP task, this provides both a natural autoregressive formulation, and overall, an increased flexibility by allowing to forecast at any time in sequence, including different time steps or irregular time steps. Starting from any latent state
2, neural solver predicts state ž(+- ät+r 2 Jetr (= with Sy a neural network with parameters to be learned, for any time step т. The autoregressive setting directly follows from this formulation. Starting from 20, and specifying series of forecast time steps kôt for k 1..... the solver call NODESolve(C
will compute predictions 2kst: = 1...., L autoregressively, i.e. using zkät as a starting point for computing 2(k-1)55c- In our experiments have used a fourth-order Mange-Kutta scheme (RK4) for solving the integral term. Using notations from Section 3.2, the predicted field at time step is can be obtained ūkst { g (e(uo))) with of indicating & successive applications of processor 96- Note that when solving the IVP problem from Section 4, two INRs are used, one for encoding the input function and one for the output function; here single modulated INR Sa.e is used to represent a physical quantity throughout the sequence any time. 0 is then shared by all the elements of a sequence and & is computed by a hypernetwork to produce function specific code.
20:
We use the two-step training procedure from Section 3.4, i.c. first the INR is trained to auto-decode the states of each training trajectory, and then the processor operating over the codes is learned through a Neural ODE solver according to Equation (5). The two training steps are separated and the codes kept fixed during the second step. This allows for a fast training as the Neural ODE solver operates on the low dimensional code embedding space.
arg min Ew 10w. (u.)l1
(5)
7

---

4.2.2 Experiment details
Datasets We consider two fluid dynamics equations for generating the datasets and refer the reader to Appendix A for additional details. 2D-Navier-Stokes equation (Navier-Stokes) for viscous, incompressible fluid in vorticity form on the unit torus:
0F + Vw vÂw + f. Vu 0 for 10-3 is the viscosity coefficient. The train and test are composed of
the
S2, > O, where 256 and 16 trajectories respectively where we observe the vorticity field for 40 timestamps. The original spatial resolution is 256 256 and we sub-sample the data obtain frames of size 64 x 64. 3D-Spherical Shallow-Water equation (Shallow-Watee
can be used as an approximation to flow on the earth's surface. The data consists of the vorticity w, and height h of the fluid. The train and test sets are composed respectively of 16 and 2 long trajectories, where we observe the vorticity and height fields for 160 timestamps. The original spatial resolution is 128 (lat) x 256 (long), which we sub-sample to obtain frames of shape 64 x 128. We model the dynamics with the complete state (h, w). Each trajectory, for both both datasets and for train and test generated from different initial condition (IC).
Setting We evaluate the ability of the model to generalize in space and time. Temporal extrapo- lation; For both datasets, we consider sub-trajectories of 40 timestamps that we split in two equal parts of size 20, with the first half denoted In-r and the second one Out-f. The training-In: In-r set is used to train the models at forecasting the horizon t = 1 to = 19. At test time, we unroll the dynamics from new IC until / 39. Evaluation in the horizon In-/ assesses CORAL'S capacity to forecast within the training horizon. Out-t allows evaluation beyond In-l, from ( = 20 to = 39. Varying sub-sampling: We randomly sub-sample æ percent of regular mesh to obtain the train grid Xe.. and a second test grid Xte. that are shared across trajectories. The train and test grids are different, but have the same level of sparsity. Up-sampling: We also evaluate the up-sampling capabilities of CORAL in Appendix C. In these experiments, we trained the model on a sparse, low-resolution grid and evaluate its performance on high resolution-grids.
Baselines To assess the performance of CORAL, we implement several baselines: two operator learning models, one mesh-based network and one coordinate-based method. . DeepONet (Lu et al., 2021): we train DeepONet in an auto-regressive manner with time removed from the trunk net's input. FNO (Li et al., 2021): we an rego-regressive version of the Fourier Neural Operator. MP-PDE (Brandstetter et al., 2022b): we use MP-PDE as the irregular mesh-based bascline. We fix MP-PDE's temporal bundling to 1, and train the model with the push- forsard trick. DINo (Yin et al., 2022): We finally compare CORAL with DINo, an INR-based model designed for dynamics modeling.
4.2.3 Results
Table 2: Temporal Extrapolation Test results. Metries in MSE.
dataset
Navier-Stokes
Shallow Water
N 1 Xie
In-t
Out-t
In-1
Out-t
DeepONet
4.72e-2 + 2.84e-2
9.58e-2 + 1.83e-2
6.54e-3 + 4.94e-4
8.93e-3 + 9.42e-5
100% regular grid
FNO
5.68e-4 + 7.62e-5
8.95e-3 +: 1.50e-3
3.20e:5 + 2.51e=5
1.17e- :: 3.0le-5
MP-PDE
4.39e-4 i 5.78e-5
4.46e-3 +: 1.28e-3
9.37e:5 + 6.56e-
1.53e-3 + 2.62e-4
DINo
1.27e-3 2.22e-5
1.1le-2 i: 2.28e-3
4.48e-5 1: 6
2.63e-3 1 1.36e-4
CORAL
1.86e-4 + 1.44e-5
1.02e-3 + 8.62e-5
3.44e-6 + 4.01e-7
4.82c-4 + 5.166-
DeepONet
8.37e-1 2.07e-2
7.80e- +: 2.36e-2
1.05e-2 + 5.01e-4
1.09e-2 + 6.16
т 20% irregular grid
FNO + lin. int.
3,97e- 1: 8.00ee- 4.03e-4
9.92e-3 :: 2.36e-3
n.a.
n.a. 2.56e-2 1 8.23e-3
MP-PDE
3,98e-2 1 1.69e-2
1.31e- 1: 5,34e-2
5.28e-3 1 5.25e
DINo
9,99e-4 1 6.71e-3
8.27e-3 :: 5.61e-3
2,20e-3 1 1.06e-4
4.94g- + 1.92e-4
CORAL
2.18e-3 6.88e-4
6.67e-3 +: 2.01e-3
1.4le:3 + 1.39e-4
2.11e-3 + 5.58e-5
DeepONet
7,86e-1 + 5.48e-2
7.48e-1 + 2.76e-2
1.1 + 6.94e-4
1.12e-2 + 7.79e-4
= 5% irregular grid
FNO + lin. int.
3.87e-2 + 1.44e-2
5.19e-2 + 1.10e-2
n.a. 1.10e-2 1 4.23e-3
n.a. 4.94e-2 1: 2.36e-2
MP-PDE
1.92e-1 1: 9.27e-2
4.73e-1 = 2.17e-T
DINo
8.65e-2 1: 1.16e-2
9.36e-2
1 9.34e-3
1.22e-3 +: 2.05e-4
1.52e-2 + 3.74e-4
CORAL
2.44e-2 + 1.96e-2
4.57e-2 + 1.78e-2
8.77e:3 + 7.20e-4
1.29e-2 + 1.92e-3
8

---

Table 2 details the performance of the different models in combined temporal and spatial evaluation setting. General remarks: CORAL demonstrates strong performance across all scenarios for both datasets. Only DINo exhibits similar properties, i.e., stability across spatial subsamplings and extrapolation horizon. We observe that all models performance degrade with lower sampling ratio. Also, as the models have been trained only on In-r horizon, error accumulates over time and thus leads to lower performance for Out-f evaluation. Analysis per model: Although achieving strong performance on some specific scenarios, DeepONet, FNO and MP-PDE results are dependent of the training grid, geometries or number of points. FNO, can only be trained and evaluated on regular grids while DeepONet is not designed to be evaluated achieves strong performance with enough sample positions, e.g. full grids here but struggles to compete irregular grids scenarios in Navier-Stokes. Inference Time: We report in Appendix C, the inference time of the baselines considered. Despite operator methods have better inference time, CORAL is faster than mesh-free methods like DINo and MP-PDE. Generalization across samplings: Coordinate-basse
different grid in the branch net. MP-PDE
methods demonstrate robustness when comes to changes in spatial resolution. In contrast, MP-PDE model exhibits strong overfitting to the training grid, resulting in decline in performance. Although MP-PDE and DINo may outperform CORAL in some settings, when changing the grid, CORAL remains stable and outperforms the other models. See Appendix C for details.
4.3 Geometry-aware inference
In this section, we wish to infer the steady state of system from its domain geometry, all other parameters being equal. The domain geometry is partially observed from the data in the form of point clouds or of a structured mesh X, C 22 The position of the nodes depends the particular object shape. Each mesh X; is obtained by deforming a reference grid X' to adjust to the shape of the sample object. This grid deformation is the input function of the operator learning setting, while the output function is the physical quantity Uq over the domain Ωz The task objective is to train a model so to generalize new geometries, e.g. new airfoil shape. Once surrogate model has been trained to learn the influence of the domain geometry on the steady state solution, it can be used to quickly evaluate new design and to solve inverse design problems (details in Appendix D).
Datasets. We used datasets generated from three different equations by Li et al. (2022a) and provide more details in Appendix A. Euler equation WVCA-EEuer for a transonic flow over NACA-airfoll. The measured quantity at each node is the Mach number. Navier-Stokes Equation (Pipe) for an incompressible flow in pipe, expressed in velocity form. The measured quantity at each node is the horizontal velocity. Hyper-elastic material (Elasticity). Each sample represents a solid body with void in the center of arbitrary shape, on which tension is applied at the top. The material is the incompressible divlin-Saunders material and the measured quantity is the stress value. We use 1000 samples for training and 200 for test with all datasets.
Baselines We use Geo-FNO (Li et al., 2022a) and Factorized-FNO (Tran et al., 2023) two SOTA models as the main baselines. We also compare our model to regular-grid methods such as FNO (Li et al., 2021) and UNet (Ronneberger al., 2015), for which we first interpolate the input.
Table 3: Geometric aware inference Test results. Relative 1.2 error.
Model
NACA-Euler
Elasticity
Pipe
FNO
3.85e-2 3.15ee 3.13e-3
4.95e-2 +: 1.21e-3
1.53e-2 = 3.19
UNet
5.05e-2 + 1.25e-3
5.34c-2 +: 2.89e-4
2.98e-2 + 1.08c-2
Geo-FNO
1.58e-2 +: 1.77e-3
3.41e-2 : 1.93e-2
6.59e-3 = 4.67e-4
Factorized-FNO
6,20e-3 + 3.00e-4
1.96e-2 + 2.00c-2
7.33e-3 + 4.66c-4
CORAL
5.90e-3 + 1.00e-4
1.67e-2 + 4.18e-4
1,20e-2 = 8.74e-
Results In Table 3 we can see that CORAL achieves state-of-the-ar results on Airfoil and Elasticity. with the lowest relative error among all models. It is slightly below Factorized-FNO and Geo-FNO on Pipe. One possible cause is that this dataset exhibits high frequency only along the vertical dimension, while SIREN might be better suited for isotropic frequencies. Through additional experiments, we demonstrate in Appendix D, how CORAL can also be used for solving inverse problem

---

corresponding to design task: optimize the airfoil geometry to minimize the drag over lift ratio. This additional task further highlights the versatility of this model.
Discussion and limitations
Although versatile model, CORAL inherits the limitations of INRs concerning the training time and representation power. then faster to train than GNNs, but slower than operators such as FNO, DeepONet and of course CNNs which might limit large scale deployments. Also some physical phenomena might not be represented via INRs. Although this is beyond the scope of this paper, remains to evaluate the methods on large size practical problems. An interesting direction for future work would be to derive an efficient spatial latent representation for INRs, taking inspiration from grid-based representation for INRs (Takikawa et al. (2022), Müller al. (2022), Saragadam et al. (2022)). Another avenue would be to leverage clifford layers (Brandstetter et al., 2022a) to model interactions between physical fields.
6
Conclusion
We have presented CORAL, a novel approach for Operator Learning that removes constraints on the input-output mesh. CORAL offers the flexibility handle spatial sampling or geometry variations, making it applicable to a wide range of scenarios. Through comprehensive evaluations on diverse tasks, We have demonstrated that consistently achieves state-of-the-art competitive results compared to baseline methods. By leveraging compact latent codes to represent functions, it enables efficient and fast inference within a condensed representation space.
Acknowledgements
We acknowledge the financial support provided by DL4CLIM (ANR-19 MAN--I9CHILA0001-000 and DEEP. NUM (ANR-21-CF223-0017-22) ANR projects, as well as the program France 2030, project 22- PETA-0002. This project was provided with computer and storage resources by GENCI at IDRIS thanks to the grant AD01 1013522 on the supercomputer Jean Zay's V100 partition.
References
Kelsey R. Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalezz, Pe- ter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. 2 2022. URL
00728.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR, abs/1506.03099, 2015. URL http::/aaxe org/abs 11066 03099.
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for pde modeling. In The Eleventh International Conference on Learning Representations, 2022a.
Johannes Brandstetter, Daniel E Worrall, and Max Welling. Message passing neural pde solvers. International Conference on Learning Representations, 2022b.
Keaton J. Burns, Geoffrey M. Vasil, Jeffrey S. Oishi, Daniel Lecoanet, and Benjamin P. Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2, 2020. ISSN 2643156 1564. doi: 10. (00ss/hyssevRessacch2200000 2.03068.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (pinns) for fluid mechanics: a review. Acta Mechanica Sinica/Lixue Xuebao, 37, 2021. ISSN 16143116. doi: 10. 00.1007/s1040--02111 148-11
Sangwon Chae, Joonhyeok Shin, Sungjun Kwon, Sangmok Lee, Sungwon Kang, and Donghyun Lee. Pm10 and pm2.5 real-time prediction models using an interpolated convolutional neural network. Scientific Reports, 11, 2021. ISSN 20452322. doi: 10. 10.1038//41598-02--912333
10

---

Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, Yue Chang, G A Pershing, Henrique Teles Maia, Maurizio Chiaramonte, Kevin Carlberg, and Eitan Grinspun. Crom: Continuous reduced-order modeling of pdes using implicit neural representations. International Conference on Learning Representation, 2022. URL
www.voorggaaaoe 2206 02607.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE Computer Society Conference on Computer Vision and Puttern Recognition, 2019-June, 2019. ISSN 10636919. doi: 10. 1111009//P..01999 19.0099.
Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: incorporating prior scientific knowledge. Journal of Statistical Mechanics: Theory and Experiment, 2019, 12 2019. ISSN 17425468. doi: 10. 1742-5466//b3311
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 10 2021. URL abs/2010 11929.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you can treat it like one. Proceedings of the 39 th International Conference on Machine Learning, 2022. URL httht:://aaxxi.or org abs/
2201. 12204.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. International Conference on Learning Representations. 2021.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Joseph Galewsky, Richard K. Scott, and Lorenzo M. Polvani. An initial-value problem for testing numerical models of the global shallow-water equations. Tellus, Series A: Dynamic Meteorology and Oceanography, 56, 2004. ISSN 02806495. doi:
William L. Hamilton. Graph Representation Learning: Foundations, Methods, Applications and Systems. Morgan and Claypool, 2020. ISBN 9781450383325. doi: 10. 10.1145744454883477882.
William L.. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs.
volume 2017-December, 2017.
Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pp. 12556- 12569. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Deee-bee,,,,r, 2016. ISSN 10636919. doi: 10. 10.1100/VP.22006..900
Ayed Ibrahim, de Bézenac Emmanueland Pajot Arthur, and Gallinari Patrick. Modelling spa- tiotemporal dynamics from Earth observation data with neural differential equations. Machine Learning, 111(6):2349 2380, 2022. ISSN 15730565. URL https:
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Neural operator: Learning maps between function spaces. 2021.
URL http:http///aiii org/ abs 22o// 08481.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep con- volutional neural networks. Communications of the ACM, 60, 2017. ISSN 15577317. doi: 10.1145/3065386,
Zhipeng Li, Fengwei Zhou, Fei Chen, Hang Li, Tongliang Liu, and Tat-Seng Chen. Meta-sgd: Learning to learn quickly for few-shot learning. In Proceedings of the 34th International Conference on Machine Learning-
Bearning-Volume Volume 70, pp. 2111-2120 JMLR. org, 2017.
11

---

Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Transactions on Machine Learning Research (April/2023). 2023.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations. 10 2021. URL http: http://axvvvv.oo/ org/ abs/ 08895.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier Neural Operator with Learned Deformations for PDEs on General Geometries. In (KKV:22077.5209vv 2022a.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar informed neural operator for learning partial differential equations. 39th International Conference on Machine Learning (ICML2022), 11 2022b.
Azizzadenesheli, and Anima Anandkumar,
URL http://///// //t://arre arxiv. org/ /2lll 03794.
David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. Conference on Computer Vision and Pattern Recognition, 12 2022. URL http: httttp://axxii... org/ abs 2aabs/210 22122.1..11 04645.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. volume 7, 2018.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. Nar Mach Intell, 3:218 229, 10 2021. doi: 10.1088/442556-22-0000000 s42256-22662 org/ org/abs// / 910. 13tttttt:::p /xx. doi. htt.://r/0....0//222222000000000 221022600000-0--2-
URL http:
abs
Lu Lu, Xuhui Meng. Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data, volume 393. Elsevier B.V., 4 2022. doi: 10.1016/j.cma. 1042.1
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019-June, 2019.
ISSN 10636919.
doi: 10. 2tt.9///PR...0...00.0
K. W. Morton and D. F. Mayers. Numerical solution partial differential equations: An introduction. 2005. doi: 10. 10.1077/BB0978051112222
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics, 41(4):1-15, jul 2022. doi: 00.1145352822333530111 URL https: //os:://doo..o wtt.:/oo.......... 1ttp///000..115525222222202 3530127.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Peter J. Olver. Introduction to partial differential equations. Undergraduate Texts in Mathematics. Springer Cham, 2014.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June, 2019. ISSN 10636919. doi: 10. 1119/CCVP..201.000022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neurul Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL /ttp::::///ppp....teem www.ripp....ppppp....eeeeee -nt---ann-mmm-ttee---e ive sty
pdf.
12

---

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, 2018. ISSN 2159-5399. doi: 10.1609/aaii.v322 11611.
Tobias Pfaff, Meire Fortunato, Alvaro Sanzhez-Gonzalez, and Peter W. Battaglia. Learning mesh- based simulation with graph networks, International Conference on Learning Representations, 10 2021. URL http::////.. arxiv www//agg.ss...oo.oo............e /2100 03409.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. volume 9351, 2015. doi: 10. 10.1007/97833311--2554444
Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard G. Baraniuk, and Ashok Veer- araghavan. Miner: Multiscale implicit neural representations, 2022.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 2020-December, 2020a. ISSN 10495258.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural 2020b. ISSN 10495258.
Information Processing Systems,
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. ACM Transactions Graphics, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 2220-Decembrr, 2020. ISSN 10495258.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2021. ISSN 10636919. doi: 10. N0.1099CVPR44643.2222... 1.00277.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. 11 2023. URL http: /atp:i////aivv///a orgg abs 2111.13800
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017-December, 2017. ISSN 10495258.
Petar Veličković, Arantxa Casanova, Pietro Liò, Guillem Cucurull, Adriana Romero, and Yoshua Bengio. Graph attention networks. 2018. doi: 0.11077997-33-01-001558
Yuan Yin, Matthieu Kirchmeyer, Jean- Yves Franceschi, Alain Rakotomamonjy, and Patrick Gallinari. Continuous pde dynamics forecasting with implicit neural representations. International Confer- on Learning Representations, 9 2022. URL /tttp://axii.... org/ o2g/abb//2299.18555
Luisa Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. 36th International Conference on Machine Learning, ICML 2019, 2019-June, 2019.
13

---

A Dataset Details
A.1 Initial Value Problem
We use the datasets from Pfaff et al. (2021), and take the first and last frames of each trajectory as the input and output data for the initial value problem.
Cylinder
The dataset includes computational fluid dynamics (CFD) simulations the flow around cylinder, governed by the incompressible Navier-Stokes equation. These simulations generated using COMSOL software, employing an irregular -riiiannul mesh. The trajectory consists 600 timestamps, with a time interval of At 0.01s between each timestamp.
The dataset contains CFD simulations of the flow around an airfoil, following the com- pressible Navier-Stokes equation. These simulations were conducted using SU2 software, using an irregular 2D-triangular mesh. The trajectory encompasses 600 timestamps, with a time interval of At 0.008s between each timestamp.
Airfoil
A.2 Dynamics Modeling
2D-Navier-Stokes (Navier-Stokes) We consider the 2D Navier-Stokes equation as presented in Li et al. (2021); Yin et al. (2022). This equation models the dynamics of an incompressible fluid on rectangular domain f2 [-1, 1/2. The PDE writes as:
+ Ow(x, t) - u(x,, 1) 1) () ( [ [0, 0t
(6)
w(x, f) V x w(x, 1), [-1, 1|, [0, Vu(x,t) 0, |- 1| [0
(7)
(8)
where u is the velocity, w the vorticity. is the fluid viscosity, and f the forcing term, given by: [(11, x2) = 0.1 (sin(2x(x) + x2)) + (2 + x2))) Ve E
For this problem, we consider periodic boundary conditions.
By sampling initial conditions in Li et al. (2021), we generated different trajectories on a 256 x 256 regular spatial grid and with a time resolution 8t We retain the trajectory starting from the 20th timestep so that the dynamics is sufficiently expressed. The final trajectories contains 40 snapshots at time = 20, 21, 59. As explained in section 4, we divide these long trajectories into 2 parts the 20 first frames are used during the training phase and are denoted In-f throughout this paper. The 20 last timesteps are reserved for evaluating the extrapolation capabilities of the models and are the Out-t part of the trajectories. In total, we collected 256 trajectories for training, and 16 for evaluation.
3D-Spherical Shallow- Water (Shallow-Watee
We consider the shallow-water equation on sphere describing the movements of the Earth's 's atmosphere:
du
dl f k x gVh vAu
(10)
dh - AV vΔh
(11)
dt
where d is the material derivative, l is the unit vector orthogonal to the spherical surface, is di the velocity field tangent to the surface of the sphere, which can be transformed into the vorticity = V x h is the height of the sphere. We generate the data with the Dedalus software (Burns et al., 2020), following the setting described in Yin et al. (2022), where a symmetric phenomena can be seen for both northern and southern hemisphere. The initial zonal velocity Uh contains two non-null symmetric bands in the both hemispheres, which are parallel to the circles of latitude. At each latitude and longitude 0 E
x n,, T|:
2 2
exp
0
if o (00,
uo(s,, 0)
Umax exp (0t0o)(0+01)
if Φ E (-01, -0o), otherwise.
(12)
(0 0,00
14

---

where umaz is the maximum velocity, @ = 7. $1 - :. height he is initialized by solving boundary value conditioned problem in Galewsky et al. (2004) which is perturbed by adding hg to ho:
Oo, and en exp( exp(-(x) (01-00))) ). The water
1o (o, 0) i cos(() () exp (-(-)
2
+ exp (- 00
(-(00t
)
Φ 2
2 3
exp
(13)
where 02 h = 120m, = 3 and 3 = are constants defined in Galewsky et al. (2004).
We simulated the phenomenon using Dedalus Burns et al. (2020) on Saittue-llnnitude grid (lat- lon). The original grid size was 128 (lat) x 256 (lon), which we downsampled to obtain grids of size 64 x 128. To generate trajectories, we sampled Umaa from uniform distribution (60, 80). Snapshots were captured every hour over a duration of 320 hours, resulting in trajectories with 320 timestamps. We created 16 trajectories for the training set and trajectories for the test set. However, since the dynamical phenomena in the initial timestamps were less significant, we only considered the last 160 snapshots. Each long trajectory then sliced into sub-trajectories of 40 timestamps each. As result, the training set contains 64 trajectories. while the test set contains 8 trajectories. It is worth noting that the data was also scaled to a reasonable range: the height was scaled by factor of 3 103, and the vorticity w was scaled by a factor of 2.
A.3 Geometric aware inference
We use the datasets provided by Li et al. (2022a) and adopt the original authors' train/test split for our experiments.
Euler's Equation (Naca-Fuler). We consider the transonic flow over an airfoil, where the governing equation is Euler equation, as follows:
Ops V. (pyu)) 0, Öt
Öpsu V (pfu pll) =
aE + V ((F + (00)))=0 = 0,
(14)
Ot
at
where Pj is the fluid density, u is the velocity vector, p is the pressure, and E is the total energy. The viscous effect is ignored. The far-field boundary condition is pro. 1, Pox 1.0, Mo. 0.8, AoA = 0. where Mx is the Mach number and AoA is the angle of attack. At the airfoil, a no- penetration condition is imposed. The shape parameterization of the airfoil follows the design element approach. The initial NACA-0012 shape is mapped onto a "cubic" design element with 8 control nodes, and the initial shape morphed control nodes of the design element. The displacements of control nodes are restricted to the vertical direction only, with prior d 41-0.05, 0.05].
different one following the displacement field of the
We have access to 1000 training data and 200 test data, generated with a second-order implicit finite volume solver. The C-grid mesh with about (200 x 50) quadrilateral elements is used, and the mesh is adapted near the airfoil but not the shock. The mesh point locations and Mach number on these mesh points are used input and output data.
Hyper-elastic material (Elasticity).
The governing equation of a solid body can be written as i2u Ps 022 V
where Ps is the mass density, is the displacement vector, and is the stress tensor. Constitutive models, which relate the strain tensor e to the stress tensor, are required to close the system. We consider the unit cell problem Ω = [0, 1] x [0, 1] with an arbitrary shape void at the center, which is depicted in Figure 2(a). The prior of the void radius is r = 0.2+0.2 0.2 with F F~ (0, 4 ) < 0.4. The unit cell is clamped on the bottom edges and tension traction t = [0, 100) is applied on the top edge. The material is the incompressible Rivlin- Saunders material with energy density function parameters C₁ 1.863 x 10 and G 9.79 10 The data was generated with a finite element solver with about 100 quadratic quadrilateral elements. The inputs given as point clouds with size around 1000. The target output is stress.
1 1+ exx which embeds the constraint 0.2
Savier-Stokes Equation (Pipe). We consider the incompressible flow a pipe, where the govern- ing equation is the incompressible Navier-Stokes equation, as following.
dv + (v V)u - vVV, v-r=0
0t
15

---

where e is the velocity vector, p is the pressure, and f 0.005 is the viscosity. The parabolic velocity profile with maximum velocity [1, 0] is imposed the inlet. A free boundary condition is imposed at the outlet, and no-slip boundary condition is imposed at the pipe surface. The pipe has length of 10 and width of 1. The centerline of the pipe is parameterized by 4 piecewise cubic polynomials, which are determined by the vertical positions and slopes on 5 spatially uniform control nodes. The vertical position at these control nodes obeys ~ U[ 2, 2]. and the slope at these control nodes obeys d ~ U 1].
We have access to 1000 training data and 200 test data, generated with an implicit finite element solver using about 4000 Taylor-Howl Q2-Q1 mixed elements. The mesh point locations (129 x 129) and horizontal velocity on these mesh points are used as input and output data.
B Implementation Details
We implemented all experiments with PyTorch (Paszke et al., 2019). The code is available at https: pupps://anonn /anonyoouu 4open. wwweeeeee///oraa-eeeeeeeeeeeee time needed for development and the different experiments to approximately 400 days.
We estimate the computation
B.1 CORAL
B.1.1 Architecture Details
SIREN initialization. We use for SIREN the same initialization scheme in Sitzmann al. (2020b), i.c., sampling the weights of the first layer according to a uniform distribution U(-1/d,1// and the next layers according to 1old(- wo
6 den
6. We use the default Py PoTorh initialization das
for the hypernetwork.
Decode with shift-modulated SIREN. Initially, we attempted modulate both the scale and shift of the activation, following the approach described in Perez al. (2018). However, did not observe any performance improvement by employing both modulations simultaneously. Consequently, we decided to focus solely shift modulations, it led to more stable training process and reduced the size the modulation space by half. We provide an overview of the decoder with the shift-modulated SIREN in Figure 3.
Encode with auto-decoder. We provide a schematic view of the input encoder in Figure 4. The auto-decoding process starts from code =a = 0 and performs K steps of gradient descent over this latent code to minimize the reconstruction loss.
Process with MLP. We use an MLP with skip connections and Swish activation functions. Its forward function writes go (z) Block; c ... Block, (z), where Block is two-layer MLP with skip connections:
Block(=) z + a (W2- a (W, + bı) + b2)
(15)
In Equation (15), a denotes the feature-wizt Swish activation. We use the version with learnable parameter (3; (2) = 2 sigmoid(32) (32).
B.1.2 Training Details
The training is done in two steps. First, we train the modulated INRs to represent the data. We show the details with the pseudo-code in Algorithms I and 2. is the inner-loop learning rate while À is the outer loop learning rate, which adjusts the weights of the INR and hypernetwork. Then, once the INRs have been fitted, we obtain the latent representations of the training data, and use these latent codes to train the forecast model gø (See Algorithm 3). We note Au the learning rate of gv-
Z-score normalization. As the data is encoded using only few steps of gradients, the resulting standard deviation of the codes very small, falling within the range of [le-3, 5e-2]. However, these "raw" latent representations are not suitable as-is for further processing. To address this, we normalize the codes by subtracting the mean and dividing by the standard deviation, yielding the normalized code: 2porm mean Depending the task, we employ slightly different types of normalization:
16

---

Stada
SIREN
fondo
SIREN
hg
Modulation
hg = x
Modulation
h1+1 sin(wal (Wihs +Viza ct
h1-1 = sin(wol (oW/ooo(WWWW (Wih: + by + +V2 + ))
foawa (x) W + br+t
founbu (2) WL++fee+1 dr+1
obs. space
input function
oulpui
oulput function
f0 ())
Ω
IR da
R du
x
x
x'
ha
hypernetwork
hu
hypernetwork
&a
Ex
Zg
input
Z
input
(a) The hypernewoork ha maps the input code to the modulations da. The modulations shift the activations each layer of the SIREN.
(b) The hypernetwork hn, maps the input code &u to the modulations é., The modulations shift the activations at each layer of the SIREN.
Figure 3: Architecture of the input and output decoders &a. Eu. They can be queried on any coordinate Ω. We use the same notation for both, even though the parameters are different.
1. Initial value problem: Cylinder: We normalize the inputs and outputs code with the same mean and standard deviation. We compute the statistics feature wise, across the inputs and outputs. Airfoil: We normalize the inputs and outputs code with their respective mean and standard deviation. The statistics are real values.
2. Dynamics modeling:
normalize the codes with the same mean and standard deviation. The statistics are computed feature-wise, over all training trajectories and all available timestamps (i.e. over In-l).
3. Geometry-aware inference: We normalize the input codes only, with feature-wise statistics.
17

---

encoding as auto-decoding
input
input
flaba (.)
Cx(ā, a
Ω
I du
I da
X
a|
Ø|xx
ha
vV Zi (a,
(k) Za
x K steps
Co
oulput
Za
Figure 4: Starting from a code 2 = the input encoder ea performs inner steps of gradient descent over 2u to minimize the reconstruction loss Lx a) and outputs the resulting code 2 K) of this optimization process. During training, we accumulate the gradients of this encoding phase and back paok-propagate through the K inner-steps to update the parameters 0g and wa- At inference, we encode new inputs with the same number steps K and the same learning rate a, unless stated otherwise. The output encoder works in the same way during training, and is not used at inference.
Lx(a, a)
Cx(u, i
alx x
x
? X'
X
X
Step 1
Ea a
eu
Su
Input INR training
Output INR training
2
Zull
Train processor
Step2 2
to forecast output code
9ub
EL
Figure 5: Proposed training for CORAL (1) We first learn to represent the data with the input and output INRs. (2) Once the INRs trained, obtain the latent represenations and fix the pairs of input and output codes (Zaz ). We then train the processor to minimize the distance between the processed code (za,) and the output code Zn..
B.1.3 Inference Details
We present the inference procedure in Algorithm 4. It is important to note that the input and output INRs, So, and Se,, respectively, accept the "raw"" codes as inputs, whereas the processor expects normalized latent code. Therefore, after the encoding steps, we normalize the input code. Additionally, we may need to denormalize the code immediately after the processing stage. It is worth mentioning that we maintain the same number of inner steps as used during training, which is 3 for all tasks.
18

---

Algorithm 1: Training of the input INR
Algorithm 2:
Training of the output INR
while no convergence do Sample batch B of data (at)iEB: Set codes to zero Eo, +0, 0, Vi E B;
while no convergence do
Sample batch B of data (a4, U, KEB; Set codes to zero ou, + 0,0 Vi E B:
for i E B and step E {1, Ko} do
for E B and step E {_, Ku) do Zu;
Eoz f-
Ea; - Va =n; Lx; )= 0i);
En; au V. V.,, Lxx (ee, u4)
zo, : // output encoding inner stop
// input encoding inner
step
end
end
/+ outer doop update
/+ outer loop update
0a
0g
t 0y
EiEB Vo. Lx, (Fon ai);
EiEg Vo, (onnh u();
18
Wa
We
LieB V. (oon
Lier Vw. (Jon Ui)
B8
B
end
end
Algorithm 3: Training of the processor
while no convergence do Sample batch B of codes (Zai: 2n DieB;
/ processor update
vb + 1o
X | EiEB V ((L4())()))) z
end
Algorithm 4: CORAL Inference, given a function a
Set code zero 2n 0: for step E {l,..., K₂) do
Fa 4- Za Oa V Lx (J (u.,))
1/ input encoding inner step
end
2,
g (Ea))
// process latent code // decode output function
B.1.4 Choice of Hyperparameters
We recall that dg denotes the size of the code, wp a hyperparameter that controls the frequency bandwith of the SIREN network, À is the outer-loop learning rate (on Se, and hw)), is the inner-loop learning rate, K is the number of inner steps used during training and encoding steps at test time, Ag is the learning rate of the MLP or NODE. In some experiments we learn the inner-loop learning rate a, as in Li et al. (2017). In such case, the meta-a learning rate is an additional parameter that controls how fast we move a from its initial value during training. When not mentioned we simply report a in the tables below, and otherwise we report the initial learning rate and this meta-learning-rate
We use the Adam optimizer during both steps of the training. For the training of the Inference / Dynamics model, we use a learning rate scheduler which reduces the learning rate when the loss has stopped improving. The threshold is set to 0.01 in the default relative threshold model in PyTorch, with a patience of 250 epochs w.r.t. the train loss. The minimum learning rate is le-5.
Initial Value Problem We provide the list of hyperparameters used for the experiments on Cylinder and Airfoil in Table
Dynamics Modeling Table 5 summarizes the hyperparameters used in our experiments for dynam- ies modeling on datasets Navier-Stokes and Shallow- Waaer (Table 2).
Furthermore, to facilitate the training of the dynamics within the NODE, we employ Scheduled Sampling, following the approach described in Bengio et al. (2015). At each timestep, there is
19

---

Table 4: CORAL hyper-parameters for IVP/ Geometry-aware inference
Hyper-parameter
Cylinder
Airfoil
NACA-Fuler
Elasticity
Pipe
128
128
128
128
128
depth
width
256
256
256
256
128
30
30 / 50
5 15
10/ 15
5/
batch size epochs
32
16
16
2000
1500
5000
5000
5000
5e-6
5e-6
lc-4
le-4
5e-5
STREN Optimization
le-2
le-2
le-2
1c-2
le-2
meta-a learning
0
5e-6
le-4
lc-4
5e-5
Kg K2
depth
3
3
3
3
3
9+
width
64
64
64
64
128
activation
Swish
Swish
Swish
Swish
Swish
batch size
32
16
64
64
64
epochs
100
100
10000
10000
10000
Inference Optimization
A
1e-3
1e-3
1c-3
1e-3
le-3
Scheduler decay
0
0
0.9
0.9
0.9
Table 5: CORAL hyper-parammetrss for dynamics modeling
Navier-Stokes
Shallow- Water
d=
128
256
depth
4
INR
width
128
256
wi0
10
batch size
64
epochs
10, 000
10, 000
INR Optimization
5e-6
5e-6
le-2
le-2
3
depth
3
width
512
512
NODE
activation
Swish
Swish
solver
RK4
RK4
batch size
32
16
epochs
000
10. 000
Dynamics Optimization
A
le-3
le-3
Scheduler decay
0.75
0.75
20

---

probability of e% for the integration of the dynamics through the ODE solver to be restarted using the training snapshots. This probability gradually decreases during the training process. Initially, we set fiii 0.99, and every 10 epochs, we multiply it by 0.99. Consequently, by the end of the training procedure, the entire trajectory is computed with the initial condition.
Geometry-aware inference We provide the list of hyperparameters used for the experiments on NACA-Euler, Elasticity, and Pipe in Table 4.
B.2 Baseline Implementation
We detail in this section the architecture and hyperparameters used for the training of the baselines presented in Section 4.
Initial Value Problem We use the following baselines for the Initial Value Problem task.
NodeMLP. We use ReLU-MLP with 3 layers and 512 neurons. We train it for 10000 epochs. We use a learning rate of 1e-3 and a batch size of 64.
GraphSAGE. We use the implementation from torch-geometric (Fey & Lenssen, 2019), with 6 layers of 64 neurons. We use ReLU activation. We train the model for 400 epochs for Airfoil and 4,000 epochs for Cylinder. We build the graph using the 16 closest nodes. We use a learning rate of 1e-3 and a batch size of 64.
MP-PDE:
We implement MP-PDE as a 1 1-step solver, where the lime-bundling and pushfor- ward trick do not apply. We use 6 peassage-passingg blocks and 64 hidden features. We build the graph with the 16 closest nodes. We use a learning rate of 1e-3 and a batch size of 16. We train for 500 epochs on Airfoil and 1000 epochs on Cylinder.
Dynamics Modeling All our baselines are implemented in auto-regressive (AR) manner to perform forecasting.
DeepONet: We use a DeepONet in which both Branch Net and Trunk Net are 4-layers MIP with 100 neurons. The model is trained for 10, 000 epochs with learning rate of le-5. To complete the upsampling studies, we used a modified DeepONet forward which computes as follows: (1) Firstly, we compute an AR pass on the training grid to obtain prediction of the complete trajectory with the model on the training grid. (2) We use these prediction as input of the branch net for a second pass on the up-sampling grid to obtain the final prediction on the new grid.
FNO is trained for 2, 000 epochs with a learning rate of le-3. We used 12 modes and a width of 32 and Fourier layers. We also use a step scheduler every 100 epochs with a decay of 0.5.
FNO:
MP-PDE: We implement MP-PDE with time window of so that is becomes AR. The MP-PDE solver is composed of a 6 passage-passing blocks with 128 hidden features. To build the graphs, we limit the number of neighbors to 8. The optimization was performed on 10, 000 epochs with a learning rate of le-3 and step scheduler every 2000 epochs until 10000. We decay the learning rate of 0.4 with weight decay Ie-8.
DINo: DINo uses MFN model with respectively width and depth of 64 and for Navier- Stokes (NS), and 256 and 6 for Shallow-Water (SW). The encoder proceeds to 300 (NS) or 500 (SW) steps to optimize the codes whose size is set to 100 (NS) or 200 (SW). The dynamic is solved with NODE that uses 4-layers MLP and a hidden dimension of 512 (NS) or 800 (SW). This model is trained for 10, 000 epochs with a learning rate of 5e-3, We use the same scheduled sampling as for the CORAL training (see appendix 1.1)
Gearetry-aware inference Except on Pipe, the numbers for FactorizedFNO are taken from Tran et al. (2023). In the latter we take the 12-layer version which has a comparable model size. We train the 12-layer FactorizedFNO on Pipe with AdamW W for 200 epochs with modes (32, 16), a width of 64, a learning rate of le-3 and a weight decay of le-4. We implemented the baselines GeoFNO. FNO. UNet according to the code provided in Li et al. (2022a)
21

---

C Supplementary Results for Dynamics Modeling
C.1 Robustness to Resolution Changes
We present in Tables 6 and 7 the up-sampling capabilities of CORAL and relevant baselines both In-1 and Out-t, respectively for Navier-Stokes and Shallow-Ware Water.
Table 6: Up-sampling capabilities
Test results on Navier-Stokes dataset. Metries in MSE
dataset +
Navier-Stokes
Mer +
Xar X1e
64 x 64
N
64 x 64
128 X 128
256 x 256
In-t
Out-1
In-t
Out-t
In-t
Out-t
In-t
Out-t
DeepONet
1.47-22
7.90e-2
1.47e-2
7.90e-2
1.82e-1
7.90e-2
1.82e-2
7.90e-2
Fer = 100%
FNO
7.97e-3
1.77e-2
7.97e-3
1.77e-2
8.04e-3
1.80e-2
1.81e-2
7.90e-2
regular grid
MP-PDE
5.98c-4
2.80e-3
5.98c-4
2.80c-3
2.36e-2
4.61e-2
4.26c-2
9.77e-2
DINo
1.25c-3
1.13e-2
1.25e-3
1.13e-2
1.25e-3
1.13e-2
1.26c-3
1.13e-2
CORAL
2.02e-4
1.07e-3
2.02c-4
1.07e-3
2.08e-4
1.06e-3
2.19c-4
1.07e-3
DeepONet
8.35c-1
7.74c-1
8.28e-1
7.74c-1
8.32c-1
7.74c-1
8.28e-1
7.73e-1
Mrr 20%
MP-PDE
2.36e-2
1.11e-1
7.42e-2
2.13e-1
1.18e-1
2.95e-1
1.37e-1
3.39e-1
irregular grid
DINo
1.30e-3
9.58e-3
1.30e-3
9.59e-3
1.31e-3
9.63e-3
1.32-3
9.65e-3
CORAL
1.73e.3
5.61e-3
1.55e-3
4.34e-3
1.6le-3
4.38e-3
1.65e-3
4.4le-3
DeepONet
7.12c-1
7.16e-1
7.22c-1
7.26c-1
7.24e-1
7.28c-1
7.26e-1
7.30c-1
= 5%
MP-PDE
1.25e-1
2.92e-1
4.83e-1
1.08
6.11e-1
1.07
6.49e-1
1.08
irregular grid
DINo
8.21e-2
1.03e-1
7.73e-2
7.49e-2
7.87e-2
7.63e-2
7.96e-2
7.73e-2
CORAL
1.56e-2
3.65e-2
4.19e-3
1.12c-2
4.30e-3
1.14e-2
4.37e-3
1.14e-2
Table 7: Up-sampling capabilities Test results on Shallow- Water dataset. Metrics in MSE.
dataset
Shallow Water
Ma
Xrr
128
X1c
X
128 64
128
256
In-1
Out-1
In-1
Out-1
In-t
Out-1
In-1
Out-1
DeepONet
7.07e-3
9.02e-3
1.18e-2
1.66e-2
7.07e-3
9.02e-3
1.18e-2
1.66e-2
Wer 100%
FNO
6.75e-5
1.49e-4
7.54e-5
1.78e-4
6.75e-5
1.49e-4
6.91e-5
1.52e-4
regular grid
MP-PDE
2.66e-5
4.35e-4
4.80e-2
1.42c-2
2.66e-5
4.35e-4
4.73e-3
1.73e-3
DINo
4.12e-5
2.91c-3
5.77c-5
2.55e-3
4.12e-5
2.91c-3
6.04e-5
2.58c-3
CORAL
3.52c-6
4.99c-4
1.86c-5
5.32c-4
3.52e-6
4.99c-4
4.96c-6
4.99c-4
DeepONet
1.08--2
1.10e-2
2.49e-2
3.25e-2
2,49e-2
3.25e-2
2.49e-2
3.22e-2
irregular grid Wer 20%
MP-PDE
4.54e-3
1.48e-2
4.08e-3
1.30le-2
5.46e-3
1.74e-2
4.98e-3
1.43e-2
DINo
2.32e-3
5.18e-3
2.22e-3
4.80e-3
2.16e-3
4.64e-3
2.16e-3
4.64e-3
CORAL
1.36e-3
2.17e-3
1.24e-3
1.95e-3
1.21e-3
1.95e-3
1.21e-3
1.95e-3
DeepONet
1.02c-2
1.01e-2
1.57e-2
1.93e-2
1.57e-2
1.93c-2
1.57e-2
1.93e-2
irregular grid mr 5%
MP-PDE
5.36e-3
1.81c-2
5.53e-3
1.80e-2
4.33e-3
1.32c-2
5.48e-3
1.74c-2
DINo
1.25e-2
1.51e-2
1.39e-2
1.54e-2
1.39e-2
1.54e-2
1.39e-2
1.54e-2
CORAL
8.40e-3
1.25e-2
9.27e-3
1.15e-2
9.26e-3
1.16e-2
9.26e-3
1.16e-2
These tables show that CORAL remains competitive and robust on up-sampled inputs. Other baselines can also predict on denser grids, except for MP-PDE, which over-fitted the training grid.
C.2 Learning Dynamics on Different Grids
To extend our work, we propose to study how robust CORAL changes in grids. In our classical setting, we keep the same grid for all trajectories in the training set and evaluate it on a new grid for the test set. Instead, here, both in train and test sets, each trajectory has its own grid X, Thus, we evaluate CORAL's capability to generalize to grids. We present the results in Table 8. Overall, coordinate-baase based methods generalize better over grids compared to operator based and discrete methods like DeepONet and MP-PDE which show better equivalent performance when trained only on one grid. CORAL's performance increased when trained on different grids; one possible reason is that CORAL overfits the training grid used for all trajectories in our classical setting.
22

---

Table 8: Learning dynamics on different grids MSE.
Test results in the extrapolation setting. Metrics in
dataset +
Navier-Stokes
Shallow- Water
Ner & Lie
In-t
Out-t
In-t
Out-1
DeepONet
5.22E-1
5.00E-1
1.11E-2
1.12E-2
π 20% irregular grid
MP-PDE
6.11E-1
6.10E-1
6.80E-3
1.87E-2
DINo
1.30E-3
1.01E-2
4.12E-4
3,05E-3
CORAL
3.21E-4
3.03k-3
1.15E-4
7.75E-4
DeepONet MP-PDE
4.11E-1
4.38E-1
1.11E-2
1.12E-2
5% irregular grid
8.15E-1
1.10E-1
1.22E-2
4.29E-2
DINo
1.26E-3
1.04E-2
3.890-3
7.41E-3
CORAL
9.82E-4
9.71k-3
2.22e-3
4.89e-3
Table 9: Training time comparison Expressed in days (d) hours (h) on several datasets.
Model
Cylinder
Navier-Stoker
Shallow- Water
Elasticity
NACA
CORAL (INR)
6h
5d
4h
2d
CORAL (Process)
lh
4h
lh
1h
1h
NodeMLP
0.5h
GraphSAGE
1d
MP-PDE
7h
19h
21h
DINo
8h
2d
DeepONet
6h
5h
FNO
8h
6h
0.5h
0.5h
UNet
0.5h
0.5h
Geo-FNO
1.0h
1.0h
Factorized-FNO
1.0h
1.0h
C.3 Training Time
In Table 9, we present the training time for CORAL and different baselines for comparison. Since, the training of CORAL separated in 2 steps, show in line "INR" the training time for INR fitting and in line "Process" the second step to train the forecast model. We see that the longest part of the training procedure in CORAL is the fitting of the INR. MP-PDE is the slowest baseline train, due to the KNN graph creation. DeepONet and FNO are the fastest baselines to train because they only need a forward pass.
C.4 Inference Time
In this section, evaluate the inference time of CORAL and other baselines w.r.t. the input grid size. We study the impact of the training grid size (different models trained with 5%, 20% and 100% the grid) (Figure 6a) and the time needed for model trained (5%) on given grid to make computation on finer grid size resolution (cvaluation grid size) (Figure 6b).
On the graphs presented in Figure 6, we observe that except for the operator baselines, CORAL is also competitive in terms of inference time. MP-PDE inference time increases strongly when inference grid gets denser. The DINo model, which is the only to propose the same properties as CORAL, is much slower when both inference and training grid size evolve. This difference mainly explained by the number of steps needed to optimize DINo codes. Indeed, at test time DINO's adaptation requires 100x more optimization steps. MPPDE's computation slow due to the KNN graph creation and slower message passing. DeepONet and FNO are faster due to forward computation only. CORAL's encoding/decoding is relatively resolution-insensitive and performed parallel across all sensors. Process operates fixed dimension independent of the resolution. FNO is fast due FFT but
23

---

time subsampling
Inference up up-sampling rate.
30
10
307
105
(a) Average inference time (in seconds) of the implemented baselines to unroll trajectory until 40 on Navier-Stokes. For irregular grids, FNO is performed following linear interpolation.
(b) Inference time w.r.t. evaluation grid size (same models).
Figure 6
cannot be used on irregular grids. For these experiments, we used linear interpolation which slowed the inference time.
C.5 Propagation Errors Through Time
In Figures 7a to 7c, we show the evolution of errors the extrapolation horizon evolves. First, observe that all baselines propagate error through time, since the trajectories are computed using an auto-regressive approach. Except for the 100%. DeepONet had difficulties to handle the dynamic. has on all settings the highest error. Then, we observe that for MP-PDE and FNO, the error increases quickly at the beginning of the trajectories. This means that these two models propagating error. Finally, both DINo and CORAL have slower increase of the error during In-t and Out-I periods. However, clearly on the graphs that DINo has more difficulties than CORAL to make predictions out-range. Indeed, while CORAL's error augmentation remains constant as long as the time evolves, DINo has clear increase.
rapidly
C.6 Benchmarking INRs for CORAL
We provide some additional experiments for dynamics modeling with CORAL, but with diffrents INRs: MFN (Fathony et al., 2021), BACON (Lindell et al., 2022) and FourierFeatures (Tancik et al., 2020). Experiments have been done on Navier-Stokes on irregular grids sampled from grids of size 128 x 128. All training trajectories share the same grid and are evaluated trajectories. Results are reported in Table 10. Note that we used the same learning hyper-parameters for the baselines than those used for SIREN in CORAL. SIREN seems to produce the best codes for dynamics modeling, both for in-range and out-range prediction.
new grid for test
Table 10: CORAL results with different INRs. Test results in the extrapolation setting on Navier- Stokes dataset. Metries in MSE.
Xx, 4 Xic
INR
In-t
Out-1
SIREN
5.76e-4
2.57e-3
= 20% irregular grid
MFN
2.21e-3
5.17e-3
BACON
2.90e-2
3.32e-2
Fourier
1.70e-3
5.67e-3
SIREN
1.81e-3
4.15e-3
क 5% irregular grid
MFN
9.97e-1
9.58e-1
BACON
1.06
8.06e-1
FourierFeaturee
3.60e-1
3.62e-1
24

---

Errors WTt time
nppple
Dreponet
20"4
(a) Evolution of errors over time and across test samples for a model trained on 100% of the grid.
Errors time
coral
dino
mppde
deeponet
ste11
30"
30-
(b) Evolution of errors time and test samples for model trained on 20% of the grid.
Errors wrt time
mppdd
deeponet
10
(c) Evolution of errors over time and across test samples for a model trained on 5% of the grid.
Figure Errors along given trajectory.
25

---

C.7 Impact of 2nd order meta-learning
We provide in Figure the evolution of the reconstruction error through the training epochs for Navier-Stokes with first-order and second-order meta-learning. The first order method is able to correctly train until falls into an unstable regime for the common parameters. The second order method is much more stable and achieves 10x reduction in MSF
INR training loss with 1st order meta-learning.
INR training loss with 2nd order meta-leaming.
X
x
x--
Figure 8: Training of the modulated INR Comparison on Navier-Stokes over three independent runs of first order and second order meta- meta-learning. We use the same number of inner-steps.
C.8 Key hyper parameter analysis
Table 11 presents hyperparameter study reconstruction and forecasting tasks for Navier-Stokes dataset. Three ayperparameters-innit weight wo, latent dimension d2, and width-are varied to observe their impact. We notice that:
* wtp = 30 slightly improves the reconstruction on the test set.
d = 64 yields a better forecasting In-t performance.
width 256 significantly improves the model's 's performance across nearly all metrics.
Table 11: Hyper parameter study Reconstruction and forecasting results on Navier-Stokes dataset.
Metrics in MSE. Reconstruction are reported on the Train (In-f) and on the Test (In-1 + Out-f).
Reconstruction
Forecasting
Param |
Value
Train
Test
In-t
Out-1
3,62c-5
6,86c-5
2,78c-4
1,88c-3
W/
30
3,66e-5
5,85e-5
4,03e-4
2,28e-3
64
3,94e-5
1.1le-4
1,22e-4
1,42e-3
d;
256
2,73e-5
8,03e-5
1,63e-4
2,12c-3
64
1,50e-4
2,87e-4
2.84e-4
2,39e-3
width
256
1,60e-5
6,4le-5
1,23e
2,04e-3
CORAL bascline
1.05e-4
1.21e-4
1.86c-4
1.02e-3
D Supplementary results for geometry-aware inference
D.I Inverse Design for NACA-airfoil
Once trained on NACA-Euler, CORAL can be used for the inverse design of NACA airfoil. We consider an airfoil's shape parameterized by seven spline nodes and wish to minimize drag and maximize lift. We optimize the design parameters in an end-to-end manner. The spline nodes create
26

---

the input mesh, which CORAL maps to the output velocity field. This velocity field is integrated to compute the drag and the lift, and the loss objective the squared drag over lift ratio. As can be seen in Figure 9, iterative optimization results in an asymmetric airfoil shape, enhancing progressively the lift coefficient in line with physical expectations. At the end of the optimization we reach a drag value of 0.042 and lift value of 0.322.
(a) Step = 0
(b) Step 1000
(c) Step 3000
(d) Step 5000
Figure Design optimization of a NACo-Airfol.
E
Qualitative results
In this section, we show different visualization of the predictions made by CORAL on the three considered tasks in this paper.
E.1 Initial Value Problem
We provide in Figure 10 and Figure and Airfoil.
visualizations of the inferred values of CORAL on Cylinder
E.2 Dynamics modeling
We provide in Figure 13 and Figure 12 visualization of the predicted trajectories of CORAL Navier-Stokes and Shallow- Water.
27

---

Initial
175
0.75
150
0.00
1.0
0.59
25
0.2
5.0
1.5
0.09
1.5
-0.5
Pred pressure
Pred VX
Prod Vy
0.4
1.0
0.4
0.4
125
0.3
5.5
100
0.2
0.
0.2
02
0.00
-225
0.1
-050
0.00
-075
-0.25
155
0.5
1.5
0.5
1.5
True presture
True Vx
True Vy
0.75
050
10
025
0.00
8.0
-05
0.0
0.0
0.5
1.
15
0.0
0.5
1.0
15
0.0
0.5
1.0
1.5
Figure 10: CORAL prediction on Cylinder
Inttal pressure
deneity
Initiel
Intiaa Wy
2.00
3.00
0.28 0.55
0.32
0.50
-2.
0.92
6:25
0.50
0.55
3.00
pressure
-6
6.54
0.34
0.53
5.23
8.50
3.00
5.03
5/3
dendfy
VX
0.24
0.22
0.22
2 0.50
-1.0
-302
-6.22
4022
104
-300
6.24
6.24
Figure 11: CORAL prediction on Airfoil
28

---

Subsampling Obscracion grid
traeetory t
rate
t=0
Ogto E
20%
== 10%
Ground truth
Figure 12: Prediction MSE per frame for CORAL on Kavier-Stoker with its corresponding training grid X' Each row corresponds different sampling rate and the last row is the ground truth. The predicted trajectory predicted from to t T'. In our setting, 19 and T 39.
Subsampling rate
Proficted trajectory
grid
b-]
Inrt
T-T
Outi
1-
35G
20%
100%
Figure 13: Prediction MSE per frame for CORAL on Shallow- Water with its corresponding training grid X' Each row corresponds different sampling rate and the last is the ground truth. The predicted trajectory is predicted from to / T". In our setting, T = 19 and T" 39.
F.3 Geometry-aware inference
We provide in Figure 14, Figure 15, Figure 16 visualization of the predicted values of CORAL on Pipe and Flasticity.
29

---

Input mach
RIME men
Iosu medh
0.504
0.522
0.90
6.500
2.84
6.00
0.44
3.V%4
0.554
14
0.52
38
1.00
1.54
0466
24
32
Prodiction
Frod ciio
Prodiction
0.504
0.544
8.300
12
0332
C50E
5.20
38
38
6ASE
46
5.94
1.00
Figure 14: CORAL predictions NACA-Euler
Inuu mesh
Inpor mesh
Inbur mesh
877
64
825
08
5.7
635
1.4
0.5-
220
02-
62
Predition
Predciio
Predctioon
0.28
8.7
625
8.5
8.5
4us
88-
6.4
635
620
055
-000
630
0.0
64
405
625
6.0
52-
0.00
700
Figure 15: CORAL predictions on Pipe
30

---

IOGT men
FINE mech
IONIT mess
8.4
0.2
6.5
66
66
Predict
Prediction
Predict
33-
0.4
54
82
1.
Figure 16: CORAL predictions Elasticity
31


Reviews:
[{"review_id": "uax9ldLn8d", "reviewer": "Reviewer_A6MB", "summary": "In the paper, the author proposed to use neural field in operator learning. In the CORAL model, it first encodes the input into some codes, apply an MLP on the codes, and then decode into the output function. Since the neural field can be continuous evaluated, the CORAL model can be applied to general geometries.", "questions": "What is the dimension/size of the latent code?\n\nOne of the major advantage of this CORAL is time extrapolation. Do the authors have some insight why neural representation does better on this task?\n\nIn the experiment, the authors compared CORAL with another neural representation based method DINO. How does CORAL compared to DINO in the model design? Does it also use encoder and decoder? It will be great to address DINO also in the related work section.\n\nIn neural representation, many recent work use context grid and voxels to enhance the representation. Is it possible to compensate CORAL with grid-based representation?", "limitations": "The paper did not address its limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper uses neural presentation in operator learning which is a very natural choice and worthy investigation. The proposed method show promising results on time extrapolation and geometry design problem.", "weaknesses": "The overall framework is quite simple. The encode-decode structure has been studied in PCA basis [1] and Fourier basis [2], which are also applicable for general geometries. Compared to these well-studied conventional basis, neural representation usually use a higher number of parameters, which in the end, make the encoding unstable. \n\nIn the CORAL framework, the latent code is obtained using gradient descent. However it's possible to get very different latent code given the same input. Would such non-uniqueness cause any issue?\n\n[1] Bhattacharya, Kaushik, et al. \"Model reduction and neural networks for parametric PDEs.\" The SMAI journal of computational mathematics 7 (2021): 121-157.\n[2] Fanaskov, Vladimir, and Ivan Oseledets. \"Spectral neural operators.\" arXiv preprint arXiv:2205.10573 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and have addressed the raised concerns below.\n\n## CORAL framework \n> The overall framework is quite simple. \n\n CORAL adopts a classical encode-process-decode paradigm for learning operators. However, training an auto-decoder presents challenges, necessitating robust regularization techniques. The principal strength of CORAL lies in its auto-decoding mechanism, which is stabilized through careful optimization design: (i) CORAL encodes fields within a compact, low-dimensional code space, and (ii) leverages meta-learning optimization to adapt codes within a limited number of steps (3 in the conducted experiments). This strategic approach averts overfitting and ensures the development of a smooth, low-dimensional code manifold. We performed additional experiments to illustrate the stability of the learned encoding. Results provided in Fig. 1, in the pdf rebuttal page, show that this is indeed extremely stable.\n\n> In the CORAL framework, the latent code is obtained using gradient descent. However it's possible to get very different latent code given the same input. Would such non-uniqueness cause any issue?\n\nThis is like for any non convex optimization problem, where the resulting solution from the INR method could correspond to different local optima based on the initial conditions and the optimization path taken.\nIn all of our conducted experiments, one makes use of a shared encoder for the fields associated with a particular PDE along with consistent initialization and optimization techniques. As a result, for a given input, a unique solution code is obtained, and similar inputs tend to yield close  codes within the latent space. We provide the inference pseudo-code in Algorithm 4 in Appendix B.1.3.\n\n> What is the dimension/size of the latent code?\n\nThe dimension of the latent code is detailed in Appendix B.1.4 in Tables 4 and 5.  We use 256 for *Shallow-Water* and 128 for all the other datasets.\n\n> One of the major advantage of this CORAL is time extrapolation. Do the authors have some insight why neural representation does better on this task?\n\nAcross all models, time extrapolation is realized through an auto-regressive framework. In contrast to models like DeepONet, FNO, and MP-PDE, CORAL operates within a reduced representation space. DINO also operates in a reduced space, but we hypothesize that the CORAL optimization methodology contributes to the smoothing of the modulation space, thereby facilitating the training of the NeuralODE component. Within CORAL the dynamics' representation space is deliberately constrained, leading to a concentration of codes within a compact, low-dimensional manifold. This design choice enables comprehensive exploration of potential trajectories during training of the NeuralODE, resulting in enhanced generalization capabilities beyond the training horizon.\n\n## CORAL vs DINO\n> How does CORAL compared to DINO in the model design?\n\n DINO also operates within an encode-process-decode framework. However, our model diverges from DINO in several aspects encompassing design, training, and objectives. In terms of the objective, while DINO is tailored for dynamic spatio-temporal forecasting, requiring inputs and outputs to share the same space, CORAL functions as an operator capable of mapping input functional spaces to distinct functional spaces. This flexibility is demonstrated experimentally in Section 4.1 Initial Value Problem and Section 4.3 Geometric Design.\nRegarding design and training, our model exhibits key variations driven by preliminary trials. DINO employs a Multiplicative Fourier Network as its INR backbone, whereas CORAL relies on a SIREN architecture. Moreover, the optimization approach in CORAL is second-order, enabling code adaptation within a few gradient descent iterations (3 steps in contrast to the roughly 100 steps in DINO).\n\n## Possible extensions with context-grid / voxels\n> Is it possible to compensate CORAL with grid-based representation?\n\nWhile it's possible to enhance CORAL's training efficiency on 3D data using grid-based representations, such as those seen in [1] or [2], it remains a challenge to derive an effective grid-based representation optimized for dynamics modeling within the CORAL framework. An alternative grid or voxel-based enhancement involves segmenting the input space and fitting distinct Neural Fields for each subspace, effectively reducing parameter count while preserving finer representation nuances [3,4]. Incorporating these strategies into CORAL could involve creating a new latent representation by concatenating latent codes obtained from individual smaller Neural Fields.\n\nReferences:\n\n[1] Variable Bitrate Neural Fields. Takikawa et. al. SIGGRAPH 2022 \n\n[2] Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  Müller et al. SIGGRAPH 2022 \n\n[3] Neural Sparse Voxel Fields, Lingjie Liu et al. Neurips 2020. \n\n[4] MINER: Multiscale Implicit Neural Representations. Saragadam et al. ECCV 2022. \n\n\n## Limitations\n\n> The paper did not address its limitations.\n\nLimitations are discussed in Section 5 of the paper. Thanks to the reviewers' feedback, we will also discuss on the potential risks of overfitting.\n\n\n"}, {"review_id": "GSCw8y6n3h", "reviewer": "Reviewer_e6Go", "summary": "This work tried to introduce an implicit neural field-based framework for solving PDEs for different applications on general geometries. The method represents the input and output function spaces as an implicit neural representation. The method consists of a two-stage pipeline: first, the authors train two auto-decoders over the input and output function domains. Then, using paired data, they learn a mapping between the spatial latent code between domains. The paper shows results in initial value problem, dynamics, and predicting PDE solution from different geometry.\n\nI think the paper is tackling a very important problem - generalizing operator and neural field method to solving PDE with different geometry (in the boundary/shape). And I also like the approach to this problem as they leveraged neural fields techniques to achieve discretization-free effect and operator techniques to allow relatively fast influence time. However, the presentation of the paper confuses me in several places (see weakness section) and that holds me back from full acceptance in the first read. ", "questions": "See weakness section.", "limitations": "Section 5 discussed limitation", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is discretization free. It doesn't require the user to discretize the domain, which can have effect on the simulation quality. This method only need to run optimization with auto decoder, which is capable of taking partial observation and predict a continuous function that satisfies these observations.\n- This method has shown advantages compared to many operator methods in that it doesn't require operations such as Fourier transformation which requires regular samples in the geometry. This advantages the method to potentially work with different domain geometry, as well as different observation density.", "weaknesses": "1. Can this method be used for Inverse design? The experiment named \"geometry design\" is a bit misleading as it hints at the designing the geometry (the domain). But to the best of my reading, the geometric design experiment is set-up to predict PDE solution from different geometric, rather than designing the domain geometry. It would be great if the authors can add a note on this in the supplementary.\n\n2. It's not clear to me how does the method actually generalize to different geometry - I might miss something here. For example, how does the method encode where the boundary of the geometry is? It occurs to me that all functions are defined in the ambient space of physics domain. This makes it unclear to me how does the effect of domain boundary on the solution gets encoded. For example, if we have a bunch of samples (x_i, u_0(x_i)) that indicates the initial condition of a IVP problem, but we set two different boundary condition -  one circle these samples with a sphere and the other with a rectangular box. The current method seems to make the same prediction as the latent code is inferred from the set of samples samples (x_i, u_0(x_i)), but I can't believe that's the case in the real world - i.e. the boundary has no effect on the solution.\n\n3. Justification for the requirement of simulated data. The system still requires paired data for training similar to the neural operator, which is more supervision required by the PINN or neural fields methods (which only requires the PDE definition). I wonder what's the way to justify this additional data?\n\n4. Risk of overclaiming. I think \"general geometries\" has become very vaguely define throughout the paper. I'm under the impression that the method is able to solve equation with different boundary condition and different geometry of the boundary. But it's not clear to me how the current method is able to handle different boundary conditions, also it's not clear to me how does the method take into account of the different geometry of the simulation domain. For example, if we have the same initialization function, but different shape, would the latent code we obtained from test-time optimization of the auto-decoder be the same or different? If in test time, we want to change the boundary condition of this problem, how do I specify this? In general, I think the proper claim of the method is that it allows the neural operator method to work with irregular samples, but every trained model might still be restricted to solving the same set of problem (maybe the same boundary geometry) but with different initialization. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're thankful for the reviewer's helpful feedback and have addressed the raised concerns below.\n\n## Application to inverse design \n\n> Can this method be used for Inverse design? \n\nWe agree, this will be clarified in the final version. As you mention, the geometric design section in the core paper amounts to predicting PDE solutions based on diverse geometric settings. However, once trained the model could be used for \"true\" inverse design. This is  indicated in l.358 in the paper with preliminary experiments on the design of Airfoils shapes in Appendix D and Fig. 8.\n\n## Operator Learning setting \n\n> Justification for the requirement of simulated data.\n\nThese are two different settings targeting two different problems, considered separately in the literature. Ours is a pure data-based approach (same as for FNO, DeepONet, MP-PDE, etc), the assumption is that no prior explicit knowledge is available on the physical phenomenon (hence no PDE equation, no explicit Boundary Condition (BC), etc) and only data is available for training. The model is trained on a number of contexts (initial values etc.) and is due to generalize on different but similar contexts.\nPINNs aim at replacing traditional solvers for explicitly-known PDEs and is coined a data-free approach.  Moreover, a Vanilla PINN is trained for one single condition (initial value and boundary conditions) and does not generalize to different conditions.\n\n## Geometries and meshes\n\n> It's not clear to me how does the method actually generalize to different geometry.\n\nThis is a pure data-based approach where no prior knowledge of the PDE is incorporated into the framework. Therefore, the PDE conditions (initial or boundary) are not stated analytically and never used as such. Let's consider an Initial Value Problem (IVP) as exemplified by the cylinder scenario (Figure 1). When presented with two different obstacles, our method uses the initial condition on two distinct meshes, which include disparities at the obstacle boundary. This leads to a varying sampling for the INR encoding and one obtains a different code in each situation. The processor operating on the corresponding representation will output two different values. Our approach does not presume that the boundary has no influence on the solution. Instead, it captures these distinctions through the mesh differentiation and as result in the INR codes. This entails that for each situation, a unique code is acquired due to the employment of distinct mesh configurations at the obstacle boundary. Regarding the question of whether a model trained on a certain geometry distribution can generalize to entirely different geometries (e.g., circles vs. squares), we would like to acknowledge that this aspect was not explored within our current investigation. It remains an interesting avenue for future exploration.\n\n> Risk of overclaiming. \n\nWe will clarify the claim in the paper. As indicated above, the boundary conditions are provided by the mesh definition and the corresponding sampling for learning the INR. Consider for example the airfoil (NACA-Euler) experiment in section 4.3 and Fig 1, 13, (the same holds for Fig. 14, 15). The model is trained on multiple shapes (BC) and evaluated at test time on similar but different airfoil shapes.  In this sense the model is able to handle different geometries at test time. For any new example (airfoil shape) the learned code will be different thus encoding the geometry. Note that this is also what allows us to perform inverse design (Appendix D.1)."}, {"review_id": "Wuel7Y4PdA", "reviewer": "Reviewer_5RAL", "summary": "This paper proposes a method for solving PDEs with neural networks in continuous space (and optionally time as well). The authors leverage the success of coordinate-based neural networks (or implicit neural representations – INRs) and formulate the problem as an operator learning one, akin to the Neural Operator family, bypassing the need for discretisation. In particular, the method is based on a typical encode-process-decode strategy: \n\n1. *Encode*: the input condition of the PDE (i.e. the initial value in an initial value problem – IVP) is projected to the input latent vector (via gradient-based optimisation), which is then transformed (via an MLP) to a latent modulation of the input INR (SIREN), such that the INR is fitted to the input condition (auto-decoding). \n2. *Process*: the input latent vector is transformed (via an MLP when projecting to a single output or via a neural ODE when solving over continuous time) to the output one. \n3. *Decode*: the output latent vector is transformed (via an MLP) to a latent modulation of the output INR, which encodes the output signal and can then be queried in arbitrary spatial positions. To promote training stability, the authors employ a two-step process: First the parameters of the input/output INRs are fitted via a meta-learning strategy, similar to Dupont et al., ICML’22, using a shared INR across all input conditions and learning both the parameters of the INR and the latent vectors simultaneously and efficiently. Then, the parameters of the encode-process-decode pipeline are learnt by doing optimisation in the latent space.\n\nThe method is evaluated on several PDE tasks, showing competitive results against other continuous neural solvers.\n", "questions": "The major questions that I would like to be clarified and discussed are included in the \"Weaknesses\" section. Below, I enlist a few extra minor comments:\n\n**Minor comments**:\n\n-\tCould the authors explain their inference-time findings? Why is CORAL faster than DINO and MPPDE and slower than DeepONet and FNO? Why does CORAL scale batter (almost constant time regardless of the resolution)? Why didn’t the authors include results for FNO across all resolution setups?\n-  Perhaps it would have been beneficial to also test on more challenging setups such as the turbulent regime of Navier-Stokes (potentially to identify limitations and failure cases).\n- A recent paper that improves upon prior work and might be useful to also include as a baseline is the “Clifford neural layers for PDE modelling”, Brandstetter et al., ICLR’23.\n-\tL72: “infinite-dimensional functions“: This should be rephrased - maybe mappings between functions (infinite-dimensional vectors)?\n-\tL76: Regarding DeepONet, if I am not mistaken it is required to have the same observation grid only for training and not for testing.\n-\tL265: RK4 – maybe explain that you are referring to fourth-order Runge-Kutta. \n", "limitations": "The authors have included a section discussing the limitations of their work and were upfront about them. A few things remain to be clarified (e.g. time and tuning required for training, generalisation across input conditions, and more challenging experimental PDE setups). No foreseeable negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-\t**Clarity and presentation quality**. The paper is well-presented, has a good flow and guides the reader smoothly throughout the important ideas. Also, it is mathematically clean and concise. Finally, it is mostly well-contextualised with the relevant literature.\n-\t**Simplicity and Scalability (inference)**. Although the method is less theoretically motivated compared to Neural Operators, it builds on top of existing tools in the INR literature, making it easy to implement. In addition, forecasting happens in a compressed version of the physical process (a latent vector), which improves scalability.\n-\t**Novelty and Significance**. The paper proposes a new framework for continuous simulations, departing from the Neural Operator family, advancing the field further, in terms of efficiency and performance. The claims are empirically evaluated on a broad spectrum of PDE tasks (from IVPs and dynamics modelling to geometric design and optimisation) showcasing competitive generalisation performance across input conditions, resolutions and time horizons (for trajectory unrolls).\n", "weaknesses": "- **Connections with Dupont et al., ICML’22**. First off, it is important to acknowledge that the method, from a technical point of view, is largely based on the “functa” concept of Dupont et al., ICML’22. The authors do cite this work, but not as prominently as one would expect. Although the aforementioned paper focuses more abstractly on function representation learning, and less on PDE solvers, the current work is a natural follow-up, so I believe that the connections between the two should be stated more clearly.\n-\t**Efficiency and ease of training**. The training process might be slow since it happens in two steps. Although the second step, might be relatively fast since it happens in the latent space, the first one requires fitting INRs to all possible inputs and outputs in the training set. The meta-learning strategy might potentially alleviate this problem, but the authors have not discussed this matter. Could the authors provide more details on this step? How long does it take? Does it require extensive hyperparameter tuning? How sensitive are the results to different hyperparameters? \n-\t**Potential risk of overfitting (w.r.t. input/output signals not coordinates)**: Experimentally, it seems from the paper that the method generalises well across different input conditions, including the particularly challenging problem of trajectory unrolls. Let me explain why I am a bit surprised by this:\n\n  - I am puzzled about the inductive biases of the encoding process (Eq. (1)). In particular, it is unclear what are the hidden assumptions regarding the input/output signals. Think of a CNN encoder for example – in that case, we would silently assume signal stationarity and shift invariance (due to the application of the same filter across different regions), locality, etc. On the other hand, the corresponding assumptions of the auto-decoder are unclear, even in an intuitive manner. I would appreciate it if the authors could comment on this, and it might be also useful for future readers.\n   - Given the above, I am wondering how well the encoding process, and more importantly the entire pipeline, generalises across different input signals. In particular, in case the encoding process overfits, then small errors in the predictions of the latent codes (especially when unrolling trajectories where errors accumulate), might result in regions of the latent space “unknown” to the decoder, and therefore to unpredictable outcomes. To clarify this, it might be useful to discuss the distributions of the input conditions used. Are they challenging enough? How important is the Z-score normalisation (Appendix B.1.2.)?\n   - My concern about this also comes from the fact that only a latent-space objective is minimised, which in principle does not rule out the possibility of small errors in the latent space to translate to large errors in the signal space (e.g. assume that the INR decoder has a large Lipschitz constant). \n-\t**Empirical evaluation and comparison against relevant methods**: I think there is room for improvement in the experimental section (extra discussion and comparisons) in order to make the claims more convincing. In particular, \n  - It would be helpful to also include discrete neural solver baselines (e.g. based on CNNs, as in Stachenfeld, ICLR’22, or mesh GNNs, as in Pfaff et al., ICLR’21).\n  - In addition, it would be nice to see a more thorough discussion regarding the comparison between continuous neural solvers (MPPDE, Neural Operators etc.). It is unclear to me why and when the current method performs better. For example, intuitively I don’t understand why MPPDE is dependent/overfits on the training grid. Have the authors considered comparing against the Graph/Multipole Graph Neural Operator methods (NeurIPS’20)? Does CORAL have any stronger inductive biases compared to Neural Operators? Could the authors elaborate on the above?\n  - Regarding the dynamics modelling experiments, I have a few questions: (1) Have the authors compared against autoregressive one-step predictors (e.g. as in the MPPDE paper, or in Sanchez-Gonzalez et al., ICML’20)?  Obviously, the NeuralODE formulation allows for evaluation across arbitrary timestamps, but I am wondering if there is any disadvantage in this approach. (2) Have the authors considered comparing to discrete autoregressive solvers, such as the LE-PDE paper (NeurIPS’22)? (3) In certain papers (e.g. MPPDE and Sanchez-Gonzalez et al., ICML’20) the authors use a noise trick to stabilise trajectory unrolls and mitigate error accumulation. I am curious if the authors used it here as well, and if not why?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the reviewer's extensive and detailed feedback and have effectively addressed the raised concerns as follows.\n\n>Connections with Dupont et al., ICML’22.\n\nThe work by Dupont indeed inspired the INR part of our contribution, and will be better acknowledged in the final version.\n\n> Efficiency and ease of training.\n\nCertainly, training the INR is the most time-consuming step while time integration by the solver is fast. We provide additional details on the training time in Table 2 of the pdf rebuttal page. Note that accelerating INR training is an active research topic. Besides, we found out that training is relatively robust to the choice of hyperparameters. Key parameters include SIREN frequency $\\omega_0$, code size $d_z$, and SIREN depth.\n\n> Potential risk of overfitting\n\nThis is a great question which deserves further investigation. For now we can only answer intuitively. INRs encode global image representations in a spectral form [1] so that the role of the code is to modulate this frequential representation. [2] perform some experiments showing that each dimension of the code induces some frequential pattern in the reconstructed image. \nIn a simplifying view, we could think of the code features corresponding to coefficients in a spectral basis ([2]). Then in CORAL the solver can be thought of as operating on this coordinate space. If the flow is smooth and regular the associated trajectory could be more easily learned.\n\nHow is this achieved in CORAL ? CORAL is constrained (i) to operate in small dimensional code spaces, and (ii) optimization with meta learning is constrained for each image, to adapt the codes from a shared initial value to a final value with only a few (3 in the experiments) gradient steps. This avoids overfitting and constrains the code manifold to be smooth and low dimensional. For example in a forecasting experiment, two successive images will have close representations in the code space, and the code trajectories are deemed to be regular. This strategy enforces the robustness of the trajectories to small variations. This helps training the neural ODE and could explain the good behavior in extrapolation.\n\n[1] A Structured Dictionary Perspective on Implicit Neural Representations. Yüce et al. CVPR 2022\n\n[2] From data to functa: Your data point is a function and you can treat it like one. Dupont et al. ICML 2022.\n\n## Baselines\n\n> Inclusion of discrete neural solver baselines.\n\nWe would like to clarify that we did experiments with discrete solvers. The graph based encode-process-decode architecture was proposed in Pfaff 2021 and re-used within an updated version in MP-PDE, which is one of our baselines. These methods are discrete both for the spatial and time dimensions. We found out through preliminary experiments that MP-PDE was SOTA among GNN solvers which motivated our choice. CNNs weren't compared due to grid limitations.\n\n> Thorough discussion on continuous neural solvers comparison.\n\nMP-PDE, FNO and CORAL all perform well in many settings, and have their respective advantages, and weaknesses. Our point is that CORAL is a general framework that allows dealing (i) with multiple tasks, (ii) in a mesh-free setting, (iii) in a reduced latent space and generalizes well on multiple conditions. MP-PDE's mesh dependence hinders generalization in drastic mesh changes or sparse sub-sampling. As for FNO, being based on FFT it is limited to regular grids. Graph-Multipole was unstable and excluded from the baselines.\n\n> (1) Comparison against autoregressive one-step predictors.\n\nYes, all the experiments involving dynamics modeling are performed with autoregressive one-step predictors for all the models.\n\n> (2) Comparison to discrete autoregressive solvers.\n\nMaybe this was not clear enough. There are two dynamics involved for forecasting. One is auto-regressive, $u_t = g(u_{t-1})$ and the second one is how we proceed from $u_{t-1}$ to $u_{t}$. The solver is involved for the latter computation: for a 4-step method like RK4 this requires 4 iterations and for a simple Euler forward, this would require 1 iteration only. Said otherwise, NODE generalizes the basic AR formulation. \n\n> (3) Noise trick for trajectory unrolls.\n\nOur NeuralODE training employed exponential sampling, initially using ground truth codes as inputs. We didn't use additional tricks. \n\n## Minor comments\n\n> Inference-time analysis.\n\nAt test time, DINO's adaptation requires 100x more optimization steps. MPPDE's computation is slow due to the KNN graph creation and slower message passing. DeepONet and FNO are faster due to forward computation only. CORAL's encoding/decoding is relatively resolution-insensitive and performed in parallel across all sensors.  Process operates on a fixed dimension independent of the resolution.\n\nFNO is fast due to FFT but cannot used on irregular grids, hence its previous omission. We included additional lin. interpo. + FNO results in Table 1 with corresponding inference time in Figure 2 of the PDF rebuttal page.\n\n> Turbulent regime of NS.\n\nYes, we agree but this deserves a specific treatment and probably alternative models. Initial tests with large Reynolds numbers have shown an order of magnitude decrease for all the models.\n\n> Clifford neural layers for PDE modeling.\n\nThis paper focuses on modeling the interactions between the physical fields and the variables of the model when they are intrinsically connected. We focused solely on model comparisons without Clifford algebra. But we agree that this would be an interesting extension for several PDE families.\n\n> L76:  DeepONet's observation grid.\n\nAt test time, DeepONet requires the same number of sensors in their input to the Branch Net, preferably matching the training sensors. However, DeepONet can infer fields at any query coordinate through its trunk-net, akin to an INR.\n\n> L72: Infinite-dimensional rephrasing and L265: Clarification of RK4.\n\nAdjusted accordingly; your feedback is appreciated."}, {"review_id": "2NctVZVfYu", "reviewer": "Reviewer_K5kw", "summary": "This paper creatively employs Implicit Neural Representations (INR) for operator learning on irregular domains. This novel method benefits from INR's capability to adaptively handle irregular grid distributions or irregular geometric areas, making learning the mapping from input to output in the INR's latent space quite natural. The primary experiments of the paper are on initial value problems, where it shows a significant improvement. However, a notable challenge is that the introduction of INR necessitates bi-level optimization, which potentially introduces instability and difficulty into the training process.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.The proposed method is highly innovative.\n2.The quality of writing is excellent, and the paper is easy to read.\n3. As an empirical paper, the experimental results surpass the baselines.", "weaknesses": "1.The paper lacks research on related work. Besides Geo-FNO and FFNO, transformer-based methods can also naturally handle irregular geometric areas. The authors should at least mention these works. Here are some references for consideration [1,2,3].\n2. Operator learning tasks are not computationally demanding, so the paper should explore the method's stability and report error bars (at least for some cases).\n3. More explanation or supplementation is needed concerning the stability of the bi-level optimization, the difficulty of training, and why second-order meta-learning is adopted.\n\nOverall, this is a good paper. If the authors can clarify the issues mentioned, I am inclined to accept it.\n\nReference\n1. Transformer for Partial Differential Equations' Operator Learning (https://arxiv.org/abs/2205.13671)\n2. GNOT: A General Neural Operator Transformer for Operator Learning (https://arxiv.org/abs/2302.14376)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and have addressed the raised concerns below.\n\n## Transformer-based methods\n\n> [...] Besides Geo-FNO and FFNO, transformer-based methods can also naturally handle irregular geometric areas. The authors should at least mention these works.\n\nWe thank the reviewer for the references, we will add a discussion on these\nrecent contributions and how they position w.r.t. other methods.\n\n## Stability\n\n> Operator learning tasks are not computationally demanding, so the paper\nshould explore the method’s stability and report error bars (at least for some\ncases).\n\nWe have provided error bars in Table 1 of the PDF rebuttal\npage for the *Navier-Stokes* dataset, for CORAL and for the baselines. We\ncompute the mean and standard deviation of the test MSE out of three runs with\ndifferent seeds. Each seed induces a different parameter initialization for all the\nmodels and a different pair of train and test grids $X_{tr}$ , $X_{te}$ for the subsampling\nexperiments.\nNote that the variance of all models increases when the subsampling ratio\ndecreases, since the independent runs sample a different training and test grids.  \n\nAdditionally, training times for CORAL have been provided in Table 2 of the PDF rebuttal page.\n\n## Optimization procedure\n\n> More explanation or supplementation is needed concerning the stability\nof the bi-level optimization, the difficulty of training, and why second-order\nmeta-learning is adopted.\n\nThe bi-level optimization that is referred throughout the paper as the inner-loop and outer-loop, is a crucial aspect for the success of our method. We show in Figure 1 of the PDF rebuttal page a comparison between 1st order and 2nd order meta-learning for training the modulated INRs.\n\nAs can be seen, the 1st order method struggles to find a good local optimum for the shared parameters of the modulated INR. In contrast, the 2nd order meta-learning is more stable and is able to converge consistently to similarly-performing local optima across different runs.\n\nThe intuition to stabilize the meta-learning training is to have a large learning rate for the inner-loop update (1e-2), and a much smaller one for the outer-loop (from 1e-4 to 1e-6). Increasing the batch size also helps stabilizing the training. Overall, this increases the number of epochs required to achieve a good reconstruction of the fields but favors a smooth training. "}]

Author rebuttal:
We thank all the reviewers for their comments and suggestions. We carefully answered all the questions raised by each reviewer. We have also added new experimental results following the suggestions of the reviewers concerning:

* Consolidated results with error bars on *Navier-Stokes* (Reviewer K5kw)
* Stability of the training of the modulated INR with a 1st order vs 2nd order meta-learning on *Navier-Stokes*. (Reviewer K5kw)
* Additional FNO + linear interpolation results with irregular grids on *Navier-Stokes*. (Reviewer 5RAL)
* Additional inference time on FNO + linear interpolation (Reviewer 5RAL)
* Additional information on the training time of CORAL (Reviewer 5RAL)

These new results are summarized in the PDF rebuttal page.


Meta-review: All reviewers liked the paper's approach to combine the advantages of neural field representation in the context of neural operators. While it would have been great to have better comparisons with modern U-Nets rather than the basic 2015 version that continues to remain a baseline in this domain, ultimately the paper's contribution has more to do with generalized geometries which neither FNO or U-Nets are good at handling.
Decision: Accept (poster)