Title: Operator Learning with Neural Fields: Tackling PDEs on General Geometries
Venue: NeurIPS 2023
Accepted: yes
Keywords: PDEs

Physics

Operator Learning

Deep Learning

Spatiotemporal
Abstract: Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neural networks are constrained to discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely on some form of discretization. In order to alleviate such limitations, we present CORAL, a new method that leverages coordinate-based networks for solving PDEs on general geometries. CORAL is designed to remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends to diverse problem domains, including PDE solving, spatio-temporal forecasting, and inverse problems like geometric design. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-art models.

Operator Learning with Neural Fields: Tackling PDEs on General Geometries

Abstract

Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neural networks are constrained to discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely on some form of discretization. In order to alleviate such limitations, we present CORAL, a new method that leverages coordinate-based networks for solving PDEs on general geometries. CORAL is designed to remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends to diverse problem domains, including PDE solving, spatio-temporal forecasting, and geometry-aware inference. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-art models.

1 Introduction

Modeling physics dynamics entails learning mappings between function spaces, a crucial step in formulating and solving partial differential equations (PDEs). In the classical approach, PDEs are derived from first principles, and differential operators are utilized to map vector fields across the variables involved in the problem. To solve these equations, numerical methods like finite elements, finite volumes, or spectral techniques are employed, requiring the discretization of spatial and temporal components of the differential operators (Morton & Mayers, 2005; Olver, 2014).

Building on successes in computer vision and natural language processing (Krizhevsky et al., 2017; He et al., 2016; Dosovitskiy et al., 2021; Vaswani et al., 2017), deep learning models have recently gained attention in physical modeling. They have been applied to various scenarios, such as solving PDEs (Cai et al., 2021), forecasting spatio-temporal dynamics (de B´ ezenac et al., 2019), and addressing inverse problems (Allen et al., 2022). Initially, neural network architectures with spatial inductive biases like ConvNets (Long et al., 2018; Ibrahim et al., 2022) for regular grids or GNNs (Pfaff et al., 2021; Brandstetter et al., 2022b) for irregular meshes were explored. However, these models are limited to specific mesh points and face challenges in generalizing to new topologies. The recent trend of neural operators addresses these limitations by modeling mappings between functions, which can be seen as infinite-dimensional vectors. Popular models like DeepONet (Lu et al., 2022) and Fourier Neural Operators (FNO) (Li et al., 2022b) have been applied in various domains. However, they still have design rigidity, relying on fixed grids during training and inference, which limits their use in real-world applications involving irregular sampling grids or new geometries. A variant of

FNO tailored for more general geometries is presented in (Li et al., 2022a), but it focuses on design tasks.

To overcome these limitations, there is a need for flexible approaches that can handle diverse geometries, metric spaces, irregular sampling grids, and sparse measurements. We introduce CORAL, a COordinate-based model for opeRAtor Learning that addresses these challenges by leveraging implicit neural representations (INR). CORAL encodes functions into compact, low-dimensional latent spaces and infers mappings between function representations in the latent space. Unlike competing models that are often task-specific, CORAL is highly flexible and applicable to various problem domains. We showcase its versatility in PDE solving, spatio-temporal dynamics forecasting, and design problems.

Our contributions are summarized as follows:

CORAL can learn mappings between functions sampled on an irregular mesh and maintains consistent performance when applied to new grids not seen during training. This characteristic makes it well-suited for solving problems in domains characterized by complex geometries or non-uniform grids.

We highlight the versatility of CORAL by applying it to a range of representative physical modeling tasks, such as initial value problems (IVP), geometry-aware inference, dynamics modeling, and forecasting. Through extensive experiments on diverse datasets, we consistently demonstrate its state-of-the-art performance across various geometries, including convex and non-convex domains, as well as planar and spherical surfaces. This distinguishes CORAL from alternative models that are often confined to specific tasks.

CORAL is fast. Functions are represented using a compact latent code in CORAL, capturing the essential information necessary for different inference tasks in a condensed format. This enables fast inference within the compact representation space, whereas alternative methods often operate directly within a higher-dimensional representation of the function space.

2 Related Work

Mesh-based networks for physics. The initial attempts to learn physical dynamics primarily centered around convolutional neural networks (CNNs) and graph neural networks (GNNs). Both leverage discrete convolutions to extract relevant information from a given node's neighborhood Hamilton (2020). CNNs expect inputs and outputs to be on regular grid. Their adaptation to irregular data through interpolation (Chae et al., 2021) is limited to simple meshes. GNNs work on irregular meshes (Hamilton et al., 2017; Veliˇ ckovi´ c et al., 2018; Pfaff et al., 2021) and have been used e.g. for dynamics modeling (Brandstetter et al., 2022b) or design optimization (Allen et al., 2022). They typically select nearest neighbors within a small radius, which can introduce biases towards the type of meshes seen during training. In Section 4.2, we show that this bias can hinder their ability to generalize to meshes with different node locations or levels of sparsity. Additionally, they require significantly more memory resources than plain CNNs to store nodes' neighborhoods, which limits their deployment for complex meshes.

Operator learning. Operator learning is a burgeoning field in deep learning for physics that focuses on learning mappings between infinite-dimensional functions. Two prominent approaches are DeepONet (Lu et al., 2021) and Fourier Neural Operator (FNO; Li et al., 2021). DeepONet can query any coordinate in the domain for a value of the output function. However, the input function must be observed on a set of predefined locations, requiring the same observation grid for all observations, for training and testing. FNO is an instance of neural operators (Kovachki et al., 2021), a family of approaches that integrate kernels over the spatial domain. Since this operation can be expensive, FNO addresses the problem by employing the fast Fourier transform (FFT) to transform the inputs into the spectral domain. As a consequence it cannot be used with irregular grids. Li et al. (2022a) introduce an FNO extension to handle more flexible geometries, but it is tailored for design problems. To summarize, despite promising for several applications, current operator approaches still face limitations to extrapolate to new geometries; they do not adapt to changing observation grids or are limited to fixed observation locations. Recently, Li et al. (2023); Hao et al. (2023) explored transformer-based architectures as an alternative approach.

Initial Value Problem

(a)

Cylinder

Dynamics modeling

(b)

Navier-Stokes

(c)

Shallow-Water

Geometry-aware inference

(d)

Elasticity

(e)

NACA-Euler

Momentum

O

Wo→WT

Vorticity

Vorticity

Wt→→Wt+1

Wt→Wt+1

x H→ v

Stress

bx

Spatial INRs. Spatial INRs are a class of coordinate-based neural networks that model data as the realization of an implicit function of a spatial location x ∈ Ω ↦→ f θ ( x ) (Tancik et al., 2020; Sitzmann et al., 2020b; Fathony et al., 2021; Lindell et al., 2022). An INR can be queried at any location, but encodes only one data sample or function. Previous works use meta-learning (Tancik et al., 2021; Sitzmann et al., 2020a), auto-encoders (Chen & Zhang, 2019; Mescheder et al., 2019), or modulation (Park et al., 2019; Dupont et al., 2022) to address this limitation by enabling an INR to decode various functions using per-sample parameters. INRs have started to gain traction in physics, where they have been successfully applied to spatio-temporal forecasting (Yin et al., 2022) and reduced-order modeling (Chen et al., 2022). The former work is probably the closest to ours but it is designed for forecasting and cannot handle the range of tasks that CORAL can address. Moreover, its computational cost is significantly higher than CORAL's, which limits its application in real-world problems. The work by Chen et al. (2022) aims to inform the INR with known PDEs, similar to PINNs, whereas our approach is entirely data-driven and without physical prior.

3 The CORAL Framework

In this section, we present the CORAL framework, a novel approach that employs an encode-processdecode structure to achieve the mapping between continuous functions. We first introduce the model and then the training procedure.

3.1 Problem Description

Let Ω ⊂ R d be a bounded open set of spatial coordinates. We assume the existence of a mapping G ∗ from one infinite-dimensional space A ⊂ L 2 (Ω , R d a ) to another one U ⊂ L 2 (Ω , R d u ) , such that for any observed pairs ( a i , u i ) ∈ A × U , u i = G ∗ ( a i ) . We have a i ∼ ν a , u i ∼ ν u where ν a is a probability measure supported on A and ν u the pushforward measure of ν a by G ∗ . We seek to approximate this operator by an i.i.d. collection of point-wise evaluations of input-output functions through a highly flexible formulation that can be adapted to multiple tasks. In this work, we target three different tasks as examples: · solving an initial value problem, i.e. mapping the initial condition u 0 . = x ↦→ u ( x, t = 0) to the solution at a predefined time u T . = x ↦→ u ( x, t = T ) , · modeling the dynamics of a physical system over time ( u t → u t + δt ) over a given forecasting horizon · or

prediction based on geometric configuration. At training time, we have access to n tr pairs of input and output functions ( a i , u i ) n tr i =1 evaluated over a free-form spatial grid X i . We denote a | X i = ( a ( x )) x ∈X i and u | X i = ( u ( x )) x ∈X i the vectors of the function values over the sample grid. In the context of the initial value and geometry-aware problems, every sample is observed on a specific grid X i . For dynamics modeling, we use a unique grid X tr for all the examples to train the model and another grid X te for testing.

obs.

space

input

function

output

function

output

space

predicted output

values on query

grid

encode

inputs

forecasted

code

3.2 Model

CORAL makes use of two modulated INRs, f θ a ,ϕ a and f θ u ,ϕ u , for respectively representing the input and output functions of an operator. While θ a and θ u denote shared INR parameters that contribute in representing all functions a i and u i , the modulation parameters ϕ a i and ϕ u i are specific to each function a i and u i . Given input/output INR functions representation, CORAL then learns a mapping between latent representations inferred from the two INRs' modulation spaces. The latent representations z a i , z u i are low dimensional embeddings, capturing within a compact code information from the INRs' parameters. They are used as inputs to hypernetworks h a and h u to compute the modulation parameters ϕ a i = h a ( z a i ) and ϕ u i = h u ( z u i ) . The weights of the input and output hypernetworks are respectively denoted w a and w u .

CORAL proceeds in three steps: encode , to project the input data into the latent space; process , to perform transformations in the latent space; and decode , to project the code back to the output function space. First, the input function a is encoded into the small input latent code z a using a spatial encoder e a : A ↦→ R d z . Next, a parameterized model g ψ : R d z ↦→ R d z is used to infer an output latent code. Depending on the target task, g ψ can be as simple as a plain MLP or more complex as for example a neural ODE solver (as detailed later). Finally, the processed latent code is decoded into a spatial function using a decoder ξ u : R d z ↦→U . The resulting CORAL operator then writes as G = ξ u ◦ g ψ ◦ e a , as shown in Figure 2. The three steps are detailed below.

Encode Given an input function a i and a learned shared parameter θ a , the encoding process provides a code z a i = e a ( a i ) . This code is computed by solving an inverse problem through a procedure known as auto-decoding , which proceeds as follows. We want to compress into a compact code z a i the information required for reconstructing the original field a i through the input INR, i.e.: ∀ x ∈ X i , f θ a ,ϕ a i ( x ) = ˜ a i ( x ) ≈ a i ( x ) with ϕ a i = h a ( z a i ) . See Figure 3a in Appendix B for details.

The approximate solution to this inverse problem is computed as the solution e a ( a i ) = z ( K ) a i of a gradient descent optimization:

where α is the inner loop learning rate, K the number of inner steps, and L µ i ( v, w ) = E x ∼ µ i [( v ( x ) -w ( x )) 2 ] for a measure µ i over Ω . Note that in practice, µ i is defined through the observation grid

X i , µ i ( · ) = ∑ x ∈X i δ x ( · ) where δ x ( · ) is the Dirac measure. Since we can query the INRs anywhere within the domain, we can hence freely encode functions without mesh constraints. This is the essential part of the architecture that enables us to feed data defined on different grids to the model. We show the encoding flow in Appendix B, Figure 4.

Process Once we obtain z a i , we can infer the latent output code ˆ z u i = g ψ ( z a i ) . For simplification, we consider first that g ψ is implemented through an MLP with parameters ψ . For dynamics modeling, in Section 4.2, we will detail why and how to make use of a Neural ODE solver for g ψ .

Decode We decode ˆ z u i with the output hypernetwork h u and modulated INR and denote ξ u the mapping that associates to code ˆ z u i the function f θ u , ˆ ϕ u i , where ˆ ϕ u i = h u (ˆ z u i ) . Since f θ u , ˆ ϕ u i is an INR, i.e. a function of spatial coordinates, it can be freely queried at any point within the domain. We thus have ∀ x ∈ Ω , ˆ u i ( x ) = ξ u (ˆ z u i )( x ) = f θ u , ˆ ϕ u i ( x ) . See Figure 3b in Appendix B for details.

During training, we will need to learn to reconstruct the input and output functions a i and u i . This requires training a mapping associating an input code to the corresponding input function ξ a : R d z ↦→A and a mapping associating a function to its code in the output space e u : U ↦→ R d z , even though they are not used during inference.

3.3 Practical implementation: decoding by INR Modulation

We choose SIREN (Sitzmann et al., 2020b) - a state-of-the-art coordinate-based network - as the INR backbone of our framework. SIREN is a neural network that uses sine activations with a specific initialization scheme (Appendix B).

where η 0 = x and ( η i ) i ≥ 1 are the hidden activations throughout the network. ω 0 ∈ R ∗ + is a hyperparameter that controls the frequency bandwidth of the network, W and b are the network weights and biases. We implement shift modulations (Perez et al., 2018) to have a small modulation space and reduce the computational cost of the overall architecture. This yields the modulated SIREN:

with shared parameters θ = ( W i , b i ) L i =0 and example associated modulations ϕ = ( ϕ i ) L -1 i =0 . We compute the modulations ϕ from z with a linear hypernetwork , i.e. for 0 ≤ i ≤ L -1 , ϕ i = V i z + c i . The weights V i and c i constitute the parameters of the hypernetwork w = ( V i , c i ) L -1 i =0 . This implementation is similar to that of Dupont et al. (2022), which use a modulated SIREN for representing their modalities.

3.4 Training

We implement a two-step training procedure that first learns the modulated INR parameters, before training the forecast model g ψ . It is very stable and much faster than end-to-end training while providing similar performance: once the input and output INRs have been fitted, the training of g ψ is performed in the small dimensional modulated INR z -code space. Formally, the optimization problem is defined as:

Note that functions ( e u , ξ u ) and ( e a , ξ a ) are parameterized respectively by the weights ( θ u , w u ) and ( θ a , w a ) , of the INRs and of the hypernetworks. In Equation (4), we used the ( e u , ξ u ) & ( e a , ξ a ) description for clarity, but as they are functions of ( θ u , w u ) & ( θ a , w a ) , optimization is tackled on the latter parameters. We outline the training pipeline in Appendix B, Figure 5. During training, we

constrain e u , e a to take only a few steps of gradient descent to facilitate the processor task. This regularization prevents the architecture from memorizing the training set into the individual codes and facilitates the auto-decoding optimization process for new inputs. In order to obtain a network that is capable of quickly encoding new physical inputs, we employ a second-order meta-learning training algorithm based on CAVIA (Zintgraf et al., 2019). Compared to a first-order scheme such as Reptile (Nichol et al., 2018), the outer loop back-propagates the gradient through the K inner steps, consuming more memory as we need to compute gradients of gradients but yielding higher reconstruction results with the modulated SIREN. We experimentally found that using 3 inner-steps for training, or testing, was sufficient to obtain very low reconstruction errors for most applications.

4 Experiments

To demonstrate the versatility of our model, we conducted experiments on three distinct tasks (Figure 1): (i) solving an initial value problem (Section 4.1), (ii) modeling the dynamics of a physical system (Section 4.2), and (iii) learning to infer the steady state of a system based on the domain geometry (Section 4.3) plus an associated design problem in Appendix D. Since each task corresponds to a different scenario, we utilized task-specific datasets and employed different baselines for each task. This approach was necessary because existing baselines typically focus on specific tasks and do not cover the full range of problems addressed in our study, unlike CORAL. We provide below an introduction to the datasets, evaluation protocols, and baselines for each task setting. All experiments were conducted on a single GPU: NVIDIA RTX A5000 with 25 Go. Code will be made available.

4.1 Initial Value Problem

An IVP is specified by an initial condition (here the input function providing the state variables at t = 0 ) and a target function figuring the state variables value at a given time T . Solving an IVP is a direct application of the CORAL framework introduced in Section 3.2.

Datasets We benchmark our model on two problems with non-convex domains proposed in Pfaff et al. (2021). In both cases, the fluid evolves in a domain - which includes an obstacle - that is more densely discretized near the boundary conditions (BC). The boundary conditions are provided by the mesh definition, and the models are trained on multiple obstacles and evaluated at test time on similar but different obstacles. · Cylinder simulates the flow of water around a cylinder on a fixed 2D Eulerian mesh, and is characteristic of incompressible fluids. For each node j we have access to the node position x ( j ) , the momentum w ( x ( j ) ) and the pressure p ( x ( j ) ) . We seek to learn the mapping ( x, w 0 ( x ) , p 0 ( x )) x ∈X → ( w T ( x ) , p T ( x )) x ∈X . · Airfoil simulates the aerodynamics around the cross-section of an airfoil wing, and is an important use-case for compressible fluids. In this dataset, we have in addition for each node j the fluid density ρ ( x ( j ) ) , and we seek to learn the mapping ( x, w 0 ( x ) , p 0 ( x ) , ρ 0 ( x )) x ∈X → ( w T ( x ) , p T ( x ) , ρ T ( x )) x ∈X . For both datasets, each example is associated to a mesh and the meshes are different for each example. For Airfoil the average number of nodes per mesh is 5233 and for Cylinder 1885 .

Evaluation protocols Training is performed using all the mesh points associated to an example. For testing we evaluate the following two settings. · Full , we validate that the trained model generalizes well to new examples using all the mesh location points of these examples. · Sparse We assess the capability of our model to generalize on sparse meshes: the original input mesh is down-sampled by randomly selecting 20% of its nodes. We use a train, validation, test split of 1000 / 100 / 100 samples for all the evaluations.

Baselines We compare our model to · NodeMLP , a FeedForward Neural Network that ignores the node neighbors and only learns a local mapping · GraphSAGE (Hamilton et al., 2017), a popular GNN architecture that uses SAGE convolutions · MP-PDE (Brandstetter et al., 2022b), a message passing GNN that builds on (Pfaff et al., 2021) for solving PDEs.

Results. We show in Table 1 the performance on the test sets for the two datasets and for both evaluation settings. Overall, CORAL is on par with the best models for this task. For the Full setting, it is best on Cylinder and second on Airfoil behind MP-PDE. However, for the sparse protocol, it can infer the values on the full mesh with the lowest error compared to all other models. Note that this

second setting is more challenging for Cylinder than for Airfoil given their respective average mesh size. This suggests that the interpolation of the model outputs is more robust on the Airfoil dataset, and explains why the performance of NodeMLP remains stable between the two settings. While MP-PDE is close to CORAL in the sparse setting, GraphSAGE fails to generalize, obtaining worse predictions than the local model. This is because the model aggregates neighborhood information regardless of the distance between nodes, while MP-PDE does consider node distance and difference between features.

4.2 Dynamics Modeling

For the IVP problem, in section 4.1, the objective was to infer directly the state of the system at a given time T given an initial condition (IC). We can extend this idea to model the dynamics of a physical system over time, so as to forecast state values over a given horizon. We have developed an autoregressive approach operating on the latent code space for this problem. Let us denote ( u 0 , u δt , ..., u Lδt ) a target sequence of observed functions of size L +1 . Our objective will be to predict the functions u kδt , k = 1 , ..., L , starting from an initial condition u 0 . For that we will encode z 0 = e ( u 0 ) , then predict sequentially the latent codes z kδt , k = 1 , ..., L using the processor in an auto regressive manner, and decode the successive values to get the predicted ˆ u kδt , k = 1 , ..., L at the successive time steps.

4.2.1 Implementation with Neural ODE

The autoregressive processor is implemented by a Neural ODE solver operating in the latent z -code space. Compared to the plain MLP implementation used for the IVP task, this provides both a natural autoregressive formulation, and overall, an increased flexibility by allowing to forecast at any time in a sequence, including different time steps or irregular time steps. Starting from any latent state z t , a neural solver predicts state z t + τ as z t + τ = z t + ∫ t + τ t ζ ψ ( z s ) ds with ζ ψ a neural network with parameters to be learned, for any time step τ . The autoregressive setting directly follows from this formulation. Starting from z 0 , and specifying a series of forecast time steps kδt for k = 1 , ..., L , the solver call NODESolve ( ζ ψ , z 0 , { kδt } k =1 ,...,L ) will compute predictions z kδt , k = 1 , ..., L autoregressively, i.e. using z kδt as a starting point for computing z ( k +1) δt . In our experiments we have used a fourth-order Runge-Kutta scheme (RK4) for solving the integral term. Using notations from Section 3.2, the predicted field at time step k can be obtained as ˆ u kδt = ξ ◦ g k ψ ( e ( u 0 ))) with g k ψ indicating k successive applications of processor g ψ . Note that when solving the IVP problem from Section 4.1, two INRs are used, one for encoding the input function and one for the output function; here a single modulated INR f θ,ϕ is used to represent a physical quantity throughout the sequence at any time. θ is then shared by all the elements of a sequence and ϕ is computed by a hypernetwork to produce a function specific code.

We use the two-step training procedure from Section 3.4, i.e. first the INR is trained to auto-decode the states of each training trajectory, and then the processor operating over the codes is learned through a Neural ODE solver according to Equation (5). The two training steps are separated and the codes are kept fixed during the second step. This allows for a fast training as the Neural ODE solver operates on the low dimensional code embedding space.

4.2.2 Experiment details

Datasets We consider two fluid dynamics equations for generating the datasets and refer the reader to Appendix A for additional details. · 2D-Navier-Stokes equation ( Navier-Stokes ) for a viscous, incompressible fluid in vorticity form on the unit torus: ∂w ∂t + u · ∇ w = ν ∆ w + f , ∇ u = 0 for x ∈ Ω , t > 0 , where ν = 10 -3 is the viscosity coefficient. The train and test sets are composed of 256 and 16 trajectories respectively where we observe the vorticity field for 40 timestamps. The original spatial resolution is 256 × 256 and we sub-sample the data to obtain frames of size 64 × 64 . · 3D-Spherical Shallow-Water equation ( Shallow-Water ) can be used as an approximation to a flow on the earth's surface. The data consists of the vorticity w , and height h of the fluid. The train and test sets are composed respectively of 16 and 2 long trajectories, where we observe the vorticity and height fields for 160 timestamps. The original spatial resolution is 128 (lat) × 256 (long), which we sub-sample to obtain frames of shape 64 × 128 . We model the dynamics with the complete state ( h, w ) . Each trajectory, for both both datasets and for train and test is generated from a different initial condition (IC).

Setting We evaluate the ability of the model to generalize in space and time. · Temporal extrapolation: For both datasets, we consider sub-trajectories of 40 timestamps that we split in two equal parts of size 20, with the first half denoted In-t and the second one Out-t . The trainingIn-t set is used to train the models at forecasting the horizon t = 1 to t = 19 . At test time, we unroll the dynamics from a new IC until t = 39 . Evaluation in the horizon In-t assesses CORAL's capacity to forecast within the training horizon. Out-t allows evaluation beyond In-t , from t = 20 to t = 39 . · Varying sub-sampling: We randomly sub-sample π percent of a regular mesh to obtain the train grid X tr , and a second test grid X te , that are shared across trajectories. The train and test grids are different, but have the same level of sparsity. · Up-sampling: We also evaluate the up-sampling capabilities of CORAL in Appendix C. In these experiments, we trained the model on a sparse, low-resolution grid and evaluate its performance on high resolution-grids.

Baselines To assess the performance of CORAL, we implement several baselines: two operator learning models, one mesh-based network and one coordinate-based method. · DeepONet (Lu et al., 2021): we train DeepONet in an auto-regressive manner with time removed from the trunk net's input. · FNO (Li et al., 2021): we use an auto-regressive version of the Fourier Neural Operator. · MP-PDE (Brandstetter et al., 2022b) : we use MP-PDE as the irregular mesh-based baseline. We fix MP-PDE's temporal bundling to 1, and train the model with the push-forward trick. · DINo (Yin et al., 2022) : We finally compare CORAL with DINo, an INR-based model designed for dynamics modeling.

4.2.3 Results

Table 2 details the performance of the different models in a combined temporal and spatial evaluation setting. · General remarks: CORAL demonstrates strong performance across all scenarios for both datasets. Only DINo exhibits similar properties, i.e., stability across spatial subsamplings and extrapolation horizon. We observe that all models performance degrade with lower sampling ratio. Also, as the models have been trained only on In-t horizon, error accumulates over time and thus leads to lower performance for Out-t evaluation. · Analysis per model: Although achieving strong performance on some specific scenarios, DeepONet, FNO and MP-PDE results are dependent of the training grid, geometries or number of points. FNO, can only be trained and evaluated on regular grids while DeepONet is not designed to be evaluated on a different grid in the branch net. MP-PDE achieves strong performance with enough sample positions, e.g. full grids here, but struggles to compete on irregular grids scenarios in Navier-Stokes. · Inference Time: We report in Appendix C, the inference time of the baselines considered. Despite operator methods have better inference time, CORAL is faster than mesh-free methods like DINo and MP-PDE. · Generalization across samplings: Coordinate-based methods demonstrate robustness when it comes to changes in spatial resolution. In contrast, MP-PDE model exhibits strong overfitting to the training grid, resulting in a decline in performance. Although MP-PDE and DINo may outperform CORAL in some settings, when changing the grid, CORAL remains stable and outperforms the other models. See Appendix C for details.

4.3 Geometry-aware inference

In this section, we wish to infer the steady state of a system from its domain geometry, all other parameters being equal. The domain geometry is partially observed from the data in the form of point clouds or of a structured mesh X i ⊂ Ω i . The position of the nodes depends on the particular object shape. Each mesh X i is obtained by deforming a reference grid X to adjust to the shape of the sample object. This grid deformation is the input function of the operator learning setting, while the output function is the physical quantity u i over the domain Ω i . The task objective is to train a model so as to generalize to new geometries, e.g. a new airfoil shape. Once a surrogate model has been trained to learn the influence of the domain geometry on the steady state solution, it can be used to quickly evaluate a new design and to solve inverse design problems (details in Appendix D).

Datasets. We used datasets generated from three different equations by Li et al. (2022a) and provide more details in Appendix A. · Euler equation ( NACA-Euler ) for a transonic flow over a NACA-airfoil. The measured quantity at each node is the Mach number. · Navier-Stokes Equation ( Pipe ) for an incompressible flow in a pipe, expressed in velocity form. The measured quantity at each node is the horizontal velocity. · Hyper-elastic material ( Elasticity ). Each sample represents a solid body with a void in the center of arbitrary shape, on which a tension is applied at the top. The material is the incompressible Rivlin-Saunders material and the measured quantity is the stress value. We use 1000 samples for training and 200 for test with all datasets.

Baselines We use · Geo-FNO (Li et al., 2022a) and · Factorized-FNO (Tran et al., 2023) two SOTA models as the main baselines. We also compare our model to regular-grid methods such as · FNO (Li et al., 2021) and · UNet (Ronneberger et al., 2015), for which we first interpolate the input.

Results In Table 3 we can see that CORAL achieves state-of-the-art results on Airfoil and Elasticity , with the lowest relative error among all models. It is slightly below Factorized-FNO and Geo-FNO on Pipe . One possible cause is that this dataset exhibits high frequency only along the vertical dimension, while SIREN might be better suited for isotropic frequencies. Through additional experiments, we demonstrate in Appendix D, how CORAL can also be used for solving an inverse problem

corresponding to a design task: optimize the airfoil geometry to minimize the drag over lift ratio. This additional task further highlights the versatility of this model.

5 Discussion and limitations

Although a versatile model, CORAL inherits the limitations of INRs concerning the training time and representation power. It is then faster to train than GNNs, but slower than operators such as FNO, DeepONet and of course CNNs which might limit large scale deployments. Also some physical phenomena might not be represented via INRs. Although this is beyond the scope of this paper, it remains to evaluate the methods on large size practical problems. An interesting direction for future work would be to derive an efficient spatial latent representation for INRs, taking inspiration from grid-based representation for INRs (Takikawa et al. (2022), M¨ uller et al. (2022), Saragadam et al. (2022)). Another avenue would be to leverage clifford layers (Brandstetter et al., 2022a) to model interactions between physical fields.

6 Conclusion

We have presented CORAL, a novel approach for Operator Learning that removes constraints on the input-output mesh. CORAL offers the flexibility to handle spatial sampling or geometry variations, making it applicable to a wide range of scenarios. Through comprehensive evaluations on diverse tasks, we have demonstrated that it consistently achieves state-of-the-art or competitive results compared to baseline methods. By leveraging compact latent codes to represent functions, it enables efficient and fast inference within a condensed representation space.

References

Kelsey R. Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. 2 2022. URL http://arxiv.org/abs/2202.00728 .

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR , abs/1506.03099, 2015. URL http://arxiv. org/abs/1506.03099 .

Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for pde modeling. In The Eleventh International Conference on Learning Representations , 2022a.

Johannes Brandstetter, Daniel E Worrall, and Max Welling. Message passing neural pde solvers. International Conference on Learning Representations , 2022b.

Keaton J. Burns, Geoffrey M. Vasil, Jeffrey S. Oishi, Daniel Lecoanet, and Benjamin P. Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research , 2, 2020. ISSN 26431564. doi: 10.1103/PhysRevResearch.2.023068.

Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physicsinformed neural networks (pinns) for fluid mechanics: a review. Acta Mechanica Sinica/Lixue Xuebao , 37, 2021. ISSN 16143116. doi: 10.1007/s10409-021-01148-1.

Sangwon Chae, Joonhyeok Shin, Sungjun Kwon, Sangmok Lee, Sungwon Kang, and Donghyun Lee. Pm10 and pm2.5 real-time prediction models using an interpolated convolutional neural network. Scientific Reports , 11, 2021. ISSN 20452322. doi: 10.1038/s41598-021-91253-9.

Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, Yue Chang, G A Pershing, Henrique Teles Maia, Maurizio Chiaramonte, Kevin Carlberg, and Eitan Grinspun. Crom: Continuous reduced-order modeling of pdes using implicit neural representations. International Conference on Learning Representation , 6 2022. URL http://arxiv.org/abs/2206.02607 .

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June, 2019. ISSN 10636919. doi: 10.1109/CVPR.2019.00609.

Emmanuel de B´ ezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: incorporating prior scientific knowledge. Journal of Statistical Mechanics: Theory and Experiment , 2019, 12 2019. ISSN 17425468. doi: 10.1088/1742-5468/ab3195.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations. , 10 2021. URL http://arxiv.org/ abs/2010.11929 .

Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. Proceedings of the 39 th International Conference on Machine Learning , 1 2022. URL http://arxiv.org/abs/ 2201.12204 .

Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. International Conference on Learning Representations. , 2021.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Joseph Galewsky, Richard K. Scott, and Lorenzo M. Polvani. An initial-value problem for testing numerical models of the global shallow-water equations. Tellus, Series A: Dynamic Meteorology and Oceanography , 56, 2004. ISSN 02806495. doi: 10.1111/j.1600-0870.2004.00071.x.

William L. Hamilton. Graph Representation Learning: Foundations, Methods, Applications and Systems . Morgan and Claypool, 2020. ISBN 9781450383325. doi: 10.1145/3447548.3470824.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. volume 2017-December, 2017.

Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning , pp. 12556-12569. PMLR, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2016-December, 2016. ISSN 10636919. doi: 10.1109/CVPR.2016.90.

Ayed Ibrahim, de B´ ezenac Emmanueland Pajot Arthur, and Gallinari Patrick. Modelling spatiotemporal dynamics from Earth observation data with neural differential equations. Machine Learning , 111(6):2349-2380, 2022. ISSN 15730565. URL https://doi.org/10.1007/ s10994-022-06139-2 .

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. 8 2021. URL http://arxiv.org/abs/2108.08481 .

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60, 2017. ISSN 15577317. doi: 10.1145/3065386.

Zhipeng Li, Fengwei Zhou, Fei Chen, Hang Li, Tongliang Liu, and Tat-Seng Chen. Meta-sgd: Learning to learn quickly for few-shot learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 2111-2120. JMLR. org, 2017.

Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Transactions on Machine Learning Research (April/2023) , 2023.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations. , 10 2021. URL http://arxiv.org/ abs/2010.08895 .

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier Neural Operator with Learned Deformations for PDEs on General Geometries. In arXiv:2207.05209v1 , 2022a.

Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. 39th International Conference on Machine Learning (ICML2022) , 11 2022b. URL http://arxiv.org/abs/2111.03794 .

David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. Conference on Computer Vision and Pattern Recognition , 12 2022. URL http://arxiv.org/abs/2112.04645 .

Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. volume 7, 2018.

Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. Nat Mach Intell , 3:218-229, 10 2021. doi: 10.1038/s42256-021-00302-5. URL http://arxiv. org/abs/1910.03193http://dx.doi.org/10.1038/s42256-021-00302-5 .

Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data , volume 393. Elsevier B.V., 4 2022. doi: 10.1016/j.cma. 2022.114778.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June, 2019. ISSN 10636919. doi: 10.1109/CVPR.2019.00459.

W. Morton and D. F. Mayers. Numerical solution of partial differential equations: An introduction . 2005. doi: 10.1017/CBO9780511812248.

Thomas M¨ uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics , 41(4):1-15, jul 2022. doi: 10.1145/3528223.3530127. URL https://doi.org/10.1145%2F3528223. 3530127 .

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.

Peter J. Olver. Introduction to partial differential equations . Undergraduate Texts in Mathematics. Springer Cham, 2014.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2019-June, 2019. ISSN 10636919. doi: 10.1109/CVPR.2019.00025.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .

Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 , 2018. ISSN 2159-5399. doi: 10.1609/aaai.v32i1.11671.

Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning meshbased simulation with graph networks. International Conference on Learning Representations. , 10 2021. URL http://arxiv.org/abs/2010.03409 .

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. volume 9351, 2015. doi: 10.1007/978-3-319-24574-4 28.

Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard G. Baraniuk, and Ashok Veer-

araghavan. Miner: Multiscale implicit neural representations, 2022.

Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems , 2020-December, 2020a. ISSN 10495258.

Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems , 2020-December, 2020b. ISSN 10495258.

Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas M¨ uller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. ACM Transactions on Graphics , 2022.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 2020-December, 2020. ISSN 10495258.

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2021. ISSN 10636919. doi: 10.1109/CVPR46437.2021.00287.

Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. 11 2023. URL http://arxiv.org/abs/2111.13802 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 2017-December, 2017. ISSN 10495258.

Petar Veliˇ ckovi´ c, Arantxa Casanova, Pietro Li` o, Guillem Cucurull, Adriana Romero, and Yoshua Bengio. Graph attention networks. 2018. doi: 10.1007/978-3-031-01587-8 7.

Yuan Yin, Matthieu Kirchmeyer, Jean-Yves Franceschi, Alain Rakotomamonjy, and Patrick Gallinari. Continuous pde dynamics forecasting with implicit neural representations. International Conference on Learning Representations , 9 2022. URL http://arxiv.org/abs/2209.14855 .

Luisa Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. 36th International Conference on Machine Learning, ICML 2019 , 2019-June, 2019.

A Dataset Details

A.1 Initial Value Problem

We use the datasets from Pfaff et al. (2021), and take the first and last frames of each trajectory as the input and output data for the initial value problem.

Cylinder The dataset includes computational fluid dynamics (CFD) simulations of the flow around a cylinder, governed by the incompressible Navier-Stokes equation. These simulations were generated using COMSOL software, employing an irregular 2D-triangular mesh. The trajectory consists of 600 timestamps, with a time interval of ∆ t = 0 . 01 s between each timestamp.

Airfoil The dataset contains CFD simulations of the flow around an airfoil, following the compressible Navier-Stokes equation. These simulations were conducted using SU2 software, using an irregular 2D-triangular mesh. The trajectory encompasses 600 timestamps, with a time interval of ∆ t = 0 . 008 s between each timestamp.

A.2 Dynamics Modeling

2D-Navier-Stokes ( Navier-Stokes ) We consider the 2D Navier-Stokes equation as presented in Li et al. (2021); Yin et al. (2022). This equation models the dynamics of an incompressible fluid on a rectangular domain Ω = [ -1 , 1] 2 . The PDE writes as :

where u is the velocity, w the vorticity. ν is the fluid viscosity, and f is the forcing term, given by:

For this problem, we consider periodic boundary conditions.

By sampling initial conditions as in Li et al. (2021), we generated different trajectories on a 256 × 256 regular spatial grid and with a time resolution δt = 1 . We retain the trajectory starting from the 20th timestep so that the dynamics is sufficiently expressed. The final trajectories contains 40 snapshots at time t = 20 , 21 , · · · , 59 . As explained in section 4, we divide these long trajectories into 2 parts : the 20 first frames are used during the training phase and are denoted as In-t throughout this paper. The 20 last timesteps are reserved for evaluating the extrapolation capabilities of the models and are the Out-t part of the trajectories. In total, we collected 256 trajectories for training, and 16 for evaluation.

3D-Spherical Shallow-Water ( Shallow-Water ). We consider the shallow-water equation on a sphere describing the movements of the Earth's atmosphere:

where d d t is the material derivative, k is the unit vector orthogonal to the spherical surface, u is the velocity field tangent to the surface of the sphere, which can be transformed into the vorticity w = ∇× u , h is the height of the sphere. We generate the data with the Dedalus software (Burns et al., 2020), following the setting described in Yin et al. (2022), where a symmetric phenomena can be seen for both northern and southern hemisphere. The initial zonal velocity u 0 contains two non-null symmetric bands in the both hemispheres, which are parallel to the circles of latitude. At each latitude and longitude ϕ, θ ∈ [ -π 2 , π 2 ] × [ -π, π ] :

where u max is the maximum velocity, ϕ 0 = π 7 , ϕ 1 = π 2 -ϕ 0 , and e n = exp( -4 ( ϕ 1 -ϕ 0 ) 2 ) . The water height h 0 is initialized by solving a boundary value conditioned problem as in Galewsky et al. (2004) which is perturbed by adding h ′ 0 to h 0 :

where ϕ 2 = π 4 , ˆ h = 120 m , α = 1 3 and β = 1 15 are constants defined in Galewsky et al. (2004). We simulated the phenomenon using Dedalus Burns et al. (2020) on a latitude-longitude grid (latlon). The original grid size was 128 (lat) × 256 (lon), which we downsampled to obtain grids of size 64 × 128 . To generate trajectories, we sampled u max from a uniform distribution U (60 , 80) . Snapshots were captured every hour over a duration of 320 hours, resulting in trajectories with 320 timestamps. We created 16 trajectories for the training set and 2 trajectories for the test set. However, since the dynamical phenomena in the initial timestamps were less significant, we only considered the last 160 snapshots. Each long trajectory is then sliced into sub-trajectories of 40 timestamps each. As a result, the training set contains 64 trajectories, while the test set contains 8 trajectories. It is worth noting that the data was also scaled to a reasonable range: the height h was scaled by a factor of 3 × 10 3 , and the vorticity w was scaled by a factor of 2.

A.3 Geometric aware inference

We use the datasets provided by Li et al. (2022a) and adopt the original authors' train/test split for our experiments.

Euler's Equation ( Naca-Euler ). Weconsider the transonic flow over an airfoil, where the governing equation is Euler equation, as follows:

where ρ f is the fluid density, u is the velocity vector, p is the pressure, and E is the total energy. The viscous effect is ignored. The far-field boundary condition is ρ ∞ = 1 , p ∞ = 1 . 0 , M ∞ = 0 . 8 , AoA = 0 , where M ∞ is the Mach number and AoA is the angle of attack. At the airfoil, a nopenetration condition is imposed. The shape parameterization of the airfoil follows the design element approach. The initial NACA-0012 shape is mapped onto a 'cubic' design element with 8 control nodes, and the initial shape is morphed to a different one following the displacement field of the control nodes of the design element. The displacements of control nodes are restricted to the vertical direction only, with prior d ∼ U [ -0 . 05 , 0 . 05] .

We have access to 1000 training data and 200 test data, generated with a second-order implicit finite volume solver. The C-grid mesh with about (200 × 50) quadrilateral elements is used, and the mesh is adapted near the airfoil but not the shock. The mesh point locations and Mach number on these mesh points are used as input and output data.

Hyper-elastic material ( Elasticity ). The governing equation of a solid body can be written as

where ρ s is the mass density, u is the displacement vector, and σ is the stress tensor. Constitutive models, which relate the strain tensor ε to the stress tensor, are required to close the system. We consider the unit cell problem Ω = [0 , 1] × [0 , 1] with an arbitrary shape void at the center, which is depicted in Figure 2(a). The prior of the void radius is r = 0 . 2+0 . 2 with ˜ r ∼ N (0 , 42( -∇ +32) -1 ) , 1+exp(˜ r ) , which embeds the constraint 0 . 2 ≤ r ≤ 0 . 4 . The unit cell is clamped on the bottom edges and tension traction t = [0 , 100] is applied on the top edge. The material is the incompressible RivlinSaunders material with energy density function parameters C 1 = 1 . 863 × 10 5 and C 1 = 9 . 79 × 10 3 . The data was generated with a finite element solver with about 100 quadratic quadrilateral elements. The inputs a are given as point clouds with a size around 1000. The target output is stress.

Navier-Stokes Equation ( Pipe ). We consider the incompressible flow in a pipe, where the governing equation is the incompressible Navier-Stokes equation, as following,

where v is the velocity vector, p is the pressure, and µ = 0 . 005 is the viscosity. The parabolic velocity profile with maximum velocity v = [1 , 0] is imposed at the inlet. A free boundary condition is imposed at the outlet, and a no-slip boundary condition is imposed at the pipe surface. The pipe has a length of 10 and width of 1. The centerline of the pipe is parameterized by 4 piecewise cubic polynomials, which are determined by the vertical positions and slopes on 5 spatially uniform control nodes. The vertical position at these control nodes obeys d ∼ U [ -2 , 2] , and the slope at these control nodes obeys d ∼ U [ -1 , 1] .

We have access to 1000 training data and 200 test data, generated with an implicit finite element solver using about 4000 Taylor-Hood Q2-Q1 mixed elements. The mesh point locations (129 × 129) and horizontal velocity on these mesh points are used as input and output data.

B Implementation Details

We implemented all experiments with PyTorch (Paszke et al., 2019). The code is available at https://anonymous.4open.science/r/coral-0348/ . We estimate the computation time needed for development and the different experiments to approximately 400 days.

B.1 CORAL

B.1.1 Architecture Details

SIREN initialization. We use for SIREN the same initialization scheme as in Sitzmann et al. (2020b), i.e., sampling the weights of the first layer according to a uniform distribution U ( -1 /d, 1 /d ) and the next layers according to U ( -1 w 0 √ 6 d in , 1 w 0 √ 6 d in ) . We use the default PyTorch initialization for the hypernetwork.

Decode with shift-modulated SIREN. Initially, we attempted to modulate both the scale and shift of the activation, following the approach described in Perez et al. (2018). However, we did not observe any performance improvement by employing both modulations simultaneously. Consequently, we decided to focus solely on shift modulations, as it led to a more stable training process and reduced the size of the modulation space by half. We provide an overview of the decoder with the shift-modulated SIREN in Figure 3.

Encode with auto-decoder . We provide a schematic view of the input encoder in Figure 4. The auto-decoding process starts from a code z a = 0 and performs K steps of gradient descent over this latent code to minimize the reconstruction loss.

Process with MLP. We use an MLP with skip connections and Swish activation functions. Its forward function writes g ψ ( z ) = Block k ◦ ... ◦ Block 1 ( z ) , where Block is a two-layer MLP with skip connections:

In Equation (15), σ denotes the feature-wise Swish activation. We use the version with learnable parameter β ; σ ( z ) = z · sigmoid ( βz ) .

B.1.2 Training Details

The training is done in two steps. First, we train the modulated INRs to represent the data. We show the details with the pseudo-code in Algorithms 1 and 2. α is the inner-loop learning rate while λ is the outer loop learning rate, which adjusts the weights of the INR and hypernetwork. Then, once the INRs have been fitted, we obtain the latent representations of the training data, and use these latent codes to train the forecast model g ψ (See Algorithm 3). We note λ ψ the learning rate of g ψ .

Z-score normalization. As the data is encoded using only a few steps of gradients, the resulting standard deviation of the codes is very small, falling within the range of [1e-3, 5e-2]. However, these 'raw' latent representations are not suitable as-is for further processing. To address this, we normalize the codes by subtracting the mean and dividing by the standard deviation, yielding the normalized code: z norm = z -mean std . Depending on the task, we employ slightly different types of normalization:

Modulation

SIREN

input

function

obs.

space

hypernetwork

input

output

function

output

space

hypernetwork

Modulation

SIREN

input

(a) The hypernetwork h a maps the input code z a to the modulations ϕ a . The modulations shift the activations at each layer of the SIREN.

(b) The hypernetwork h u maps the input code z u to the modulations ϕ u . The modulations shift the activations at each layer of the SIREN.

Initial value problem: · Cylinder : We normalize the inputs and outputs code with the same mean and standard deviation. We compute the statistics feature-wise, across the inputs and outputs. · Airfoil : We normalize the inputs and outputs code with their respective mean and standard deviation. The statistics are real values.

Dynamics modeling: We normalize the codes with the same mean and standard deviation. The statistics are computed feature-wise, over all training trajectories and all available timestamps (i.e. over In-t ).

Geometry-aware inference: We normalize the input codes only, with feature-wise statistics.

steps

encoding as

auto-decoding

input

input

output

Train processor

to forecast

output code

Step 1

Step 2

Input INR

training

Output INR

training

B.1.3 Inference Details

We present the inference procedure in Algorithm 4. It is important to note that the input and output INRs, f θ a and f θ u , respectively, accept the 'raw' codes as inputs, whereas the processor expects a normalized latent code. Therefore, after the encoding steps, we normalize the input code. Additionally, we may need to denormalize the code immediately after the processing stage. It is worth mentioning that we maintain the same number of inner steps as used during training, which is 3 for all tasks.

Algorithm 1: Training of the input INR

Algorithm 3: Training of the processor

Algorithm 4: CORAL Inference, given a function a

B.1.4 Choice of Hyperparameters

We recall that d z denotes the size of the code, w 0 is a hyperparameter that controls the frequency bandwith of the SIREN network, λ is the outer-loop learning rate (on f θ,ϕ and h w ), α is the inner-loop learning rate, K is the number of inner steps used during training and encoding steps at test time, λ ψ is the learning rate of the MLP or NODE. In some experiments we learn the inner-loop learning rate α , as in Li et al. (2017). In such case, the metaα learning rate is an additional parameter that controls how fast we move α from its initial value during training. When not mentioned we simply report α in the tables below, and otherwise we report the initial learning rate and this meta-learning-rate.

We use the Adam optimizer during both steps of the training. For the training of the Inference / Dynamics model, we use a learning rate scheduler which reduces the learning rate when the loss has stopped improving. The threshold is set to 0.01 in the default relative threshold model in PyTorch, with a patience of 250 epochs w.r.t. the train loss. The minimum learning rate is 1e-5.

Initial Value Problem We provide the list of hyperparameters used for the experiments on Cylinder and Airfoil in Table 4.

Dynamics Modeling Table 5 summarizes the hyperparameters used in our experiments for dynamics modeling on datasets Navier-Stokes and Shallow-Water (Table 2).

Furthermore, to facilitate the training of the dynamics within the NODE, we employ Scheduled Sampling, following the approach described in Bengio et al. (2015). At each timestep, there is a

Algorithm 2: Training of the output INR

probability of ϵ % for the integration of the dynamics through the ODE solver to be restarted using the training snapshots. This probability gradually decreases during the training process. Initially, we set ϵ init = 0 . 99 , and every 10 epochs, we multiply it by 0.99. Consequently, by the end of the training procedure, the entire trajectory is computed with the initial condition.

Geometry-aware inference We provide the list of hyperparameters used for the experiments on NACA-Euler , Elasticity , and Pipe in Table 4.

B.2 Baseline Implementation

We detail in this section the architecture and hyperparameters used for the training of the baselines presented in Section 4.

Initial Value Problem We use the following baselines for the Initial Value Problem task.

NodeMLP . We use a ReLU-MLP with 3 layers and 512 neurons. We train it for 10000 epochs. We use a learning rate of 1e-3 and a batch size of 64.

GraphSAGE . We use the implementation from torch-geometric (Fey & Lenssen, 2019), with 6 layers of 64 neurons. We use ReLU activation. We train the model for 400 epochs for Airfoil and 4,000 epochs for Cylinder . We build the graph using the 16 closest nodes. We use a learning rate of 1e-3 and a batch size of 64.

MP-PDE : We implement MP-PDE as a 1-step solver, where the time-bundling and pushforward trick do not apply. We use 6 message-passing blocks and 64 hidden features. We build the graph with the 16 closest nodes. We use a learning rate of 1e-3 and a batch size of 16. We train for 500 epochs on Airfoil and 1000 epochs on Cylinder .

Dynamics Modeling All our baselines are implemented in an auto-regressive (AR) manner to perform forecasting.

DeepONet : We use a DeepONet in which both Branch Net and Trunk Net are 4-layers MLP with 100 neurons. The model is trained for 10 , 000 epochs with a learning rate of 1e-5. To complete the upsampling studies, we used a modified DeepONet forward which computes as follows: (1) Firstly, we compute an AR pass on the training grid to obtain a prediction of the complete trajectory with the model on the training grid. (2) We use these prediction as input of the branch net for a second pass on the up-sampling grid to obtain the final prediction on the new grid.

FNO : FNO is trained for 2 , 000 epochs with a learning rate of 1e-3. We used 12 modes and a width of 32 and 4 Fourier layers. We also use a step scheduler every 100 epochs with a decay of 0.5.

MP-PDE : We implement MP-PDE with a time window of 1 so that is becomes AR. The MP-PDE solver is composed of a 6 message-passing blocks with 128 hidden features. To build the graphs, we limit the number of neighbors to 8. The optimization was performed on 10 , 000 epochs with a learning rate of 1e-3 and a step scheduler every 2000 epochs until 10000. We decay the learning rate of 0.4 with weight decay 1e-8.

DINo : DINo uses MFN model with respectively width and depth of 64 and 3 for NavierStokes (NS), and 256 and 6 for Shallow-Water (SW). The encoder proceeds to 300 (NS) or 500 (SW) steps to optimize the codes whose size is set to 100 (NS) or 200 (SW). The dynamic is solved with a NODE that uses 4 -layers MLP and a hidden dimension of 512 (NS) or 800 (SW). This model is trained for 10 , 000 epochs with a learning rate of 5e-3. We use the same scheduled sampling as for the CORAL training (see appendix B.1.4).

Geometry-aware inference Except on Pipe , the numbers for FactorizedFNO are taken from Tran et al. (2023). In the latter we take the 12-layer version which has a comparable model size. We train the 12-layer FactorizedFNO on Pipe with AdamW for 200 epochs with modes (32, 16), a width of 64, a learning rate of 1e-3 and a weight decay of 1e-4. We implemented the baselines GeoFNO , FNO , UNet according to the code provided in Li et al. (2022a)

C Supplementary Results for Dynamics Modeling

C.1 Robustness to Resolution Changes

We present in Tables 6 and 7 the up-sampling capabilities of CORAL and relevant baselines both In-t and Out-t , respectively for Navier-Stokes and Shallow-Water .

These tables show that CORAL remains competitive and robust on up-sampled inputs. Other baselines can also predict on denser grids, except for MP-PDE, which over-fitted the training grid.

C.2 Learning a Dynamics on Different Grids

To extend our work, we propose to study how robust is CORAL to changes in grids. In our classical setting, we keep the same grid for all trajectories in the training set and evaluate it on a new grid for the test set. Instead, here, both in train and test sets, each trajectory i has its own grid X i . Thus, we evaluate CORAL's capability to generalize to grids. We present the results in Table 8. Overall, coordinate-based methods generalize better over grids compared to operator based and discrete methods like DeepONet and MP-PDE which show better or equivalent performance when trained only on one grid. CORAL's performance is increased when trained on different grids; one possible reason is that CORAL overfits the training grid used for all trajectories in our classical setting.

C.3 Training Time

In Table 9, we present the training time for CORAL and different baselines for comparison. Since, the training of CORAL is separated in 2 steps, we show in line 'INR' the training time for INR fitting and in line 'Process' the second step to train the forecast model. We see that the longest part of the training procedure in CORAL is the fitting of the INR. MP-PDE is the slowest baseline to train, due to the KNN graph creation. DeepONet and FNO are the fastest baselines to train because they only need a forward pass.

C.4 Inference Time

In this section, we evaluate the inference time of CORAL and other baselines w.r.t. the input grid size. We study the impact of the training grid size (different models trained with 5%, 20% and 100% of the grid) (Figure 6a) and the time needed for a model trained (5%) on a given grid to make computation on finer grid size resolution (evaluation grid size) (Figure 6b).

On the graphs presented in Figure 6, we observe that except for the operator baselines, CORAL is also competitive in terms of inference time. MP-PDE inference time increases strongly when inference grid gets denser. The DINo model, which is the only to propose the same properties as CORAL, is much slower when both inference and training grid size evolve. This difference is mainly explained by the number of steps needed to optimize DINo codes. Indeed, at test time DINO's adaptation requires 100x more optimization steps. MPPDE's computation is slow due to the KNN graph creation and slower message passing. DeepONet and FNO are faster due to forward computation only. CORAL's encoding/decoding is relatively resolution-insensitive and performed in parallel across all sensors. Process operates on a fixed dimension independent of the resolution. FNO is fast due to FFT but

Inference time wrt the subsamplingrate.

101

coral

dino

mppde

一

deeponet

* fno

100

10-1

10-2

0.05

0.2

(a) Average inference time (in seconds) of the implemented baselines to unroll a trajectory until

T

= 40

on

Navier-Stokes

. For irregular grids,

FNO is performed following linear interpolation.

cannot be used on irregular grids. For these experiments, we used linear interpolation which slowed the inference time.

C.5 Propagation of Errors Through Time

In Figures 7a to 7c, we show the evolution of errors as the extrapolation horizon evolves. First, we observe that all baselines propagate error through time, since the trajectories are computed using an auto-regressive approach. Except for the 100% , DeepONet had difficulties to handle the dynamic. It has on all settings the highest error. Then, we observe that for MP-PDE and FNO, the error increases quickly at the beginning of the trajectories. This means that these two models are rapidly propagating error. Finally, both DINo and CORAL have slower increase of the error during In-t and Out-t periods. However, we clearly see on the graphs that DINo has more difficulties than CORAL to make predictions out-range. Indeed, while CORAL's error augmentation remains constant as long as the time evolves, DINo has a clear increase.

C.6 Benchmarking INRs for CORAL

We provide some additional experiments for dynamics modeling with CORAL, but with diffrents INRs: MFN (Fathony et al., 2021), BACON (Lindell et al., 2022) and FourierFeatures (Tancik et al., 2020). Experiments have been done on Navier-Stokes on irregular grids sampled from grids of size 128 × 128 . All training trajectories share the same grid and are evaluated on a new grid for test trajectories. Results are reported in Table 10. Note that we used the same learning hyper-parameters for the baselines than those used for SIREN in CORAL. SIREN seems to produce the best codes for dynamics modeling, both for in-range and out-range prediction.

Inference time wrt to up-sampling rate.

101

coral

dino

—mppde

deeponet

fno

100

10-1

10-2

0.05

2

Errorswrttime

coral

oup

mppde

deeponet

fno

x_tr

64x64

128x128

256x256

10-

10-1

10-

10-1

102

102

102

10-²

103

10-3

103

10-3,

104

10-

10-

10-

In-t

1-1no

In-t

1-1no

In-t

1-1no

In-t

1-1no

10

20

30

40

10

20

30

40

10

20

30

10

20

30

40

Errors wrt time

coral

dino

mppde

deeponet

x_tr

64x64

128x128

256x256

100

100

100

100

10-

10-

10-1

10-1

10-2

10-2

10-²

10-2

103

10-3

10-3

10-3

In-t

1-no

In-t

1-no

In-t

1-1no

In-t

1-1no

10

20

OE

40

10

20

30

40

10

20

30

40

10

20

30

40

coral

dino

mppde

deeponet

x_tr

64x64

128x128

256x256

100

100

10°

100

101

10-

10-1

101

102

10-2

10-2

102

In-t

Out-t

In-t

Out-t

In-t

11no

In-t

1-no

10

20

30

40

10

20

30

40

0

10

20

30

40

10

20

30

40

C.7 Impact of 2nd order meta-learning

We provide in Figure 8 the evolution of the reconstruction error through the training epochs for Navier-Stokes with first-order and second-order meta-learning. The first order method is able to correctly train until it falls into an unstable regime for the common parameters. The second order method is much more stable and achieves a 10x reduction in MSE.

INR-trainingloss with2nd order meta-learning.

100

seed 1

100

seed 1

seed2

seed2

seed 3

seed 3

101

10-1

102

10-2

10-3

103

104

10~4

0

2000

4000

6000

8000

10000

D

2000

4000

6000

8000

10000

C.8 Key hyper parameter analysis

Table 11 presents a hyperparameter study on reconstruction and forecasting tasks for Navier-Stokes dataset. Three hyperparameters-initial weight w 0 , latent dimension d z , and width-are varied to observe their impact. We can notice that:

w 0 = 30 slightly improves the reconstruction on the test set.

d z = 64 yields a better forecasting In-t performance.

width = 256 significantly improves the model's performance across nearly all metrics.

D Supplementary results for geometry-aware inference

D.1 Inverse Design for NACA-airfoil

Once trained on NACA-Euler , CORAL can be used for the inverse design of a NACA airfoil. We consider an airfoil's shape parameterized by seven spline nodes and wish to minimize drag and maximize lift. We optimize the design parameters in an end-to-end manner. The spline nodes create

the input mesh, which CORAL maps to the output velocity field. This velocity field is integrated to compute the drag and the lift, and the loss objective is the squared drag over lift ratio. As can be seen in Figure 9, iterative optimization results in an asymmetric airfoil shape, enhancing progressively the lift coefficient in line with physical expectations. At the end of the optimization we reach a drag value of 0.042 and lift value of 0.322.

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.4900

0.4925

0.4950

0.4975

0.5000

0.5025

0.5050

0.5075

0.496

0.498

0.500

0.502

0.504

(a) Step = 0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.4900

0.4925

0.4950

0.4975

0.5000

0.5025

0.5050

0.5075

0.496

0.498

0.500

0.502

0.504

(b) Step = 1000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.4900

0.4925

0.4950

0.4975

0.5000

0.5025

0.5050

0.5075

0.496

0.498

0.500

0.502

0.504

(c) Step = 3000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.4900

0.4925

0.4950

0.4975

0.5000

0.5025

0.5050

0.5075

0.496

0.498

0.500

0.502

0.504

(d) Step = 5000

E Qualitative results

In this section, we show different visualization of the predictions made by CORAL on the three considered tasks in this paper.

E.1 Initial Value Problem

We provide in Figure 10 and Figure 11 visualizations of the inferred values of CORAL on Cylinder and Airfoil .

E.2 Dynamics modeling

We provide in Figure 13 and Figure 12 visualization of the predicted trajectories of CORAL on Navier-Stokes and Shallow-Water .

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Initial pressure

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Initial Vx

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Initial Vy

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Pred pressure

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Pred Vx

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

Pred Vy

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

True pressure

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

True Vx

0.0

0.5

1.0

1.5

0.0

0.1

0.2

0.3

0.4

True Vy

0.0

0.5

1.0

1.5

2.0

2.5

0.5

0.0

0.5

1.0

0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.25

0.00

0.25

0.50

0.75

1.00

1.25

1.50

0.0

0.5

1.0

1.5

0.75

0.50

0.25

0.00

0.25

0.50

0.75

0.75

0.50

0.25

0.00

0.25

0.50

0.75

0.75

0.50

0.25

0.00

0.25

0.50

0.75

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Initial pressure

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Initial density

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Initial Vx

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Initial Vy

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Pred pressure

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Pred density

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Pred Vx

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

Pred Vy

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

True pressure

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

True density

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

True Vx

0.00

0.05

0.04

0.02

0.00

0.02

0.04

0.06

0.08

True Vy

1

0

1

2

3

4

6

4

2

0

6

4

2

0

1

0

1

2

3

6

4

2

0

6

4

2

0

2.0

1.5

1.0

0.5

0.0

3

2

1

0

1

3

2

1

0

1

2

1

0

1

4

3

2

1

0

5

4

3

2

1

0

.......

Subsampling

Observation

Predicted trajectory

rate

grid

t=0

In-t

→t=T

Out-t

→t=T

π=5%

π=100%

Ground truth

Subsampling

Observation

Predicted trajectory

rate

grid

t=0

In-t

t=T

Out-t

t=T

5

π=5%

π=100%

Ground truth

E.3 Geometry-aware inference

We provide in Figure 14, Figure 15, Figure 16 visualization of the predicted values of CORAL on NACA-Euler , Pipe and Elasticity .

0.496

0.498

0.500

0.502

0.504

Input mesh

0.496

0.498

0.500

0.502

0.504

Ground truth

0.496

0.498

0.500

0.502

0.504

Prediction

0.496

0.498

0.500

0.502

0.504

Input mesh

0.496

0.498

0.500

0.502

0.504

Ground truth

0.496

0.498

0.500

0.502

0.504

Prediction

0.496

0.498

0.500

0.502

0.504

Input mesh

0.496

0.498

0.500

0.502

0.504

Ground truth

0.496

0.498

0.500

0.502

0.504

Prediction

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.2

0.4

0.6

0.8

1.0

1.2

0.2

0.4

0.6

0.8

1.0

1.2

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Input mesh

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Ground truth

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Prediction

0.2

0.3

0.4

0.5

0.6

0.7

Input mesh

0.2

0.3

0.4

0.5

0.6

0.7

Ground truth

0.2

0.3

0.4

0.5

0.6

0.7

Prediction

0.2

0.4

0.6

0.8

Input mesh

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Ground truth

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Prediction

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.25

0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.25

0.0

0.2

0.4

0.6

0.8

1.0

Input mesh

0.0

0.2

0.4

0.6

0.8

1.0

Ground truth

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

0.0

0.2

0.4

0.6

0.8

1.0

Input mesh

0.0

0.2

0.4

0.6

0.8

1.0

Ground truth

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

0.0

0.2

0.4

0.6

0.8

1.0

Input mesh

0.0

0.2

0.4

0.6

0.8

1.0

Ground truth

0.0

0.2

0.4

0.6

0.8

1.0

Prediction

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

0

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

Peer reviews:
comments: SUMMARY:
This paper creatively employs Implicit Neural Representations (INR) for operator learning on irregular domains. This novel method benefits from INR's capability to adaptively handle irregular grid distributions or irregular geometric areas, making learning the mapping from input to output in the INR's latent space quite natural. The primary experiments of the paper are on initial value problems, where it shows a significant improvement. However, a notable challenge is that the introduction of INR necessitates bi-level optimization, which potentially introduces instability and difficulty into the training process.

STRENGTHS:
1.The proposed method is highly innovative.
2.The quality of writing is excellent, and the paper is easy to read.
3. As an empirical paper, the experimental results surpass the baselines.

WEAKNESSES:
1.The paper lacks research on related work. Besides Geo-FNO and FFNO, transformer-based methods can also naturally handle irregular geometric areas. The authors should at least mention these works. Here are some references for consideration [1,2,3].
2. Operator learning tasks are not computationally demanding, so the paper should explore the method's stability and report error bars (at least for some cases).
3. More explanation or supplementation is needed concerning the stability of the bi-level optimization, the difficulty of training, and why second-order meta-learning is adopted.

Overall, this is a good paper. If the authors can clarify the issues mentioned, I am inclined to accept it.

Reference
1. Transformer for Partial Differential Equations' Operator Learning (https://arxiv.org/abs/2205.13671)
2. GNOT: A General Neural Operator Transformer for Operator Learning (https://arxiv.org/abs/2302.14376)

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for solving PDEs with neural networks in continuous space (and optionally time as well). The authors leverage the success of coordinate-based neural networks (or implicit neural representations – INRs) and formulate the problem as an operator learning one, akin to the Neural Operator family, bypassing the need for discretisation. In particular, the method is based on a typical encode-process-decode strategy: \n\n1. *Encode*: the input condition of the PDE (i.e. the initial value in an initial value problem – IVP) is projected to the input latent vector (via gradient-based optimisation), which is then transformed (via an MLP) to a latent modulation of the input INR (SIREN), such that the INR is fitted to the input condition (auto-decoding). \n2. *Process*: the input latent vector is transformed (via an MLP when projecting to a single output or via a neural ODE when solving over continuous time) to the output one. \n3. *Decode*: the output latent vector is transformed (via an MLP) to a latent modulation of the output INR, which encodes the output signal and can then be queried in arbitrary spatial positions. To promote training stability, the authors employ a two-step process: First the parameters of the input/output INRs are fitted via a meta-learning strategy, similar to Dupont et al., ICML’22, using a shared INR across all input conditions and learning both the parameters of the INR and the latent vectors simultaneously and efficiently. Then, the parameters of the encode-process-decode pipeline are learnt by doing optimisation in the latent space.\n\nThe method is evaluated on several PDE tasks, showing competitive results against other continuous neural solvers.\n\nSTRENGTHS:\n-\t**Clarity and presentation quality**. The paper is well-presented, has a good flow and guides the reader smoothly throughout the important ideas. Also, it is mathematically clean and concise. Finally, it is mostly well-contextualised with the relevant literature.\n-\t**Simplicity and Scalability (inference)**. Although the method is less theoretically motivated compared to Neural Operators, it builds on top of existing tools in the INR literature, making it easy to implement. In addition, forecasting happens in a compressed version of the physical process (a latent vector), which improves scalability.\n-\t**Novelty and Significance**. The paper proposes a new framework for continuous simulations, departing from the Neural Operator family, advancing the field further, in terms of efficiency and performance. The claims are empirically evaluated on a broad spectrum of PDE tasks (from IVPs and dynamics modelling to geometric design and optimisation) showcasing competitive generalisation performance across input conditions, resolutions and time horizons (for trajectory unrolls).\n\nWEAKNESSES:\n- **Connections with Dupont et al., ICML’22**. First off, it is important to acknowledge that the method, from a technical point of view, is largely based on the “functa” concept of Dupont et al., ICML’22. The authors do cite this work, but not as prominently as one would expect. Although the aforementioned paper focuses more abstractly on function representation learning, and less on PDE solvers, the current work is a natural follow-up, so I believe that the connections between the two should be stated more clearly.\n-\t**Efficiency and ease of training**. The training process might be slow since it happens in two steps. Although the second step, might be relatively fast since it happens in the latent space, the first one requires fitting INRs to all possible inputs and outputs in the training set. The meta-learning strategy might potentially alleviate this problem, but the authors have not discussed this matter. Could the authors provide more details on this step? How long does it take? Does it require extensive hyperparameter tuning? How sensitive are the results to different hyperparameters? \n-\t**Potential risk of overfitting (w.r.t. input/output signals not coordinates)**: Experimentally, it seems from the paper that the method generalises well across different input conditions, including the particularly challenging problem of trajectory unrolls. Let me explain why I am a bit surprised by this:\n\n  - I am puzzled about the inductive biases of the encoding process (Eq. (1)). In particular, it is unclear what are the hidden assumptions regarding the input/output signals. Think of a CNN encoder for example – in that case, we would silently assume signal stationarity and shift invariance (due to the application of the same filter across different regions), locality, etc. On the other hand, the corresponding assumptions of the auto-decoder are unclear, even in an intuitive manner. I would appreciate it if the authors could comment on this, and it might be also useful for future readers.\n   - Given the above, I am wondering how well the encoding process, and more importantly the entire pipeline, generalises across different input signals. In particular, in case the encoding process overfits, then small errors in the predictions of the latent codes (especially when unrolling trajectories where errors accumulate), might result in regions of the latent space “unknown” to the decoder, and therefore to unpredictable outcomes. To clarify this, it might be useful to discuss the distributions of the input conditions used. Are they challenging enough? How important is the Z-score normalisation (Appendix B.1.2.)?\n   - My concern about this also comes from the fact that only a latent-space objective is minimised, which in principle does not rule out the possibility of small errors in the latent space to translate to large errors in the signal space (e.g. assume that the INR decoder has a large Lipschitz constant). \n-\t**Empirical evaluation and comparison against relevant methods**: I think there is room for improvement in the experimental section (extra discussion and comparisons) in order to make the claims more convincing. In particular, \n  - It would be helpful to also include discrete neural solver baselines (e.g. based on CNNs, as in Stachenfeld, ICLR’22, or mesh GNNs, as in Pfaff et al., ICLR’21).\n  - In addition, it would be nice to see a more thorough discussion regarding the comparison between continuous neural solvers (MPPDE, Neural Operators etc.). It is unclear to me why and when the current method performs better. For example, intuitively I don’t understand why MPPDE is dependent/overfits on the training grid. Have the authors considered comparing against the Graph/Multipole Graph Neural Operator methods (NeurIPS’20)? Does CORAL have any stronger inductive biases compared to Neural Operators? Could the authors elaborate on the above?\n  - Regarding the dynamics modelling experiments, I have a few questions: (1) Have the authors compared against autoregressive one-step predictors (e.g. as in the MPPDE paper, or in Sanchez-Gonzalez et al., ICML’20)?  Obviously, the NeuralODE formulation allows for evaluation across arbitrary timestamps, but I am wondering if there is any disadvantage in this approach. (2) Have the authors considered comparing to discrete autoregressive solvers, such as the LE-PDE paper (NeurIPS’22)? (3) In certain papers (e.g. MPPDE and Sanchez-Gonzalez et al., ICML’20) the authors use a noise trick to stabilise trajectory unrolls and mitigate error accumulation. I am curious if the authors used it here as well, and if not why?\n\nQUESTIONS:\nThe major questions that I would like to be clarified and discussed are included in the \"Weaknesses\" section. Below, I enlist a few extra minor comments:\n\n**Minor comments**:\n\n-\tCould the authors explain their inference-time findings? Why is CORAL faster than DINO and MPPDE and slower than DeepONet and FNO? Why does CORAL scale batter (almost constant time regardless of the resolution)? Why didn’t the authors include results for FNO across all resolution setups?\n-  Perhaps it would have been beneficial to also test on more challenging setups such as the turbulent regime of Navier-Stokes (potentially to identify limitations and failure cases).\n- A recent paper that improves upon prior work and might be useful to also include as a baseline is the “Clifford neural layers for PDE modelling”, Brandstetter et al., ICLR’23.\n-\tL72: “infinite-dimensional functions“: This should be rephrased - maybe mappings between functions (infinite-dimensional vectors)?\n-\tL76: Regarding DeepONet, if I am not mistaken it is required to have the same observation grid only for training and not for testing.\n-\tL265: RK4 – maybe explain that you are referring to fourth-order Runge-Kutta.\n\nLIMITATIONS:\nThe authors have included a section discussing the limitations of their work and were upfront about them. A few things remain to be clarified (e.g. time and tuning required for training, generalisation across input conditions, and more challenging experimental PDE setups). No foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tried to introduce an implicit neural field-based framework for solving PDEs for different applications on general geometries. The method represents the input and output function spaces as an implicit neural representation. The method consists of a two-stage pipeline: first, the authors train two auto-decoders over the input and output function domains. Then, using paired data, they learn a mapping between the spatial latent code between domains. The paper shows results in initial value problem, dynamics, and predicting PDE solution from different geometry.\n\nI think the paper is tackling a very important problem - generalizing operator and neural field method to solving PDE with different geometry (in the boundary/shape). And I also like the approach to this problem as they leveraged neural fields techniques to achieve discretization-free effect and operator techniques to allow relatively fast influence time. However, the presentation of the paper confuses me in several places (see weakness section) and that holds me back from full acceptance in the first read.\n\nSTRENGTHS:\n- The method is discretization free. It doesn't require the user to discretize the domain, which can have effect on the simulation quality. This method only need to run optimization with auto decoder, which is capable of taking partial observation and predict a continuous function that satisfies these observations.\n- This method has shown advantages compared to many operator methods in that it doesn't require operations such as Fourier transformation which requires regular samples in the geometry. This advantages the method to potentially work with different domain geometry, as well as different observation density.\n\nWEAKNESSES:\n1. Can this method be used for Inverse design? The experiment named \"geometry design\" is a bit misleading as it hints at the designing the geometry (the domain). But to the best of my reading, the geometric design experiment is set-up to predict PDE solution from different geometric, rather than designing the domain geometry. It would be great if the authors can add a note on this in the supplementary.\n\n2. It's not clear to me how does the method actually generalize to different geometry - I might miss something here. For example, how does the method encode where the boundary of the geometry is? It occurs to me that all functions are defined in the ambient space of physics domain. This makes it unclear to me how does the effect of domain boundary on the solution gets encoded. For example, if we have a bunch of samples (x_i, u_0(x_i)) that indicates the initial condition of a IVP problem, but we set two different boundary condition -  one circle these samples with a sphere and the other with a rectangular box. The current method seems to make the same prediction as the latent code is inferred from the set of samples samples (x_i, u_0(x_i)), but I can't believe that's the case in the real world - i.e. the boundary has no effect on the solution.\n\n3. Justification for the requirement of simulated data. The system still requires paired data for training similar to the neural operator, which is more supervision required by the PINN or neural fields methods (which only requires the PDE definition). I wonder what's the way to justify this additional data?\n\n4. Risk of overclaiming. I think \"general geometries\" has become very vaguely define throughout the paper. I'm under the impression that the method is able to solve equation with different boundary condition and different geometry of the boundary. But it's not clear to me how the current method is able to handle different boundary conditions, also it's not clear to me how does the method take into account of the different geometry of the simulation domain. For example, if we have the same initialization function, but different shape, would the latent code we obtained from test-time optimization of the auto-decoder be the same or different? If in test time, we want to change the boundary condition of this problem, how do I specify this? In general, I think the proper claim of the method is that it allows the neural operator method to work with irregular samples, but every trained model might still be restricted to solving the same set of problem (maybe the same boundary geometry) but with different initialization.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nSection 5 discussed limitation", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the author proposed to use neural field in operator learning. In the CORAL model, it first encodes the input into some codes, apply an MLP on the codes, and then decode into the output function. Since the neural field can be continuous evaluated, the CORAL model can be applied to general geometries.\n\nSTRENGTHS:\nThe paper uses neural presentation in operator learning which is a very natural choice and worthy investigation. The proposed method show promising results on time extrapolation and geometry design problem.\n\nWEAKNESSES:\nThe overall framework is quite simple. The encode-decode structure has been studied in PCA basis [1] and Fourier basis [2], which are also applicable for general geometries. Compared to these well-studied conventional basis, neural representation usually use a higher number of parameters, which in the end, make the encoding unstable. \n\nIn the CORAL framework, the latent code is obtained using gradient descent. However it's possible to get very different latent code given the same input. Would such non-uniqueness cause any issue?\n\n[1] Bhattacharya, Kaushik, et al. \"Model reduction and neural networks for parametric PDEs.\" The SMAI journal of computational mathematics 7 (2021): 121-157.\n[2] Fanaskov, Vladimir, and Ivan Oseledets. \"Spectral neural operators.\" arXiv preprint arXiv:2205.10573 (2022).\n\nQUESTIONS:\nWhat is the dimension/size of the latent code?\n\nOne of the major advantage of this CORAL is time extrapolation. Do the authors have some insight why neural representation does better on this task?\n\nIn the experiment, the authors compared CORAL with another neural representation based method DINO. How does CORAL compared to DINO in the model design? Does it also use encoder and decoder? It will be great to address DINO also in the related work section.\n\nIn neural representation, many recent work use context grid and voxels to enhance the representation. Is it possible to compensate CORAL with grid-based representation?\n\nLIMITATIONS:\nThe paper did not address its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}