SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Louis Serrano<sup>1</sup>, Lise Le Boudec<sup>*1</sup>, Armand Kassai Koupai<sup>*1</sup>, Thomas X Wang<sup>1</sup>, Yuan Yin<sup>1</sup>, Jean-Noël Vittaut<sup>2</sup>, Patrick Gallinari<sup>1,3</sup>
$^{1}$  Sorbonne Université, CNRS, ISIR, 75005 Paris, France
2 Sorbonne Université, CNRS, LIP6, 75005 Paris, France
$^{3}$  Criteo AI Lab, Paris, France
{louis.serrano, lise.leboudec, armand.kassai, thomas.wang, yuan.yin, jean-noel.vittaut, patrick.gallinari}@sorbonne-universite.fr
Machine learning approaches for solving partial differential equations require learning mappings between function spaces. While convolutional or graph neural networks are constrained to discretized functions, neural operators present a promising milestone toward mapping functions directly. Despite impressive results they still face challenges with respect to the domain geometry and typically rely on some form of discretization. In order to alleviate such limitations, we present CORAL, a new method that leverages coordinate-based networks for solving PDEs on general geometries. CORAL is designed to remove constraints on the input mesh, making it applicable to any spatial sampling and geometry. Its ability extends to diverse problem domains, including PDE solving, spatio-temporal forecasting, and geometry-aware inference. CORAL demonstrates robust performance across multiple resolutions and performs well in both convex and non-convex domains, surpassing or performing on par with state-of-the-art models.
Modeling physics dynamics entails learning mappings between function spaces, a crucial step in formulating and solving partial differential equations (PDEs). In the classical approach, PDEs are derived from first principles, and differential operators are utilized to map vector fields across the variables involved in the problem. To solve these equations, numerical methods like finite elements, finite volumes, or spectral techniques are employed, requiring the discretization of spatial and temporal components of the differential operators (Morton & Mayers, 2005; Olver, 2014).
Building on successes in computer vision and natural language processing (Krizhevsky et al., 2017; He et al., 2016; Dosovitskiy et al., 2021; Vaswani et al., 2017), deep learning models have recently gained attention in physical modeling. They have been applied to various scenarios, such as solving PDEs (Cai et al., 2021), forecasting spatio-temporal dynamics (de Bézenac et al., 2019), and addressing inverse problems (Allen et al., 2022). Initially, neural network architectures with spatial inductive biases like ConvNets (Long et al., 2018; Ibrahim et al., 2022) for regular grids or GNNs (Pfaff et al., 2021; Brandstetter et al., 2022b) for irregular meshes were explored. However, these models are limited to specific mesh points and face challenges in generalizing to new topologies. The recent trend of neural operators addresses these limitations by modeling mappings between functions, which can be seen as infinite-dimensional vectors. Popular models like DeepONet (Lu et al., 2022) and Fourier Neural Operators (FNO) (Li et al., 2022b) have been applied in various domains. However, they still have design rigidity, relying on fixed grids during training and inference, which limits their use in real-world applications involving irregular sampling grids or new geometries. A variant of
FNO tailored for more general geometries is presented in (Li et al., 2022a), but it focuses on design tasks.
To overcome these limitations, there is a need for flexible approaches that can handle diverse geometries, metric spaces, irregular sampling grids, and sparse measurements. We introduce CORAL, a COordinate-based model for opeRAtor Learning that addresses these challenges by leveraging implicit neural representations (INR). CORAL encodes functions into compact, low-dimensional latent spaces and infers mappings between function representations in the latent space. Unlike competing models that are often task-specific, CORAL is highly flexible and applicable to various problem domains. We showcase its versatility in PDE solving, spatio-temporal dynamics forecasting, and design problems.
- CORAL can learn mappings between functions sampled on an irregular mesh and maintains consistent performance when applied to new grids not seen during training. This characteristic makes it well-suited for solving problems in domains characterized by complex geometries or non-uniform grids.
- We highlight the versatility of CORAL by applying it to a range of representative physical modeling tasks, such as initial value problems (IVP), geometry-aware inference, dynamics modeling, and forecasting. Through extensive experiments on diverse datasets, we consistently demonstrate its state-of-the-art performance across various geometries, including convex and non-convex domains, as well as planar and spherical surfaces. This distinguishes CORAL from alternative models that are often confined to specific tasks.
- CORAL is fast. Functions are represented using a compact latent code in CORAL, capturing the essential information necessary for different inference tasks in a condensed format. This enables fast inference within the compact representation space, whereas alternative methods often operate directly within a higher-dimensional representation of the function space.
Mesh-based networks for physics. The initial attempts to learn physical dynamics primarily centered around convolutional neural networks (CNNs) and graph neural networks (GNNs). Both leverage discrete convolutions to extract relevant information from a given node's neighborhood Hamilton (2020). CNNs expect inputs and outputs to be on regular grid. Their adaptation to irregular data through interpolation (Chae et al., 2021) is limited to simple meshes. GNNs work on irregular meshes (Hamilton et al., 2017; Velicković et al., 2018; Pfaff et al., 2021) and have been used e.g. for dynamics modeling (Brandstetter et al., 2022b) or design optimization (Allen et al., 2022). They typically select nearest neighbors within a small radius, which can introduce biases towards the type of meshes seen during training. In Section 4.2, we show that this bias can hinder their ability to generalize to meshes with different node locations or levels of sparsity. Additionally, they require significantly more memory resources than plain CNNs to store nodes' neighborhoods, which limits their deployment for complex meshes.
Operator learning. Operator learning is a burgeoning field in deep learning for physics that focuses on learning mappings between infinite-dimensional functions. Two prominent approaches are DeepONet (Lu et al., 2021) and Fourier Neural Operator (FNO; Li et al., 2021). DeepONet can query any coordinate in the domain for a value of the output function. However, the input function must be observed on a set of predefined locations, requiring the same observation grid for all observations, for training and testing. FNO is an instance of neural operators (Kovachki et al., 2021), a family of approaches that integrate kernels over the spatial domain. Since this operation can be expensive, FNO addresses the problem by employing the fast Fourier transform (FFT) to transform the inputs into the spectral domain. As a consequence it cannot be used with irregular grids. Li et al. (2022a) introduce an FNO extension to handle more flexible geometries, but it is tailored for design problems. To summarize, despite promising for several applications, current operator approaches still face limitations to extrapolate to new geometries; they do not adapt to changing observation grids or are limited to fixed observation locations. Recently, Li et al. (2023); Hao et al. (2023) explored transformer-based architectures as an alternative approach.
Initial Value Problem
(a) Cylinder
Dynamics modeling
(b) Navier-Stokes
(c) Shallow-Water
Geometry-aware inference
(d) Elasticity
Figure 1: Illustration of the problem classes addressed in this work: Initial Value Problem (IVP) (a), dynamic forecasting (b and c) and geometry-aware inference (d and e).
(e)NACA-Euler
Spatial INRs. Spatial INRs are a class of coordinate-based neural networks that model data as the realization of an implicit function of a spatial location  $x \in \Omega \mapsto f_{\theta}(x)$  (Tancik et al., 2020; Sitzmann et al., 2020b; Fathony et al., 2021; Lindell et al., 2022). An INR can be queried at any location, but encodes only one data sample or function. Previous works use meta-learning (Tancik et al., 2021; Sitzmann et al., 2020a), auto-encoders (Chen & Zhang, 2019; Mescheder et al., 2019), or modulation (Park et al., 2019; Dupont et al., 2022) to address this limitation by enabling an INR to decode various functions using per-sample parameters. INRs have started to gain traction in physics, where they have been successfully applied to spatio-temporal forecasting (Yin et al., 2022) and reduced-order modeling (Chen et al., 2022). The former work is probably the closest to ours but it is designed for forecasting and cannot handle the range of tasks that CORAL can address. Moreover, its computational cost is significantly higher than CORAL's, which limits its application in real-world problems. The work by Chen et al. (2022) aims to inform the INR with known PDEs, similar to PINNs, whereas our approach is entirely data-driven and without physical prior.
3 The CORAL Framework
In this section, we present the CORAL framework, a novel approach that employs an encode-process-decode structure to achieve the mapping between continuous functions. We first introduce the model and then the training procedure.
3.1 Problem Description
Let  $\Omega \subset \mathbb{R}^d$  be a bounded open set of spatial coordinates. We assume the existence of a mapping  $\mathcal{G}^*$  from one infinite-dimensional space  $\mathcal{A} \subset L^2(\Omega, \mathbb{R}^{d_a})$  to another one  $\mathcal{U} \subset L^2(\Omega, \mathbb{R}^{d_u})$ , such that for any observed pairs  $(a_i, u_i) \in \mathcal{A} \times \mathcal{U}$ ,  $u_i = \mathcal{G}^*(a_i)$ . We have  $a_i \sim \nu_a$ ,  $u_i \sim \nu_u$  where  $\nu_a$  is a probability measure supported on  $\mathcal{A}$  and  $\nu_u$  the pushforward measure of  $\nu_a$  by  $\mathcal{G}^*$ . We seek to approximate this operator by an i.i.d. collection of point-wise evaluations of input-output functions through a highly flexible formulation that can be adapted to multiple tasks. In this work, we target three different tasks as examples:  $\bullet$  solving an initial value problem, i.e. mapping the initial condition  $u_0 \dot{=} x \mapsto u(x, t = 0)$  to the solution at a predefined time  $u_T \dot{=} x \mapsto u(x, t = T)$ ,  $\bullet$  modeling the dynamics of a physical system over time ( $u_t \to u_{t + \delta t}$ ) over a given forecasting horizon  $\bullet$  or
prediction based on geometric configuration. At training time, we have access to  $n_{tr}$  pairs of input and output functions  $(a_i, u_i)_{i=1}^{n_{tr}}$  evaluated over a free-form spatial grid  $\mathcal{X}_i$ . We denote  $a|_{\mathcal{X}_i} = (a(x))_{x \in \mathcal{X}_i}$  and  $u|_{\mathcal{X}_i} = (u(x))_{x \in \mathcal{X}_i}$  the vectors of the function values over the sample grid. In the context of the initial value and geometry-aware problems, every sample is observed on a specific grid  $\mathcal{X}_i$ . For dynamics modeling, we use a unique grid  $\mathcal{X}_{tr}$  for all the examples to train the model and another grid  $\mathcal{X}_{te}$  for testing.
Figure 2: Inference for CORAL. First, the model embeds the input function  $a$  without constraints on the locations of the observed sensors into an input latent code  $z_{a}$ , then infers the output latent code  $\hat{z}_{u}$  and finally predicts the output value  $\hat{u}(x)$  for any query coordinate  $x \in \Omega$ . For the grid  $\mathcal{X}$ , we use the vector notation  $a|_{\mathcal{X}} = (a(x))_{x \in \mathcal{X}}, \hat{u}|_{\mathcal{X}} = (\hat{u}(x))_{x \in \mathcal{X}}$ .
3.2 Model
CORAL makes use of two modulated INRs,  $f_{\theta_a,\phi_a}$  and  $f_{\theta_u,\phi_u}$ , for respectively representing the input and output functions of an operator. While  $\theta_a$  and  $\theta_u$  denote shared INR parameters that contribute in representing all functions  $a_i$  and  $u_i$ , the modulation parameters  $\phi_{a_i}$  and  $\phi_{u_i}$  are specific to each function  $a_i$  and  $u_i$ . Given input/output INR functions representation, CORAL then learns a mapping between latent representations inferred from the two INRs' modulation spaces. The latent representations  $z_{a_i},z_{u_i}$  are low dimensional embeddings, capturing within a compact code information from the INRs' parameters. They are used as inputs to hypernetworks  $h_a$  and  $h_u$  to compute the modulation parameters  $\phi_{a_i} = h_a(z_{a_i})$  and  $\phi_{u_i} = h_u(z_{u_i})$ . The weights of the input and output hypernetworks are respectively denoted  $w_a$  and  $w_u$ .
CORAL proceeds in three steps: encode, to project the input data into the latent space; process, to perform transformations in the latent space; and decode, to project the code back to the output function space. First, the input function  $a$  is encoded into the small input latent code  $z_{a}$  using a spatial encoder  $e_{a}: \mathcal{A} \mapsto \mathbb{R}^{d_{z}}$ . Next, a parameterized model  $g_{\psi}: \mathbb{R}^{d_{z}} \mapsto \mathbb{R}^{d_{z}}$  is used to infer an output latent code. Depending on the target task,  $g_{\psi}$  can be as simple as a plain MLP or more complex as for example a neural ODE solver (as detailed later). Finally, the processed latent code is decoded into a spatial function using a decoder  $\xi_{u}: \mathbb{R}^{d_{z}} \mapsto \mathcal{U}$ . The resulting CORAL operator then writes as  $\mathcal{G} = \xi_{u} \circ g_{\psi} \circ e_{a}$ , as shown in Figure 2. The three steps are detailed below.
Encode Given an input function  $a_i$  and a learned shared parameter  $\theta_a$ , the encoding process provides a code  $z_{a_i} = e_a(a_i)$ . This code is computed by solving an inverse problem through a procedure known as auto-decoding, which proceeds as follows. We want to compress into a compact code  $z_{a_i}$  the information required for reconstructing the original field  $a_i$  through the input INR, i.e.:  $\forall x \in \mathcal{X}_i, f_{\theta_a, \phi_{a_i}}(x) = \tilde{a}_i(x) \approx a_i(x)$  with  $\phi_{a_i} = h_a(z_{a_i})$ . See Figure 3a in Appendix B for details.
The approximate solution to this inverse problem is computed as the solution  $e_{a}(a_{i}) = z_{a_{i}}^{(K)}$  of a gradient descent optimization:
$$
z _ {a _ {i}} ^ {(0)} = 0; z _ {a _ {i}} ^ {(k + 1)} = z _ {a _ {i}} ^ {(k)} - \alpha \nabla_ {z _ {a _ {i}} ^ {(k)}} \mathcal {L} _ {\mu_ {i}} \left(f _ {\theta_ {a}, \phi_ {a _ {i}} ^ {(k)}}, a\right); \text {w i t h} \phi_ {a _ {i}} ^ {(k)} = h _ {a} \left(z _ {a _ {i}} ^ {(k)}\right) \text {f o r} 0 \leq k \leq K - 1 \tag {1}
$$
where  $\alpha$  is the inner loop learning rate,  $K$  the number of inner steps, and  $\mathcal{L}_{\mu_i}(v,w) = \mathbb{E}_{x\sim \mu_i}[(v(x) - w(x))^2 ]$  for a measure  $\mu_{i}$  over  $\Omega$ . Note that in practice,  $\mu_{i}$  is defined through the observation grid
$\mathcal{X}_i, \mu_i(\cdot) = \sum_{x \in \mathcal{X}_i} \delta_x(\cdot)$  where  $\delta_x(\cdot)$  is the Dirac measure. Since we can query the INRs anywhere within the domain, we can hence freely encode functions without mesh constraints. This is the essential part of the architecture that enables us to feed data defined on different grids to the model. We show the encoding flow in Appendix B, Figure 4.
Process Once we obtain  $z_{a_i}$ , we can infer the latent output code  $\hat{z}_{u_i} = g_{\psi}(z_{a_i})$ . For simplification, we consider first that  $g_{\psi}$  is implemented through an MLP with parameters  $\psi$ . For dynamics modeling, in Section 4.2, we will detail why and how to make use of a Neural ODE solver for  $g_{\psi}$ .
Decode We decode  $\hat{z}_{u_i}$  with the output hypernetwork  $h_u$  and modulated INR and denote  $\xi_u$  the mapping that associates to code  $\hat{z}_{u_i}$  the function  $f_{\theta_u, \hat{\phi}_{u_i}}$ , where  $\hat{\phi}_{u_i} = h_u(\hat{z}_{u_i})$ . Since  $f_{\theta_u, \hat{\phi}_{u_i}}$  is an INR, i.e. a function of spatial coordinates, it can be freely queried at any point within the domain. We thus have  $\forall x \in \Omega, \hat{u}_i(x) = \xi_u(\hat{z}_{u_i})(x) = f_{\theta_u, \hat{\phi}_{u_i}}(x)$ . See Figure 3b in Appendix B for details.
During training, we will need to learn to reconstruct the input and output functions  $a_{i}$  and  $u_{i}$ . This requires training a mapping associating an input code to the corresponding input function  $\xi_{a}:\mathbb{R}^{d_{z}}\mapsto \mathcal{A}$  and a mapping associating a function to its code in the output space  $e_u:\mathcal{U}\mapsto \mathbb{R}^{d_z}$ , even though they are not used during inference.
3.3 Practical implementation: decoding by INR Modulation
We choose SIREN (Sitzmann et al., 2020b) – a state-of-the-art coordinate-based network – as the INR backbone of our framework. SIREN is a neural network that uses sine activations with a specific initialization scheme (Appendix B).
$$
f _ {\theta} (x) = \boldsymbol {W} _ {L} \left(\sigma_ {L - 1} \circ \sigma_ {L - 2} \circ \dots \circ \sigma_ {0} (x)\right) + \boldsymbol {b} _ {L}, \text {w i t h} \sigma_ {i} (\eta_ {i}) = \sin \left(\omega_ {0} \left(\boldsymbol {W} _ {i} \eta_ {i} + \boldsymbol {b} _ {i}\right)\right) \tag {2}
$$
where  $\eta_0 = x$  and  $(\eta_i)_{i\geq 1}$  are the hidden activations throughout the network.  $\omega_0\in \mathbb{R}_+^*$  is a hyperparameter that controls the frequency bandwidth of the network,  $W$  and  $\pmb{b}$  are the network weights and biases. We implement shift modulations (Perez et al., 2018) to have a small modulation space and reduce the computational cost of the overall architecture. This yields the modulated SIREN:
$$
f _ {\theta , \phi} (x) = \boldsymbol {W} _ {L} \left(\sigma_ {L - 1} \circ \sigma_ {L - 2} \circ \dots \circ \sigma_ {0} (x)\right) + \boldsymbol {b} _ {L}, \text {w i t h} \sigma_ {i} (\eta_ {i}) = \sin \left(\omega_ {0} \left(\boldsymbol {W} _ {i} \eta_ {i} + \boldsymbol {b} _ {i} + \phi_ {i}\right)\right) \tag {3}
$$
with shared parameters  $\theta = (\pmb{W}_i, \pmb{b}_i)_{i=0}^L$  and example associated modulations  $\phi = (\phi_i)_{i=0}^{L-1}$ . We compute the modulations  $\phi$  from  $z$  with a linear hypernetwork, i.e. for  $0 \leq i \leq L-1$ ,  $\phi_i = V_i z + c_i$ . The weights  $V_i$  and  $c_i$  constitute the parameters of the hypernetwork  $w = (V_i, c_i)_{i=0}^{L-1}$ . This implementation is similar to that of Dupont et al. (2022), which use a modulated SIREN for representing their modalities.
We implement a two-step training procedure that first learns the modulated INR parameters, before training the forecast model  $g_{\psi}$ . It is very stable and much faster than end-to-end training while providing similar performance: once the input and output INRs have been fitted, the training of  $g_{\psi}$  is performed in the small dimensional modulated INR  $z$ -code space. Formally, the optimization problem is defined as:
$$
\underset {\psi} {\arg \min } \mathbb {E} _ {a, u \sim \nu_ {a}, \nu_ {u}} \| g _ {\psi} (\tilde {e} _ {a} (a)) - \tilde {e} _ {u} (u) \| ^ {2}
$$
$$
\text {s . t .} \tilde {e} _ {a} = \underset {\xi_ {a}, e _ {a}} {\arg \min } \mathbb {E} _ {a \sim \nu_ {a}} \mathcal {L} (\xi_ {a} \circ e _ {a} (a), a) \tag {4}
$$
$$
\text{and}\tilde{e}_{u} = \operatorname *{arg  min}_{\xi_{u},e_{u}}\mathbb{E}_{u\sim \nu_{u}}\mathcal{L}(\xi_{u}\circ e_{u}(u),u)
$$
Note that functions  $(e_u,\xi_u)$  and  $(e_a,\xi_a)$  are parameterized respectively by the weights  $(\theta_u,w_u)$  and  $(\theta_{a},w_{a})$  , of the INRs and of the hypernetworks. In Equation (4), we used the  $(e_u,\xi_u)$  &  $(e_a,\xi_a)$  description for clarity, but as they are functions of  $(\theta_{u},w_{u})$  &  $(\theta_{a},w_{a})$  , optimization is tackled on the latter parameters. We outline the training pipeline in Appendix B, Figure 5. During training, we
constrain  $e_{u}, e_{a}$  to take only a few steps of gradient descent to facilitate the processor task. This regularization prevents the architecture from memorizing the training set into the individual codes and facilitates the auto-decoding optimization process for new inputs. In order to obtain a network that is capable of quickly encoding new physical inputs, we employ a second-order meta-learning training algorithm based on CAVIA (Zintgraf et al., 2019). Compared to a first-order scheme such as Reptile (Nichol et al., 2018), the outer loop back-propagates the gradient through the  $K$  inner steps, consuming more memory as we need to compute gradients of gradients but yielding higher reconstruction results with the modulated SIREN. We experimentally found that using 3 inner-steps for training, or testing, was sufficient to obtain very low reconstruction errors for most applications.
To demonstrate the versatility of our model, we conducted experiments on three distinct tasks (Figure 1): (i) solving an initial value problem (Section 4.1), (ii) modeling the dynamics of a physical system (Section 4.2), and (iii) learning to infer the steady state of a system based on the domain geometry (Section 4.3) plus an associated design problem in Appendix D. Since each task corresponds to a different scenario, we utilized task-specific datasets and employed different baselines for each task. This approach was necessary because existing baselines typically focus on specific tasks and do not cover the full range of problems addressed in our study, unlike CORAL. We provide below an introduction to the datasets, evaluation protocols, and baselines for each task setting. All experiments were conducted on a single GPU: NVIDIA RTX A5000 with 25 Go. Code will be made available.
4.1 Initial Value Problem
An IVP is specified by an initial condition (here the input function providing the state variables at  $t = 0$ ) and a target function figuring the state variables value at a given time  $T$ . Solving an IVP is a direct application of the CORAL framework introduced in Section 3.2.
Datasets We benchmark our model on two problems with non-convex domains proposed in Pfaff et al. (2021). In both cases, the fluid evolves in a domain - which includes an obstacle - that is more densely discretized near the boundary conditions (BC). The boundary conditions are provided by the mesh definition, and the models are trained on multiple obstacles and evaluated at test time on similar but different obstacles.  $\bullet$  Cylinder simulates the flow of water around a cylinder on a fixed 2D Eulerian mesh, and is characteristic of incompressible fluids. For each node  $j$  we have access to the node position  $x^{(j)}$ , the momentum  $w(x^{(j)})$  and the pressure  $p(x^{(j)})$ . We seek to learn the mapping  $(x,w_0(x),p_0(x))_{x\in \mathcal{X}}\to (w_T(x),p_T(x))_{x\in \mathcal{X}}$ .  $\bullet$  Airfoil simulates the aerodynamics around the cross-section of an airfoil wing, and is an important use-case for compressible fluids. In this dataset, we have in addition for each node  $j$  the fluid density  $\rho (x^{(j)})$ , and we seek to learn the mapping  $(x,w_0(x),p_0(x),\rho_0(x))_{x\in \mathcal{X}}\to (w_T(x),p_T(x),\rho_T(x))_{x\in \mathcal{X}}$ . For both datasets, each example is associated to a mesh and the meshes are different for each example. For Airfoil the average number of nodes per mesh is 5233 and for Cylinder 1885.
Evaluation protocols Training is performed using all the mesh points associated to an example. For testing we evaluate the following two settings. • Full, we validate that the trained model generalizes well to new examples using all the mesh location points of these examples. • Sparse We assess the capability of our model to generalize on sparse meshes: the original input mesh is down-sampled by randomly selecting  $20\%$  of its nodes. We use a train, validation, test split of 1000 / 100 / 100 samples for all the evaluations.