Continuous PDE Dynamics Forecasting with Implicit Neural Representations
Authors: Yuan Yin
Matthieu Kirchmeyer
Jean-Yves Franceschi
Alain Rakotomamonjy
patrick gallinari
Venue: ICLR 2023 notable top 25% (2023)
Keywords: spatiotemporal forecasting
Partial Differential Equations
PDEs
Implicit Neural Representations
INRs
continuous models
generalization
dynamical systems
physics

Abstract: Effective data-driven PDE forecasting methods often rely on fixed spatial and / or temporal discretizations. This raises limitations in real-world applications like weather prediction where flexible extrapolation at arbitrary spatiotemporal locations is required. We address this problem by introducing a new data-driven approach, DINo, that models a PDE's flow with continuous-time dynamics of spatially continuous functions. This is achieved by embedding spatial observations independently of their discretization via Implicit Neural Representations in a small latent space temporally driven by a learned ODE. This separate and flexible treatment of time and space makes DINo the first data-driven model to combine the following advantages. It extrapolates at arbitrary spatial and temporal locations; it can learn from sparse irregular grids or manifolds; at test time, it generalizes to new grids or resolutions. DINo outperforms alternative neural PDE forecasters in a variety of challenging generalization scenarios on representative PDE systems.

Published as a conference paper at ICLR 2023
CONTINUOUS PDE DYNAMICS PORECASTING WITH IMPLICIT NEURAL REPRESENTATIONS
Yuan Yin
Matthieu Kirchmeyer
Jean- Yves Franceschi
Alain Rakotomamonjy
Patrick Gallinari!?
'Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
"Criteo Al Lab, Paris
ABSTRACT
Effective data-driven PDE forecasting methods often rely fixed spatial and/or temporal discretizations. This raises limitations in real-world applications like weather prediction where flexible extrapolation at arbitrary spatiotemporal loca- tions is required. We address this problem by introducing a new data-driven ap- proach, DINo, that models PDE's flow with continuous time dynamics of spa- tially continuous functions. This is achieved by embedding spatial observations independently of their discretization via Implicit Neural Representations latent space temporally driven by learned ODE. This separate and flexible treat- ment of time and space makes DINo the first data-driven model to combine the following advantages. It extrapolates at arbitrary spatial and temporal locations; it can learn from sparse irregular grids or manifolds; at test time, it generalizes to new grids resolutions. DINo outperforms alternative neural PDE forecasters in a variety of challenging generalization scenarios on representative PDE systems.
small
INTRODUCTION
Modeling the dynamics and predicting the temporal evolution of physical phenomena is paramount in many fields, e.g. climate modeling, biology, fluid mechanics and energy (Willard al., 2022). Classical solutions rely on a well-established physical paradigm: the evolution is described by differential equations derived from physical first principles, and then solved using numerical analysis tools, e.g. finite elements, finite volumes or spectral methods (Olver, 2014). The availability of large amounts of data from observations or simulations has motivated data-driven approaches to this problem (Brunton & Kutz, 2022), leading to a rapid development of the field with deep learning methods. The main motivations for this research track include developing surrogate or reduced order models that can approximate high-fidelity full order models reduced computational costs (Kochkov et al., 2021), complementing classical solvers, e.g. to account for additional components of the dynamics (Yin al., 2021), improving low fidelity models (De Avila Belbute-Peres al., 2020).
Most of these attempts rely on workhorses of deep learning like CNNs (Ayed et al., 2020) or GNNs (Li et al., 2020; Pfaff et al., 2021; Brandstetter al., 2022). They all require prior space discretization either on regular or irregular grids, such that they only capture the dynamics on the train grid and cannot generalize outside it. Neural operators, a recent trend, learn mappings between function spaces (Li et al., 202 lb; Lu et al., 2021) and thus alleviate some limitations of prior discretization approaches. Yet, they still rely on fixed grid discretization for training and inference: e.g., regular grids for Li et al. (2021h) 1)) or a free-form but predetermined grid for Lu et al. (2021). Hence, the number and/ or location of the sensors has to be fixed across train and test which restrictive in many situations (Prasthofer et al., 2022). Mesh-agnostic approaches for solving canonical PDEs (Partial Differential Equations) are another trend (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018). contrast to physics-agnostic grid-based approaches, they aim at solving known PDE usual solvers do, and cannot cope with unknown dynamics. This idea was concurrently developed for computer graphics, e.g. for learning 3D shapes (Sitzmann al., 2020; Mildenhall et al., 2020; Tancik al., 2020) and coined as Implicit Neural Representations (INRs). When used as solvers, these methods can only tackle a single initial value problem and are not designed for long-term forecasting outside the training horizon.
"qqull contribution

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of data-driven approaches spatiotemporal PDE forecasting.
Model
Reference
1. PDE-agaostic prediction on new initial conditions
Train/ test space gnd independenee
3. Evalcation at unobserved rial locations
4. Frce-foom sparial domain (caanfood irrgulat mesh))
5. Time comiiuuus
6. Time extrapolation
NODE MP-PDE
Chen al. (2018)
x
K
Discrete
Brandsteterr (2022)
Lietal, (2021a)
/
Openator
DeepONet
La et al. (2021)
x
/
x
INKS
PINN.
Raissi (2019)
x
5
/
3
x
DINO
Qurs
Because of these limitations, none of the above approaches can handle situations encountered in many practical applications such as: different geometries, e.g. phenomena lying on Fuclidean plane or an Earth-like sphere; variable sampling, c.g. irregular observation grids that may evolve at train and test time as in adaptive meshing (Berger & Oliger, 1984); scarce training data, e.g. when observations only available at few spatiotemporal locations; multi-scale phenomena, e.g. in large scale- dynamics systems as climate modeling, where integrating intertwined subgrid scales a.k.a. the closure problem is ubiquitous (Zanna & Bolton, 2021). These considerations motivate the development of new machine learning models that improve existing approaches on several of these aspects.
In work, we aim forecasting PDE-based spatiotemporal physical processes with versatile model tackling the aforementioned limitations. We adopt an agnostic approach, i.e. not assuming any prior knowledge on the physics. We introduce DINo (Dynamics-aware Implicit Neural representa- tions), a model operating continuously in space and time, with the following contributions.
Continuous flow learning. DINo aims at learning the PDE's flow to forecast its solutions, in continuous manner so that it can be trained on any spatial and temporal discretization and applied to another. To this end, DINo embeds spatial observations into a small latent space via INRs; then models continuous- time evolution by learned latent Ordinary Differential Equation (ODE). Space-time separation. To efficiently encode different sequences, we propose a novel INR parame- terization, amplitude modulation, implementing a space-time separation of variables. This simplifies the learned dynamics, reduces the number of parameters and greatly improves performance.
Spatiotemporal versatility. DINo combines the benefits of prior models; cf. Table 1. It tackles new sequences via its amplitude modulation. Sequential modeling with an ODE makes it extrapolate to unseen times within or beyond the train horizon. Thanks to INRs' spatial flexibility, it generalizes to new grids or resolutions, predicts at arbitrary positions and handles sparse irregular grids or manifolds. Empirical validation. We demonstrate DINO's 's versatility and state-of-the-ard performance versus prior neural PDE forecasters, representative of grid, operator and INR-based methods, via thorough experiments on challenging multi-dimennionall PDEs in various spatiotemporal generalization settings.
2 PROBLEM DESCRIPTION
Problem setting. We aim at modeling, via data-driven approach, the temporal evolution of continuous fully-observed deterministic spatiotemporal phenomenon. It is described by trajectories v: R + V in set I; we use U/ - vtt) E V. We focus on Initial Value Problems, where only e at any time is required to infer By for t' > Hence, trajectories share the same dynamics but differ by their initial condition vo E V. R is the temporal domain and V is the functional space of the form Ω + R", where Ω IRP is a compact spatial domain and n the number of observed values. In other words, Vi is a spatial function of a E Ω, with vectorial output uy (xr) E IR"; cf. examples of Section 1. To this end, we consider the setting illustrated in Figure We observe a finite training set of trajectories D with free-form spatial observation grid X C Ω and on discrete times f E T C [o, T". At test time, we are only given a new initial condition do. with observed values vola; on a new observation grid Xg, potentially different from Xr Inference is performed on both train and test trajectories given only the initial condition, on a new free-form grid X" C Ω and times E T' C T'). Inference grid X comprises observed positions (respectively Xu and Xss for train and test trajectories) and unobserved positions corresponding to spatial interpolation. Note that the inference temporal horizon is larger than the train one: T < T'. For simplicity, In-s refers to data in X' on the observation grid (A, for train/ X₂ for test), Out-s to data in X' outside the observation grid; In-r refers to times within the train horizon T C [0, 7). and Out-t to times in T' T (T, T"), beyond T, up inference horizon T".
2

---

Published as a conference paper at ICLR 2023
In-t
Out-t
Fly
train
In-s X
In-s Ax
test
Out-s 52 (
Out-s 12
(Left) We represent time contexts. The train trajectory consists of training snapshots (m), observed in a train interval [0. 7] denoted In-t. The line (-) in continuation is forecasting of this trajectory beyond In-f, in (T,T") T denoted Out-r. The line below (-, test) is a forecasting from a new initial condition vo (m) on In-f and Out-1. (Middle and right) We illustrate spatial contexts. (Middle) Dots (o) correspond to the train observation grid Xrr. denoted In-s. Out-s denotes the complementary domain Ω Xr- (Right) New test observation grid Xg. used an initial point for forecasting (left).
Figure I:
Evaluation scenarios. The desired properties in Section I call for spatiotemporally continuous forecasting models. We select six criteria that our approach should meet; column titles of Table 1. First, the model should be robust to the change of initial condition vo. i.e. generalize to test trajectories (col. 1). Second, it should extrapolate beyond the train conditions: in space, on a test observation grid that differs from the train one, i... X" Nss # Xy (In-s) (col. 2), and outside the observed train and test grid, i.c. on X' \ Ag. X' X'), Xg (Out-s, col. 3); in time, between train snapshots (col. 5) and beyond the observed train horizon T (Out-1, col. 6). Finally, should adapt free-form spatial domains, i.e. to various geometries (e.g. manifolds) or irregular grids (col. 4). See also Figure 1.
Objective. To satisfy these requirements, learn the flow o of the physical system:
(1) Learning the flow a common strategy in sequential models to better generalize beyond the train time horizon. Yet, far, it has always been learned with discretized models, which poses generalization issues violating our requirements. We describe these issues in Section 3.
Φ: (V x RR + V,
(U,, r) (v:) 12+7 Ye I,t
RELATED WORK
We review current data-driven approaches for PDE modeling and the representative methods listed in Table 1. We express the forecasting rule using the notations in Eq. (1): is an arbitrary time; is an arbitrary time interval; št is fixed, predetermined time interval (as a model hyperparameter).
Sequential discretized models. Most sequential dynamics models are learned on fixed observed grid Xr and use discretized models, e.g. CNN or GNN to process the observations. CNNs require observations regular grid but can be extended to irregular grids through interpolation (Chae et al., 2021). GNNs are more flexible as they handle irregular grids, at an additional memory and computational cost. Yet, prediction GNÑs, as these discretized models are biased towards the training grid Xy.. as we later show in Section 5. We distinguish two types of temporal models which both extrapolate beyond the train horizon due to their sequential nature. Autoregressive models Ve 1 + Ve+6t x (Long et al., 201 8; de Bézenac et al., 2018; Pfaff et al., 2021; Brandstetter et al., 2022) predict the sequence from only at fixed time increments öt and not in between. Time-continuous extensions using numerical solvers 1ellx>7] T) + Vt+rlx (Yin et al., 2021; lakovlev al., 2021) solve this limitation as they provide (v.lx>77
grids X' * Xx fails experimentally for both CNNs and
prediction at arbitrary times, thus remove dependency on the time discretization.
Operator learning. Recently, operator-based models aim finding a parameterized mapping be- tween functions. They define in theory ppace-continuous models. First, neural operators (Kovachki et al., 2021) attempt to replace standard convolution with continuous alternatives. Fourier Neural Op- erator (FNO, Li et al., 2021b) applies convolution in the spectral domain via Fast Fourier Transforma- tion (FFT). Graph Neural Operator (GNO, Li et al., 2020) performs convolution on local interaction grid described by graph. Second, DeepONet (Lu et al., 2021) 1) uses a coordinate-basse neural network to output prediction at arbitrary time and space locations given a function observed on a fixed grid. Three types of temporal models were used for operators with some limitations. The standard ap- proach, Up + Vt models the output at a given time E [0. T] within the train horizon (Li et al., 2020). A sequential extension, U,4 +> Ut+ was proposed in Li et al. (2021a). Finally, time-continuous
3

---

Published as a conference paper at ICLR 2023
inference
training
Ol
(ār)) dr
Ott
&dyn
Neural ODE
Neural ODE
Ogg
Olo
Ox
Modulated
Modulated
Modulatod
Modulated INR
INR
Edoc
dec
Lde
Xix
X1x
Vol.x.
Ur
Figure 2: Proposed DINO model. Inference (left): given new initial condition observed on grid Xis, vo,x, forecasting amounts decoding or 0r, by unrolling dg with time-continuous ODE dynamics model fu. Train (right): given an observation grid N2 and space-continuou decoder Dø. ag learned by auto-decoding s.t. (a(a)) x velx, its evolution then modeled with fu-
version Uo + (t E [o, + vz) in DeepONet propose a solution at arbitrary time and space loca- tions. The first and third approaches not designed to generalize beyond the train horizon, i.e. when t t> T as they are not sequential. The second solves this limitation but is only able to predict solutions from t fixed time increments of öt and not in- in-between Furthermore, all existing approaches make restrictive assumptions on the space discretization. They lack flexibility when encoding spatial obser- vations: FNO limited uniform Cartesian observation grids due FFT, and while one concurrent follow-up alleviates this issue (Li et al., 2022), it still cannot perform predictions tial locations; GNO does not adapt well to changing observation grids as for the GNN-based models in the previous paragraph; DeepONet is limited input observations fixed observation locations. The latter are chosen at random spatial positions but should remain fixed throughout training and testing.
unobserved spa-
Spatiotemporal INRs.
Another class of models relies on coordinate- based neural networks, called Implicit Neural Representations (INRs, Sitzmann et al., 2020; Fathony et al., 2021; Tancik et al., 2020). These space-continuous models share similar objective as operators, despite constituting separate research field. INRs for spatiotemporal data take time as an input along spatial coordinates. Physics-informed neural networks (PINNs, Raissi al., 2019) use this formulation solve PDEs, yet are limited to a single known differential equation and a set of initial and boundary conditions. Fresca et al. (2020) and Chen al. (2023) combine INRs with reduced order models. Extensions for multi-sequee sequence learning, e.g. for video generation (Yu et al., 2022; Skorokhodov et al., 2022) or compression (Chen al., 2021), learn a latent conditioning variable from an initial condition vo, i.e. take the form 0o + (t E [0,T] T] + v2). Interestingly, these models can predict at arbitrary time t the train horizon without unrolling a sequential model up to Yet, as they only learn mappings from an initial condition train conditions, as we show Section 5. DINO is a new instance of spatiotemporal INR which solves this limitation via time-continuous dynamics model of the underlying flow, (vz, T) + Pt47-
to function of time in the train domain, they fail to predict beyond
MODEL
We present DINO, the first space space/time-contimnee time-continuous model that tackles all prediction tasks of Section 2, without the above limitations. We specify DINO's inference procedure (Section 4.1), illustrated in Fig- 2 (left), then introduce each of its components (Section 4.2) and how they trained (Section 4.3, Figure (right)). Finally, we detail our implementation based on amplitude modulation, a novel INR parameterization for spatiotemporal data which performs separation of variables (Section 4.4).
4.1 INFERENCE MODEL
As explained in Section 2, we aim at estimating the flow in Eq. (1), so that our model can be trained observed grid X and perform inference given new one Xgs both possibly irregular. To this end, we leverage a space- and time-continuous formulation, independent of a given data discretization. At inference, DINO starts from single initial condition un E V and uses flow to
4

---

Published as a conference paper at ICLR 2023
forecast its dynamics. DINo first embeds spatial observations from Bo into latent vector dg of small dimension da via an encoder of spatial functions E V - Rd (ENC). Then, it unrolls a latent time- continuous dynamics model fa Rd - I given this initial condition (DYN). Finally, it decodes latent vectors via decoder D3: I2da - y into a spatial function (DEC). At any time I, D takes input Or and outputs function ūy f2 R This results in the following model, illustrated in Figure (left) and whose components are detailed in Section 4.2:
(DYN) do dt
(ENC) ox = (vo):
fo(()))
(DEC) 1I, 62 Do(at).
4.2 COMPONENTS
Encoder: = Eyp(r)) (wr). The encoder computes a latent vector or given observation U, at any time is used in two different contexts, respectively for train and test. At train time, given observed trajectory uy = {2' JeT, it will encode any U2 into (see Section 4.3). At inference time, only vo is available, and then only og is computed to be used as initial value for the dynamics. Given the decoder Dø. Oy is a solution to the inverse problem (2((t) = et. We solve this inverse problem with auto- decoding (Park et al., 2019). Denoting (do Otz et) = ( -vll2 the decoding loss where |l-I12 the euclidean norm of a function and K the number of update steps, auto-decoding defines E as:
(v.) = at where af = azs Ak: > 0, k+1 = af - 77 Vo. loo (oo (6, af; vs) and yp = Φ. (3)
In practice, we observe discretization (Ku, Xas) and accordingly approximate the norm in lace as in Eq. (6). Compared to auto-encoding, auto-decoding underfits less (Kim et al., 2019) and is more flexible: without requiring specialized encoder architecture, handles free-formed (irregular or on a manifold) observation grids as long as the decoder shares the same property.
Decoder: 0N = Do(a)). We define a flexible decoder using a coordinate- based INR network with parameters conditioned on at,, An INR la: Q - RR" is a space- cpace-continuous model parameterized by 0 E R defined on domain Ω. approximates functions independently of the observation grid, e.g. handles irregular grids and changing observation positions unlike FNO and DeepONet. Thus, it constitutes a flexible alternative to operators suitable to auto-decoding. To implement the conditioning of the INR's parameters, we use a hypernetwork (Ha et al., 2017) hs: Rd. + Rd, as illustrated in Figure It generates high- dimensional parameters 0, Rd of the INR given the low-dimensional latent vector a2 E Rd Hence, the decoder D&. parameterized by Φ, is defined as:
α
h
modulation
INR
I8
Va E2, (x))) = (a)) A (x).
(4)
The decoder's predictions at all spatial locations æ S thus all depend on Or- We provide further details on the precise implementation in Section 4.4.
((x)
21
Dynamics model: do: = (at). Finally, the dynamics model fø: TR.
de
Rdd defines a flow via an ODE in the latent space. The initial condition can be defined at any time t by encoding with E,p the corresponding input function et-
22
Figure 3: Decod-
Overall flow. Combined altogether, our components define the following flow in the input space that can approximate the data flow & in Eq. (1):
ing via INR Eq. (4).
v(t,T), r),
(u,,7)+ + D (E,(o)) S.+T fu(ar) where a, = (v.)
(5)
To summarize, DINO defines a time-continuous latent temporal model with a space-continuous emission function Da, combining the flexibility of space and time continuity. This is fully novel to our knowledge, as prior latent approaches are discretized (cf. Fraccaro (2018) for state-space models).
4.3 TRAINING
We present the training procedure, illustrated in Figure 2 (right), of the previous components. We use two-stage optimization process, close to recent work in video prediction (Yan et al., 2021). Given the train sequences D, we first use auto-decoding to obtain the latent vectors a = {aj HET,eE and the decoder parameters é. We then learn the parameters of the dynamics by modeling the latent flow
5

---

Published as a conference paper at ICLR 2023
over az' Vo D. We detail this procedure in Appendix D.I, which can be formalized as two-stage optimization problem that we solve in parallel without inducing training instability (ef. Appendix D.2):
ming
Layn (1/, aT) _ EEED,,E7/la - (az + fo fo (a)
(6)
s.t. oc, Φ = arg min
lace aT) - EVED, -ueD (o(a))))))
4.4 DECODER IMPLEMENTATION VIA AMPLITUDE-MODULATE INRS
We now specify our implementation of decoder D& in Eq. (4). This includes the definition of the INR architecture Is and of the hypernetwork hgg We introduce for the latter a new method called amplitude modulation, which implements space-time separation of variables.
To as FourierNet. We implement I as a FourierNet, state-of-the-ard INR architecture, which instantiates Multiplicative Filter Network (MFN, Fathony et al., 2021). A FourierNet relies on the recursion in Eq. (7), where z E Ω is an input spatial location, (x) is the hidden feature vector at layer 1 for x and wwm (I) = (o)) sin(w((xx)] is Fourier basis:
2 (x) (2),(()),, 20 (x)
(z) W (1(1))))((1)(0 + b(6-1),
(7)
(((111)-((-1)(( (x)) ((-1) swih (2)) for l E
1].
where we fix W()) = 0. (10) = sw00 (x) = x and o is the Hadamard product. Denoting W = [W , b= [60)) . - [ we fit a FourierNet to an input function v observed on a grid X' by learning {W, b, w) s.t. Va E (x) = u(r). In practice, we observe that fixing w uniformly sampled performs similarly as learning them, so we exclude them from training parameters.
FourierNets are interpretable, a property we leverage to separate time and space via amplitude modulation. Fathony et al. (2021) show that for some M > L E N, there exist a set of coefficients (c.")) (o Jm= that depend individually on (W, bj as well as a set of parameters (m M= that depend individually on those of the filters w the jth dimension of 2 2(1) can be expressed as:
2)
(1) (x) (z)= -M (7mh) 8m)( (x) + bias.
(m))
(8)
Eq. (8) involves a basis of spatial functions (sym0)) J.M evaluated on x and the amplitudes of this basis (cj (m)) J. Note that Eq. (8) can be extended to other choices of swa (Fathony et al., 2021).
as amplitude modulation. generates the INR's parameters 0, given a to model target input function Ey- We implement h as elementwise shift and scale transformations (FiLM, Perez, et al., 2018) of the linear layers parameters W, b (excluding those of the filters w).
00%
10/000
14
((m) only depend on time while the basis functions s,, only depend space: this corresponds to modeling assumption of separation of variables (Ie Dret & Lucquin, 2016) in e. We call our technique amplitude modulation. In practice, as Dupont et al. (2022), we consider latent shift transformations (Figure 4), detailed in Eq. (9). Eq. (9) extends Eq. (7) by introducing shift term /4; (1-1) at each layer 1, defined as 14;
Then, in Eq. (8), amplitudes of
611
W
# (sx)
6
(00)))
(I-1) = wi(l1)) where
(879
6188
W" = -[W*(-1)]--] 1m1 is another weight matrix:
21 (())
(0)(())))((((((0 (x)++(011 + 11 11) o swes (x).
(9)
(r(xx)
The INR's 's parameters are defined as ha(a)) = [W;b + Wa where の = (W,6, 6, W'] are his parameters. Thus, amplitude modula- tion separates time and space. We show in Table 5 that it significantly improves performance, particularly time extrapolation.
Figure Amplitude modula- tion - Eq. (9). 21' _((-1) is input the 1th linear layer and com- bined with the spatial basis $y( via Hadamard product.
EXPERIMENTS
We assess the spatiotemporal versatility of DINo. following Section 2. We introduce our experimental setting (Section 5.1), which includes a variety of challenging PDE datasets, state-of ofte--of-hhear baselines and forecasting tasks. Then, present and comment the experimental results (Section 5.2).

---

Published as a conference paper at ICLR 2023
Table 2:
Space and time generalization. Train and test observation grids are equal and subsampled from an uniform x4 grid, used for inference. We report MSE (1) on the inference time interval T' divided within training horizon (In-t, T) and beyond (Out-t, outside T) across subsampling ratios.
Navier-Stokies
Wave
Model
Train
Text
Train
Test
Out-t
subsampling ratio
Discrete
[-MP-PDE
8,1548-3
8. 8.1662-3
7.926c-3
8,225t-3
7.055p-4
7.097c-4
1,138E-3
11166-3
Operator
DeepONet
3,330E-3
7.370E-3
1.346E-2
1.408e-22
8.331E-4
9.295E-3
1.692E-2
3.256E-2
SIREN
8,7416-3
L
4.303c-2
2.1266-1
2.7380-3
1.818--2
3,339c-2
6.9648-2
DINO
1.029--3
1665E-3
1.3268-3
18813-3
4.088E-5
4.1218-5
6,415--5
7.392k-5
25%
subsampling ratio
Discrete Operator
[--P--DDE
3,135 4
7.2450-4
3,476c-4
7.688c-4
3.2930-5
1.100-4
5,1420-5
1,545-44
DeepONet
9,0168
5,9368-3
9.376E-3
1.3288-2
5.722E-4
1061E-2
1.757E-2
3.221E-2
SIREN
5,1806-3
2.175m-1
2.4360-1
6-
8.9958-4
1,2920-2
1,7830-2
5.1430-2
INR
DINo
1.000-4
4.504E
2.6468-4
5.951E
3,9492-6
4.4362-6
L089E-5
1.174E-5
100%
subsampling ratio
CNODE
2.3196-2
9.652n-2
2.3050-2
1,143--1
2.337D-5
5,2800-4
3.0576-5
n.288m-4
Discrete
MP-PDE
1.140E-4
5.5008-4
1.785E-4
5,8568
1.718--7
1993E-5
9.256E-7
4.261E-5
MNO
3.1900-5
8.6788-4
2.7638-4
8.940--4
9.381
4.8908-
1.9930-4
6.128n-3
Operator
DoepONet
1.3758-3
6.573n-3
9,704E-3
1.2448-2
6.431E-4
1.293E-2
1.847E-2
3.317E-2
SIREN
1,066E-3
4.3368-1
3.874E-1
1,03760
3.674E-4
9.956E-3
3.013E-2
7.842E-2
MFN
1-651=-3
1.037H0
2.106E-1
1,09900
1.408-44
1.7638-1
4.735E-3
2.2748-1
INR
DINo (no sep.)
3.2356
1. 1.5936-3
7.850e-4
18896-3
2.6418-6
4.081R-5
5,9776-5
2.979H-4
DINo
8,339B-5
3.115E
2.0920-4
4.311--4
3.309H-6
3,506E-6
9.495E-6
9.946E-6
5.1 EXPERIMENTAL SETTING
Datasets. We consider the following PDEs defined over a spatial domain 22 with further details in H² u ar2 = c² Au. u is the displacement w.r.t. the rest position and is the traveling speed. We consider its first-order form, so that Vt (ue, dul has a two-dimensional output (n 2). 2D Navier Stokes (Navier-Stokee Stokes, at -vv
Appendix C. 2D Wave equation (Wave) is a second-order PDE
0t
V u, Vu where u is the velocity field and v the vorticity. y is the viscosity and f is constant forcing term; 1. 3D Spherical shallow water (Shallow- Water, Galewsky al., 2004): involves the vorticity w, tangent to the sphere's 's surface, and the thickness of the fluid h. The input is (w,, he): 2.
1851) corresponds an incompressible fluid dynamics du
Baselines. We reimplement representative models from Section and Table and adapt them to our multi-dimensional datasets. CNODE (Ayed et al., 2020) combines a CNN and an ODE solver to handle regular grids. MP-PDE (Brandstetter et al., 2022) uses a GNN to handle free-formed grids, yet is unable to predict outside the observation grid. We developed interpolative extension, I-MP-PDE, to handle this limitation; it performs bicubic interpolation on the observed grid and training is done of FNO (Li et al., 2021b) for regular grids; can be evaluated on new uniform grids. DeepONet (Lu al., 2021), considered autoregressively (Wang & Perdikaris, 2021) where remove time from the trunk net's input, can be evaluated on new spatial locations without interpolation. SIREN (Sitzmann al., 2020) and MFN (Fathony et al., 2021) are two INR methods which we extend fit our setting. We consider an agnostic setting, i.c. without the knowledge of the differential equation, and perform sequence conditioning generalize more than a trajectory. This is achieved by learning latent vector with auto-decoding; it is then concatenated to the spatial coordinates.
the resulting interpolation. MNO (Li et al., 2021a) is autoregressive version
Tasks. We evaluate models on various forecasting tasks which combine the evaluation scenarios of Section 2. Performance is measured by the prediction Mean Squared Error (MSE) given only an initial condition. Space and time generalization. We consider uniform grid X" for inference. Training is performed on different observations grids X₂ subsampled from X' with different ratios, s E (5%, 25%, 50%, 100%} } where s = 100% corresponds to the full inference grid, i.e. X₁ = X'. In this setting, consider that all trajectories (train and test) share the same observation grid X, = Xrs. We evaluate MSE error on X' over the train time interval (/)-)) and beyond (Out-f) at each subsampling ratio. Flexibility w.r.t. input grid. We vary the test observation grid, i.c. Xe # X₂ and perform inference on X" = Xy. i.c. on the test observation grid (In-s) under two settings: >> Generalizing across grids: Xr, Xr are subsampled differently from the same uniform grid; 8nr (resp. s.) is the train
7

---

Published as a conference paper at ICLR 2023
Table 3: Flexibility w.r.. input grid. Observed test train grid differ (Nss + XT).. We report test MSE (L) for Navier-Stokes on X' = Xy (In-s). Green Yellow Red mean excellent, good, poor MSE.
(a) Generalization across grids: Nr, Ai are subsampled with different ratios Air * B/s among {5, 50, 1003% % from the same uniform x4 grid.
Subsampling
Test
5%
%
50%
So
100%
Train
Out-t
Io-1
In-1
Out-t
5%
MP-PDE
1,30E-1
3.8526-1
1.899-1
6.680E-1
2,105E-1
7.120=-1
DINo
1,494-33
2.291
1.257=33
18838-3
1,2878-3
1.947-33
MP. MP-PDE
4.494E-2
9,403E-2
4.7936-3
1.997-22
6.330E-3
3.712F-2
50%
DINo
2,470g-4
4.697E-4
2,073k-4
4.284E-4
2,0588-4
4.361E 4
MP-PDE
1
3,3550-1
1.1826-2
2.6648-2
1.785g-4
5 5,8561-4
Av = 100%
DINo
2.495g-4
4.805E-4
2.109E-4
4.325g-4
2.092E
4.311E-4
(b) Generalization across resolutions: Nr (resp. Xe) subsampled at the same ratio {5, 100 from different uniform grids with resolution no E {32, 64, 256} (resp. f'r 64).
Test resolution-
To 32
X # N
fu = 04 - Ni Ni
256 - X # N/2
Subsampling L.
In-t
Out-1
In-1
Out-t
In.1
Out-t
MP-PDE
3
6.4726-1
2.4658-4
1.105--3
2
8.2536-1
5%
DIND
5,308E-3
9,544E-3
2,5336-4
8.8326-4 4
1.991E-3
2.942c-3
MNO
4.5476-3
9.281e-3
1,2776-4 4
8.5256-4
2.1746-3
4.9756-3
100%
MP-PDE
4.194g-2
9.1090-2
1,597E-4
6.483n-4
4.6480-2
1.381811
DINo
2.321E
6,386g-4
2,320E
6.385g-4
2.3238-4
6.385k-4
(resp. test) subsampling ratio. & Generalizing across resolutions: Xr, Ass are subsampled with the same ratio from two uniform grids with different resolutions; the train resolution is fixed to fir 64 while we vary the test resolution Tis E (32, 64, 256]. Data on manifold. We consider a PDE on a sphere and combine several evaluation scenarios, as described later. Finer time resolution. We consider an inference time grid T'' with a finer resolution than the train one T.
5.2 RESULTS
Space and time generalization. We report prediction MSE in Table 2 for varying subsampling ratios E (5%, 25%, 100%) on Navier-Stokes and Wave. Appendix A provides fine-grained evaluation inside the train observation grid (In-s) or outside (Out-s) and additionally reports the results for s 50%. We visualize some predictions in Appendix B. DINO is compared to all baselines when s = 100%, i.e. X" = X₂ = Xr. and otherwise it is compared only to models which handle irregular grids and prediction arbitrary spatial locations (DeepONet, SIREN, MFN, I-MP-PDE). General analysis. We observe that all models degrade when the subsampling ratio decreases. DINO performs competitively overall: it achieves the best Out-I performance on all subsampling settings, it outperforms all the baselines on low subsampling ratios and performs comparably to the competitive discretized (MP-PDE, CNODE) and operator (MNO) alternatives when s = 100%, i.c. when observation and inference grids are equal. Note that this fully observed setting is favorable for CNODE, MP-PDE and MNO, designed to perform inference on the observation grid. This can be seen in Table 2, where DINO is slightly outperformed only for few settings. MP-PDE is significantly better only on Wave for In-t. Overall, CNNs and GNNs exhibit good performance for spatially local dynamics like Wave, while INRs (like DINO) and MNO are more adapted to global dynamics like Navier-Stokes. Analysis per model. MP-PDE is the most competitive baseline across datasets as combines strong and flexible encoder (GNNs) to good dynamics model; however, it cannot predict outside the observation grid (Out-s). To keep strong competitor, we extend this baseline into its interpolative version J-MP-PDE on subsampled settings. I-MP-PDE is competitive for high subsampling ratios, e.g. E (50%, 100%} but underperforms w.r.t. DINO at lower subsampling ratios due to the accumulated interpolation error. MNO is a competitive baseline on Navier-Stoker performing on par with MP-PDE and DINO inside the training horizon (In-f); its performance on Out-t degrades more significantly compared to other models, especially DINO. DeepONet is more flexible than MP-PDE as it can predict at arbitrary locations. As no interpolation error is introduced, it outperforms I-MP-PDEE for s = 5% on train data. Yet, we observe that it underperforms especially on Out-I w.r.t. its alternatives. Finally, observe that SIREN and MFN fit correctly the train
8

---

Published as a conference paper at ICLR 2023
horizon In-t train. yet generalize poorly outside this horizon Out-r new initial conditions (test). This is in accordance with our analysis of Section 3; we highlight that this is not the case for DINO which extrapolates temporally and generalizes to new initial conditions thanks to its sequential modeling of the flow. Thus, DINO is currently the state-of-the-at INR model for spatiotemporal data. Modulation. We observe that modulating both amplitudes and frequencies (row "DINO (no sep.)" in Table 2) degrades performance w.r.t. DINO (row "DINO" in Table 2) that only modulates amplitudes. Amplitude modulation enables long temporal extrapolation and reduces the number of parameters. Hence, as opposed to DINo (no sep.) which is outperformed by some baselines, time-space variable separation in DINO is an essential ingredient of the model to reach state-of-the of-teoof-hhe-rt levels.
Flexibility w.r.t input grid. We consider in Table 3 Navier-Stokes and compare DINO to the most competitive baselines, MP-PDE and MNO (with 8 100% subsam- pling ratio). Generalizing across grids. In Table 3a, we consider that the test observation grid X₂ is different from the train one x, This occurs when sensors differ between two observed trajectories. We vary the subsampling ratio for the train observation grid Sy and the test one Sis- We report test MSE on new grids X" Xn. We observe that DINO is very robust to changing grids between train and test, while MP-PDE's performance degrades, especially for low subsampling ratios, c.g. 5%. For reference, we report in Table 6 Appendix A (result col. 1) the performance when X" = Xr. where MP-PDE is substantially better. Gen- eralizing across spatial resolutions. In Table 3b we vary the test resolution Tis- We train at a resolution Tir = 64 and perform inference at resolutions Vis E {32, 64, 256). For that, we build high-fidelity 256x256 simulation dataset and downscale it to obtain the other resolutions. We observe that DINO's performance is the stablest across resolutions in the uniform or irregular setting. MNO is also relatively stable but is only applicable uniform grids while MP-PDE is particularly brittle, especially for a 5% ratio.
r-0
T
DINO
Reference
Model
In-t
Out-t
I-MP-PDE DINO
1.908F-3
7.240F-3
1063E-4
6.466E-4
Figure 5: Data on manifold. DINO's Shallow- Whallow-Water superresolution test prediction (top) against the reference (middle), with test MSE (L) (bottom).
Table 4: Finer time resolution. Test MSE (L) under T' for Navier-Stokes.
Model
In-t
Out-t
Data on manifold. We consider in Figure Shallow-Wate in a super-resolution setting: test resolution is twice the train one, close to weather prediction applications. We observe
I-DINo (lincar)
3.4596-4
5.598
I-DINo (qundratic)
2.1651-4
4.4730-4
DINo (DDE solve)
2.151
4.388
an irregular 3D Euclidean coordinate grid X₂ = Xg c IR3 shared for train and test. It uniformly samples Euclidean positions on the sphere, via the quasi-uniform skipped longiude grid (Weller et al., 2012). We predict the PDE on test trajectories with conventional latitude-longitude inference grid X", At Earth scale, Xn corresponds to resolution of about 300 km, and X' to 150 km. DINO significantly outperforms 1-MP-PDE. making it a viable candidate for this complex setting.
Finer time resolution. We consider in Table 4 a longer and ten times finer test temporal grid T than the train grid T on Navier-Siokes. We observe the same spatial uniform grid across train and test and perform inference on this grid. We compare DINO that performs prediction with an ODE solver, to interpolating coarser predictions obtained at the train resolution (I-DINo). We report the corresponding test MSE We observe that the ODE solver accurately extrapolates outside the train temporal grid, outperforming interpolation. This confirms that DINO benefits from its continuous- time modeling of the flow, providing consistency and stability across temporal resolutions.
CONCLUSION
We propose DINO, a novel space- and time-continuous data-driven PDE forecaster. DINO handles free-form spatiotemporal conditions encountered in many applications, where existing methods fail. DINO outperforms recent PDE forecasters on a variety of PDEs and spatiotemporal generalization settings, including evaluation unseen sparse irregular meshes and resolutions. There many promising future work such as scaling DINo to real-world problems, e.g. weather forecasting, or incorporating recent strategies adapt changing dynamics (Kirchmeyer et al., 2022).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Emmanuel de Bézenac and Jérémie Donà for helpful insights and discussions on this project. We also acknowledge financial support from DL4CLIM LANR-21-CE23-00177 ANR projects. This study has been conducted using E.U. Copernicus Marine Service Information.
and DEEPNUM
REPRODUCIBILITY STATEMENT
We present in Section 5.1 our experimental setting with datasets, baselines and forecasting tasks. The train and test settings are detailed in Appendix including more information on the chosen physical PDE systems. We describe DINO's pseudo-code in Algorithm and provide implementation and hyperparameters details in Appendix D. We provide our source code at https: github....... com/ mk mkirchneyez/DIN..
REFERENCES
Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Learning the spatio- temporal dynamics of physical processes from partial observations. In IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. pp. and 7)
May 2020. (cit. on
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, pp. 1171-111 1179. Curran Associates, Inc., 2015. (cit. on 21)
Marsha J. Berger and Joseph Oliger. Adaptive mesh refinement for hyperbolic partial differential equations. Journal of Computational Physics, 53(3):484-512, March 1984. (cit. p. 2)
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations, 2022. (cit. on pp. 1, 2, 3, 7, and 21)
Steven Brunton and J. Nathan Kutz. Data-driven science and engineering: Machine learning. dynamical systems, and control. Cambridge University Press, 2022. (cit. on p. 1)
Keaton J. Burns, Geoffrey M. Vasil, Jeffrey S. Oishi, Daniel Lecoanet, and Benjamin P. Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2(2), April 2020. (cit. on p. 19)
Sangwon Chae, Joonhyeok Shin, Sungjun Kwon, Sangmok Lee, Sungwon Kang, and Donghyun Lee. PM10 and PM2.5 real-time prediction models using interpolated convolutional neural network. Scientific Reports, 11, 2021. (cit. on p. 3)
Hao Chen, Bo He, Hanyu Wang. Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. NeRV: Neural representations for videos. In Mare' Aurelio Ranzato, Alina Beygelzimer. Yann Dauphin, Percy Liang, and Jenn Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 21557-21568. Curran Associates, Inc., 2021. (cit. p. 4)
Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, Yue Chang, G A Pershing, Henrique Teles Maia, Maurizio M. Chiaramonte, Kevin Thomas Carlberg, and Eitan Grinspun. CROM: Continuous reduced-order modeling of PDEs using implicit neural representations. In International Conference on Learning Representations, 2023. (cit. on p. 4)
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bisnch,, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 6572-6583. Curran Associates, Inc., 2018. (cit. on pp. 2 and 21)
10

---

Published as a conference paper at ICLR 2023
Filipe De Avila Selbute-Peress Thomas Economon, and Zico Kolter. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. Hal Daumé, III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 2402-2411. PMLR, July 2020. (cit. on 1)
Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018. (cit. on pp. and 23)
Jérémie Donå, Yves Franceschi, Sylvain Lamprier, and Patrick Gallinari. PDE-driven spatiotem- poral disentanglement. In International Conference on Learning Representations, 2021. (cit. on p. 23)
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data functa: Your data point is a function and you can treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5694-5725. PMLR, July 2022. (cit. on p. 6)
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2021. (cit. on pp. 4, 6, 7, and 21)
Marco Fraccaro. Deep Latent Variable Models for Sequential Data. PhD thesis, Danmarks Tekniske Universitet, 2018. (cit. on p. 5)
Stefania Fresca, Andrea Manzoni, Luca Dedè, and Alfio Quarteroni. Deep learning- based reduced order models cardiac clectrophysiology. PLoS ONE, 15(10), October 2020. (cit. on p. 4)
Joseph Galewsky, Richard K. Scott, and Lorenzo M. Polvani. An initial-value problem for testing numerical models of the global shallow-water equations. Tellus A: Dynamic Meteorology and Oceanography, 56(5):429 440, 2004. (cit. on pp. 7, 18, and 19)
David Ha, Andrew M. Dai, and Quoe V. Le. HyperNetworks. In International Conference Learning Representations, 2017. (cit. on p. 5)
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint 100iv.1606.08415. 2016. (cit. on p. 20)
Valerii lakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-timg PDEs from sparse data with graph neural networks. In International Conference on Learning Representations, 2021. (cit. on p. 3)
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019. (cit. on p.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. (cit. on p. 21)
Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In Pro- ceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1283-11301. PMLR, July 2022. (cit. on p. 9)
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang. Michael Brenner, and Stephan Hoyer. Machine accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 11(21), 2021. (cit. on p.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv: 108.00 08881, 2021. (cit, on p. 3)
11

---

Published as a conference paper at ICLR 2023
Hervé Le Dret and Brigitte Luequin. Partial Differential Equations: Modeling, Analysis and Numerical Approximation, chapter The Heat Equation, pp. 219-251. International Series of Numerical Mathematics. Springer International Publishing. Cham, Switzerland, 2016. (cit. p. 6)
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tices Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6755- 6766. Curran Associates, Inc., 2020. (cit. on pp. and 3)
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Markov neural operators for learning chaotic systems. arXiv preprint arXiv:2 100/20000000 06898, 2021a. (cit. on pp. 2, 3, and 7)
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021b. (cit. on
pp. 3, 7, 18, and 21)
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for PDEs on general geometries. arXiv preprint arXiv:2207.05209, 2022. (cit. on p.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-Net: Learning PDEs from data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1208-3216, Stockholmsmässan, Stockholm, Sweden, July 2018. PMLR. (cit. on p. 3)
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 5:218-229, 2021. (cit. on pp. 1, 3, and 7)
Gurvan Madec and NEMO System Team. NEMO ocean engine. Technical Report 27, Scientific Notes of Climate Modelling Center, Institut Pierre-Simon Laplace (IPSL). Zenodo. (cit. on p. 23)
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). The European Conference on Computer Vision (ECCV), pp. 405-421, Cham, Switzerland, 2020. Springer International Publishing. (cit. on 1)
Peter J. Olver. Introduction to partial differential equations. Undergraduate Texts in Mathematics. Springer Cham, 2014. (cit. on p. 1)
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 165,174, June 2019. (cit. on onp.p. 5)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, perforerffe deep learning library. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 8026 8037. Curran Associates, Inc., 2019. (cit. on p. 20)
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, (2(1):3942-33) 3951,, April 2018. (cit. on p. 6)
Tobias Pfaff, Meire Fortunato, Alvaro Lunchez-Gonzale and Peter W. Battaglia. Learning mesh- based simulation with graph networks. In International Conference on Learning Representations, 2021. (cit. pp. and 3)
12

---

Published as a conference paper at ICLR 2023
Michael Prasthofer, Tim De Ryck, and Siddhartha Mishra. Variable-inuut deep operator networks. arXiv preprint arXiv:2205.114000 2022. (cit. on p. 1)
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informedt neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. (cit. on pp. 1, 2, and
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, arXiv preprint arXiv:1710.05941, 2017. (cit. p. 20)
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375: 1364, 2018. (cit. on p. 1)
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1462-7473. Curran Associates, Inc., 2020. (cit. on pp. 1, 4, 7, and 21)
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3626-3636 June 2022. (cit. p.
George Gabriel Stokes. On the effect of the internal friction of fluids on the motion of pendulums. Transactions of the Cambridge Philosophical Society, 9(2):8-106, 1851. (cit. on pp. 7 and 18)
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin Advances in Neural Information Processing Systems, volume 33, pp. 7537-7547. Curran Associates, Inc., 2020. (cit. pp. I and 4)
Advances
Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-informed DeepONets. arXiv preprint 1rXiv:2106.05384 2021. (cit. on p. 7)
Hilary Weller, John Thuburn, and Colin J. Cotter. Computational modes and grid imprinting on five quasi-uniform spherical C grids. Monthly Weather Review, (408))2733427555 2012. (cit. on p. 9)
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Computing Surveys, Jamuary 2022. (cit. on p.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.1011 10157, 2021. (cit. on p. 5)
Yuan Yin, Vincent Le Guen, Jérémie Donå, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, December 2021. (cit. on pp. 1 and 3)
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Woo Ha, and Jinwoo Shin. Generating videos with awarm implicit generative adversarial networks. In International Conference Learning Representations, 2022. (cit. on p. 4)
Çağatay Yildız, Markus Heinonen, and Harri Lähdesmäki. ODE2VAE Deep generative second order ODES with Bayesian neural networks. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 13434-1344, Curran Associates, Inc., 2019. (cit. on p. 23)
Laure Zanna and Thomas Bolton. Deep Learning of Unresolved Turbulent Ocean Processes in Climate Models, chapter 20, pp. 298-306. John Wiley Sons, Ltd, 2021. (cit. on p. 2)
13

---

Published as a conference paper at ICLR 2023
A FULL RESULTS
We provide in Table 5 a more detailed version of Table for the space-time extrapolation problem where we report the performance In-s (on the observation grid) and Out-s (outside). We add 50%.
Then, we report in Table 6 a more detailed version of Table 3a, which includes the results of Ars X This corresponds our generalization across grids problem.
Table 5: Space and time generalization. The train and test observation grids are equal: they are subsampled with ratio from uniform 64 x64 grid fixed here be the inference grid X". We report MSE (L) on X' (on the observation grid In-s, outside Our-s or on both Full) and the inference time interval T', divided within training horizon (In-1, 7) and beyond (Out-I, outside 7) across Best in bold and second best underlined.
subsampling ratios & E (5%, 25%, 50%, 100%}.
Navier-Stokes
Model
Train
Train
Tost
In-t
Out-4
In-t
Out-t
In-4
Out-t
In-t
Out-t
5%
subsampling
A.P--DD
3.525k-5 5
1,295k-3
4.554F-4
1.414F-3
1884F-6
8.672F-5
1.1138-5
1.98F-4
DeepONet
4,7780-4
4.5176-3
1.060e-2
1.059c-2
2,5466-4
8.8316-3
1.501b-2
3.196--2
SIREN
5,966E-3
1.769p-1
4.082E-2
2.150E-1
L690E-3
1.707--2
2.951B-2
6.955E-2
DINo
1.0166-4
6.945
3.6238-4
8.3068-4
2.2506-6
5,2838-6
7.530E-6
2.1468-5
I.MP PDE
8.5506-3
8.515E-3
8.306E-3
8.57
7.412F-4
7.414F-4
1.195-33
L163E-3
Out-
DeepONet
3.475e-3
7.515k-3
1.3618-2
1.426--2
8.624
9.3186-3
1,7026-2
3.259R-2
SIREN
8.8820-3
1.767p-1
4.3140-2
2.1240-1
2.791
1.823-2
3,3590-2
6.9651-2
DINo
1.076-33
1.704-33
1.3758-3
1.863-33
4.285
4.304
6.703F-5
7.659E-5
1. PDE
8.1546-3
8.166E-3
7.926E-3
8,2256-3
7,055E-4
7.0976-4
1.1388-3
1111E-3
Ealt
DeepONet
3.330R-3
7.370k-3
1.346F-2
1,408F-2
8.331
9.295R-3
1.6926-2
3.256R-2
SIREN
8.7410-3
1,7676-1
4.3030-2
2.1260-1
2.738t-3
1.8180-2
3.3396-2
6.9648-2
DINo
1.029E-3
L655r-3
1.3268-3
1.8138-3
4.088F-5
4.121E-5
6.415F-5
7.392x-5
25% subsampling
L.MP-PDE
1.4476-4
5.6776-4
1.7638-4
6.147F-4
8.251
9,253F-7
1.2276-4
DeepONet
7,5006
5,779h-3
9.2276-3
1,3006-2
5,1966-4
1.0586-2
1,7436-2
3.2468-2
SIREN
4.766-33
2.1788-1
2.461E-1
3,884c-1
8.478-44
1.2826-2
1.7336-2
5.1048-2
DINo
8.2958-5
4.273k
2.448-44
5.735k-4
3,194E-6
3,747E-6
8.907E-6
1.029E-5
I.MP PDE
3.678E-4
7.748E-4
4.026-44
8.1436-4
4.330E-5
1.206-4
6.7646-5
1648F-4
Out-
DeepONet
9,5036
5.9876-3
9.4238-3
1.3376-2
5.8916-4
T.0626-2
1,7626-2
3.218-2
SIREN
5,3050-3
2.1736-1
2.4280-1
3.8530-1
9.159g-4
1.2950-2
1,7980-2
5.1560-2
DINO
1.081
4,578E
2.7118-4
6,0218-4
4.192E-6
4.657E-6
1.153E-5
1,220E-5
1.MP PDE
3.135E-4
3.476E-4
7,658E-4
3.2938-5
L108E-4
5.1426-5
1.545-4
DeepONet
9.0166-4
5.936R-3
9.376F-3
1.3286-2
5.722F
L.061F-2
1.7576-2
12218-2
SIREN
5.1800-3
2,1758-1
2.4360-1
3.8610-1
8.995E-4
1,2920-2
1.7836-2
5.1430-2
DINo
1.0208-4
4,504:
2.646g
5.9518-4
3,949E-6
4.436E-6
1.089--5
1.174--5
50%
subsampling
I-MP-PDE
L1536-4
5.0168-4
1.59gE-4
6.043c-4
2.200E--
3.1796-5
8.846-7
5.854g-5
DeepONet
6.2166-4
4.2776-3
5.699e-3
1.082e-2
7.581
1,1876-2
1.6496-2
3.3788-2
SIREN
49110-3
6.8158-1
16676-1
6.889-1
5.1348-4
1.481022
3.0860-2
1966
DINO
8.151E-5
2.920=44
2000E-4
4.283
3.2778-6
3.659E-6
8,978866
9.572x-6
I-MP-PDE
1.160-4
5.010
1.6268-4
6.132c-4
9.638E-7
3.1530-5
2.3676-6
5.574p-5
Out-
DeepONet
6.851R-4
3.3438-3
5.7408-3
1,099F-2
7.842F-4
1
1.679k-2
3.391H-2
SIREN
5.0670-3
6.8676-1
1
6.845c-1
5.3548-4
1.4920-2
3.1136-2
8.3330-2
DINO
9.175E-5
3.41E-
2.116E-4
4.409
3.2778-6
3.659E-6
8,9788-6
9.572E-6
1-MP-PDE
1.1700-4
5.013p-4
1.611E-4
6.088c-4
6.021E-7
3.1660-5
16646-6
5.7101-5
DeepONet
6.541R-4
4.3118-3
5.720R-3
1.091R-2
7.715F-4
1,166R-2
1.6658-2
3.385E-2
SIREN
4.950-33
6.8418-
1.603-1
6.8670-1
5.2466-4
1.4868-2
3. 3.1006-2
8.265
DINO
8.6776-5
2.982
2.0626-4
4.348
3.3801-6
3.751E-6
9.251B-6
9.710E-6
100%
subsampling
CNODE
2,3190-2
9.6528-2
2.305g-2
1.143c-1
2.3378-5
5.280E-4
3.0576-5
7.2888-4
MP-PDE
5.500E-4
1.785g-4
5,866
19936-5
9.2561-7
MNO
3.1908-5
8.678
2.7638-4
8.946E
9.3818-6
4.890E
1.9938-4
6.128
Full
DeepONet
1.375E-3
6.5738-3
9.704E-3
1.244E-2
6.4318-4
1.293E-2
18847--2
3.317E-2
SIREN
1.066E-3
4.336E-1
3.874-11
1.03760
3.674E-4
9.956E-3
3.0138-2
7.842F-2
MPN
1.651R-3
1.03760
10166-1
1.05900
1,08F-4
1.763--1
4.735k-3
2.274R-1
DINo sep.)
3.2350-4
1.593p-3
7.8500-4
1,8890-3
2.6418-6
4.0810-5
5.9776-5
2.979t-4
DINO
8.339E-5
3.115E-4
2.092E-4
4.311E-4
3.309E-6
3,506E-6
9.495B-6
9,9462-6
14

---

Published as a conference paper at ICLR 2023
Table 6: Generalization across grids. Ay. du are subsampled with different ratios Str * Sis {5, 50, 100)% % from the same uniform 64 64 grid. We report test MSE within Nrs (In-s). Best in bold.
Xr
Subsampling
Test +
8/.
Bi
5%
50%
8,
100%
Train 4
In-t
Out-t
Io-t
Out-t
In-t
Out-t
In-t
Our-t
MP-PDE
1.97E-
6.631E-4
1.3300-1
3.8526-1
18896-1
6.680u-1
2.105k-1
7.120C-1
Bry
DINo
3.623g-1
8.306E-4
1494g-3
2.2918-33
1,257--3
1883:-3
1.2878-3
1947E-3
F 50%
MP PDE
1.346 4
5.110E-4
4.494F-2
9.403E-2
4.793E-3
1,997E-2
6.330e-3
3.7126-2
DINg
4.2838-4
100%
MP PDE
1.785E
5.856E-4
1,358E-1
3.355--1
1.1826-2
2.664E-2
1.783e 4
5.856E-4
DINo
4.311E-4
4,805E-4
4.311E-4
B PREDICTION
We display the test prediction of DINo (Figure 6) and I-MP-PDE (Figure 7) for various subsampling levels when X Xr = Xr., Predictions are performed on a x4 uniform grid which defines the observation grid X' via different subsampling rates. Yellow points correspond to the observation grid X (In-s) while purple points indicate off-grid points (Our-s). The prediction for I-MP-PDE t = the interpolated initial condition.
Subsampling rate grid
Observation
Predicted trajectory
T
5%
25%
100%
Ground Truth
Figure 6: Prediction MSE per frame for DINO on Navier-Stokes with its corresponding observed grid X. For each model, the first row contains the predicted trajectory from 0 to T", the second row the corresponding error maps w.r.t. the reference data (the darker the pixel, the lower the error).
15

---

Published as a conference paper at ICLR 2023
Subsampling Ohservation grid
Predicted trajectory
T
T=T
5%
25%
100%
Ground Truth
Prediction MSE per frame for I-MP-PDE on Navier-Stoker with its corresponding observed grid X. For each model, the first row contains the predicted trajectory from 0 T', the second row is the corresponding error maps w.r.t. the reference data (the darker the pixel, the lower the error).
Figure 7:
16

---

Published as a conference paper at ICLR 2023
Subsampling Ohservation grid X
Predicted trajectory
T
T=T'
5%
25%
100%
Ground Truth
Prediction MSE per frame for DINO on Wave with its corresponding observed grid X' For each model, the first row contains the predicted trajectory from T'. the second row the corresponding error maps w.r.t. the reference data (the darker the pixel, the lower the error)
Figure 8:
17

---

Published as a conference paper at ICLR 2023
C DETAILED DESCRIPTION OF DATASETS
We choose T (resp. T') on regular grid in [0, T] (resp. [0, T') with a given temporal resolution and fix T = 27' + öt, where öt is the step size of the temporal grid. Hence, we always consider 10 consecutive frames for In-I and more for Out-1 The range of depends on the nature of the dataset. We provide below further details on the choice of these parameters and other experimental parameters, such as the number of observed trajectories.
2D Wave equation (Wave). It is second-order PDE:
jau
Au,
(10)
O12
where u is function of the displacement at each point in space w.r.t the rest position, c R4 is the speed of wave traveling. We transform the equation to first-order form, considering the input
2t
Uty
du. at
so that the dimension of vy(x) at each point x E Ω is n = 2.
We generate our dataset for speed 2 with periodic boundary condition. The domain is Ω
1]2. For initial conditions = 240
Du. 0t
the initial displacement is Gaussian
1==0
function:
(x - 6)²
uo a, b, = a exp
(11)
2r
where the height of the peak displacement is a ~ (42,4) 4). the location of the peak displacement is (b,, ~ 11,1,11 1), and the standard deviation is (0.225, 0.3). The initial time derivative is
Ous
Each snapshot generated
uniform grid of x4. Each sequence is generated
f==0 with fixed interval öt = 0.25. We set the train horizon T = 2.25 and the inference horizon T 4.75.
at
We generated 512 train trajectories and 32 test trajectories.
2D Navier Stokes (Navier-Stokee Stokes, 1851). This dataset corresponds to incompressible fluid dynamics described by:
dw - vAW f. w V u.
Vu Vu=0, 0,
(12)
where u the velocity field and w the vorticity. u, w on spatial domain with periodic boundary
conditions, is the viscosity and is constant forcing term. The input e is wt (n 1). is the viscosity and f is the constant forcing term in the domain Q.
The spatial domain is Ω 1-1, 1J2, the viscosity is v 1 x 10-3, the forcing term is set as:
Yo E Q, fx1 .22) 0.1 sin (2 (a) + x2)) + cos (2 + +aa)))
(13)
The full spatial grid is of dimension 64 256x256 according experiments in Section 5. We sample initial conditions as in Li et al. (2021b) to create different trajectories. The first 20 steps of the trajectories are cut off as they are too noisy and not informative in terms of dynamics. Trajectories are collected with öt = 1. We set the training horizon I' = 19 and the inference horizon T" = 39. We generated 512 train trajectories and 32 test trajectories.
3D spherical shallow water (Shallow- Water, Galewsky et al., 2004). The following problem was originally presented for numerical model testing of global shallow- water equations. They can be written as:
du = fk X gV vAu,
dt
(14)
dh -VV u 44
di
where d is the material derivative, the unit vector orthogonal to the spherical surface, u is dt the velocity field tangent the surface of the sphere which can be transformed into the vorticity w = V x u, and is the thickness of the sphere. Note that the data we observe at each time is et (we, hy). J. v, Ω are parameters of the Earth; ef. Galewsky et al. (2004) for details.
18

---

Published as a conference paper at ICLR 2023
The initial conditions slightly modified from Galewsky et al. (2004), detailed below, create symmetric phenomena on the northern and southern hemisphere. The initial zonal velocity uo contains two non-null symmetric bands in the both hemispheres, which are parallel to the circles of latitude. At each latitude and longitude Φ 0 E [-*/2, %/2) x n.
Ummx
1
)0
exp
(0 - 20) +1)
o E (0o, Φ1),
uo(6,
(15)
Umax
exp
if o E (-01,-00)
en
+ (o) + ②1))
(0,0)
otherwise.
where Wmax is the maximum velocity, do -/7, x/2 on, and en 10/1/1/--000 oo)3). The water height ho is initialized by solving a boundary value condition problem as in Galewsky et al. (2004). It then perturbed by adding the following hig to ho:
16 0) = (o) exp (- (-12))
exp (-(12)
(
0/2 ② 2)
)
(-(()) + 2
+ exp +exp
(16)
B
8
where de 4/4, h = 120 m, a = 1/3 and B = 1/15 are constants defined in Galewsky er al. (2004).
We simulate this phenomenon with Dedalus (Burns et al., 2020) latitude-loniituee (lat-lon)
grid. The size of the grid is 128 (lat) x 256 (lon). We take different initial conditions by sampling Umax 7(60, 80) generate long trajectories. These long trajectories are then sliced into shorter ones. For simulation, we take one snapshot per hour (of internal simulation time), i.e. ôt = 1 h. We stop the simulation at the t2 hour. To construct dataset rich of dynamical phenomena, we take the snapshots within the last 160 h in long trajectory and slice them into 8 shorter trajectoires, Also note that data scaled into reasonable range: the height is scaled by factor of x 10², and the vorticity w by factor 2. In each short trajectory. T = 9 and T" = h9 In total, we generated 16 long trajectories (i.e. 128 short trajectories) for train, for test (i.e. 16 short trajectories).
D IMPLEMENTATION
We provide our code at ht httpps://ggttaab..ccm/////c/ce//
D.1 ALGORITHM
We detail the algorithm of DINO for training and test via pseudo-code Algorithm Training consists in solving Eq. (6) w.r.t 1h, ar,, Φ. Inference involves optimization only to find og.
Algorithm 1: DINO pseudo-code
Training: Input: D = {UT}. {az + OpvED. @ t do, 20 + v/o
while not converged do for v E D do at t at aa Vazlac(d,a7))
/+ Modulation +/
- 0 n1 V
EvED (2,
/+ Hypernetwork update */
+ の - V EvEDDam
V.
lym (1i a4)
/* Dynamics update x/
Test: Input: = (un). faj + OlvED' Φ², ".
while converged do
for E D' do aj t añ 1/ (ot :a);;
/* Modulation */
for u E D',,,EE do
of - a. + So for (T;
/* Unroll dynamics */
ह, + (o(a))
/* Predict x/
19

---

Published as a conference paper at ICLR 2023
10
Edve
Edgn
10-1
10:22
10-8
10-5
20000
20000
60000
80000
100000
120000
Epoch
Figure 9: Learning curves on Navier-Stokes for faee and layn throughout training (pale lines) and corresponding exponential moving averages from epoch 500 with half-life 1000 (opaque lines).
D.2 CONVERGENCE ANALYSIS
In practice, we observe no training instability induced by the two-stage learning process of Eq. (6) and Algorithm I: the objectives non-confficting. To assess this, track the evolution of the auto-decoding loss fdec and the dynamics loss fayn throughout training on Navier-Stokes (s 100%) in Figure 9. We observe that both losses smoothly converge until the end of training.
D.3 TIME EFFICIENCY
Our auto-decoding strategy coupled with a latent neural ODE makes DINO computationally efficient compared to our best competitor MP-PDE.
Inferring via auto-decoding. Given decoder and an observation frame up. finding corres- ponds to solving an inverse problem, cf. Eq. (3). inference, we use 300 steps infer og; using less steps is possible but results in slight underfitting. This represents 2.76 s for 64 trajectories on single Tesla V100 Nvidia GPU. Note that, as we unroll dynamics in the latent space, there no need to relearn o,, when / > O. Moreover, this differs from training, where Or, is continuously optimized for all t e [0, T| within the train horizon, in parallel with our INR decoder. Overall, we trained MP-PDE and DINO for approximately 7 days such that there is major additional temporal training cost for DINo.
Latent Neural ODE. Unrolling the dynamics with neural ODE is efficient (0.35 s for 19 time single Tesla V100 Nvidia GPU). Indeed, the latent space is small (at most 100 dimension) and the dynamics model uses a simple four-layer MLP for fa- With the same latent dynamics model, using RK4 numerical scheme only incurs four additional function evaluations over a discretized alternative like a standard ResNet. This incurs a minor computational cost but enables DINo to operate different temporal resolutions, unlike e.g. MP-PDE.
predictions for 64 trajectories
In comparison, the official code of MP-PDE takes 312 for inference on the same hardware for the same number of trajectories (vs 3 for DINO). MP-PDE requires building adjacency matrix and incurs for this reason a high memory cost, especially as the number of nodes increases. Interpolation also significantly increases inference time. This not the case for DINo, which faster.
D.4 ADDITIONAL IMPLEMENTATION DETAILS
We use PyTorch (Paszke et al., 2019) to implement DINO and our baselines. Hyperparameters are further defined in Appendix D.5. The dynamics model fo is a multilayer perceptron with swish activation function (Hendrycks & Gimpel, 2016; Ramachandran al., 2017). Its input and output sizes are the same as the size latent space d,,- All hidden layers share the same size. DINO's parameters are initialized with the default initialization in PyTorc,, defining do, to, in Algorithm 1.
20

---

Published as a conference paper at ICLR 2023
Table 7: DINo's hyperparameters.
Hyperparameter
Navier-Stokes
Wave
Shallow-
Decoder Do
Iho
Number of layers Number of hidden channels Frequency scale factor w; Size of latent space da
3
3
64
64
256
64
64
64
100
50
300
Dynamics model fø
Number of layers Hidden layer size Activation function
4
4
512
512
800
Swish
Swish
Swish
Optimization
Learning rate Ve Learning rate 7/a Learning rate The
10-2
10-2
10
10-3
10-3
10-3
10
10-3
10-3
Number of epochs
12 000
12000
120000
Batch size i.e. sequences per batch
64
64
16
We recall that w is fixed throughout training to reduce the number of optimized parameters without loss of performance. As related work (Sitzmann al., 2020; Fathony et al., 2021), the frequency parameters are scaled by factor, 4i, considered as hyperparameter. For dynamics learning, we use an RK4 integrator via TorchDiffEq (Chen al., 2018) and apply exponential Scheduled Sampling (Bengio al., 2015) to stabilize training. In practice, modulations ar are learned channel-wise such that lg: + Rdd has separate parameters per output dimension to make predictions less correlated across channels. We optimize all parameters o, a, v using Adam (Kingma & Ba, 2015) with decay parameters (31, 32) = (0.9, 0.999).
D.5 HYPERPARAMETERS
We list the hyperparameters of DINO for each dataset Table In practice, we observe it beneficial to decay the learning rates no. na when the loss reaches plateau.
D.6 BASELINES IMPLEMENTATION
We detail in the following the hyperparameters and architectures used in our experiments for the considered baselines, which we reimplemented for our paper.
CNODE is implemented with four two-dimensional convolutional layers with 64 hidden features, ReL.U activations, 3 x 3 kernel and zero padding. Learning rate is fixed to 10-3, We use adjoint method for integration like Chen al. (2018).
MNO. We use the FNO architecture of al. (2021b) with three FNO blocks, GeLU activations, 12 modes and width of 32. Learning rate is fixed to 10-3.
DeepONet. We consider an autoregressive formulation of DeepONet. We choose width of 1000 for hidden features with depth of for both trunk and branch nets with ReLU activations. Learning rate is fixed to 10-5
MP-PDE. We adapt the implementation in Brandstetter et al. (2022) to handle two- and three-dimensional PDEs. We use a time window of 1 with pushforward trick. Batch size and number of neighbors are fixed to 8. Learning rate is fixed to 10-3 We use ReLU activations.
SIREN. To represent data in space and time, SIREN takes space and time coordinates (x, t) as input. To handle multiple trajectories, we concatenate an optimizable per-trajectory context code
the coordinates like in DINo. We fix the hidden layer size of SIREN to 256. We initialize the parameters and use the default input scale as in Sitzmann et al. (2020). The size of the context code is d, 800. The learning rate is 10-3
21

---

Published as a conference paper at ICLR 2023
Table Long-term extrapolation performance of DINO and (1-)MP-PD in the space and time gen- eralization experiment for test trajectories on Out-t (T, T = T + AT); cf. Table 2 and Section 5.1.
Subsampling ratio
Model
AT = T
AT = 5T
AT = 10T
AT = 50T
DINO
2.017E-3
3.895E-3
1.209E-2
1.440--1
5 = 5%
I-MP-PDE
8.387
3,580E-2
3.356E-1
4.031E1
DINo
4.617E-4
2.082E-3
6,901E-3
1.215E-1
8 100%
MP-PDE
5.251E-4
3.524E-2
3.339E-1
9.755E1
Table 9: MSE reconstruction error (In-s and Out-s) of train sequences within the train horizon (In-t) for three different methods: interpolation of observed points in Nu. FourierNet learned over individual frames in Xt, and DINo (FourierNet with dynamics model).
MSE train In-t
Interpolation
FourierNet
DINo
Navier-Stokee 5% Wave, 5%
8.277E-3 7.075E-4
9.673E-4
1.029E-3
4.085E-5
4.088E-5
MFN. Similarly to the previous SIREN baseline, we concatenate the per-trajectory context code to space and time coordinates at the first layer. The hidden layer size is fixed to 256 and we use the default parameter initialization with a frequency scale ws 64 higher than DINo. The size of the context code is da 800. The learning rate is 10
The ablation "DINO (no sep.)" modulates frequencies through latent shift modulation from oy similarly to the bias terms bs in Section 4.4.
E COMPLEMENTARY ANALYSES
We detail in this section additional experiments, allowing us to further analyze and assess the performance of DINO.
E. LONG-TERM TEMPORAL EXTRAPOLATION
We provide in Table 8 an analysis of eтror accumulation over time for long-term extrapolation. More precisely, we generate Navier-Stokes dataset with longer trajectories and report MSE for T" = + AT where AT E [T, 5T, 10T, 507). Note that AT T is the setting of our main experiments (T" 2T).
We observe that DINo's MSE in long-term ter- forecasting is more than an order of magnitude smaller than for (I-)MP-PDE. This demonstrates the extrapolation abilities of our model.
E.2 INRs' ADVANTAGE OVER INTERPOLATION
We report in Table 9 the MSE of bicubic interpolation, our FourierNet's MSE Gautoddcoodigg with amplitude modulation but without dynamics model) and DINO's MSE (with dynamics model) on train In-1 for both Navier-Stoken and Wave. This corresponds to MSE averaged over all training frames within the train horizon and not only the initial condition vp-
We observe that FourierNet is better than interpolation. Indeed, interpolation is poorly adapted to sparse observation grids: the interpolation errors are clearly visible Figure 7, first row (5% setting). Interestingly, DINO's MSE is only slightly worse than the FourierNet's MSE, showing that we correctly learned the dynamics of latent modulations aę I-MP-PDE, which combines bicubic interpolation with MP-PDE, is then expectedly outperformed by DINo on this challenging 5% setting. This shows the advantage of using INRs instead of standard bicubic interpolation to interpolate between observed spatial locations.
22

---

Published as a conference paper at ICLR 2023
Table 10: SST test prediction performance for DINo and VarSep (Donà et al., 2021).
Method
MSE
VarSep
1.43
DINo
1.27
E.3 MODELING REAL- WORLD DATA
SST. We evaluate DINO on real-world data further assess its applicability. Following de Bézenac et al. (2018) and Donà et al. (2021), we model the Sea Surface Temperature (SST) of the Atlantic ocean, derived from the data-assimilation engine NEMO (Nucleus for European Modeling of the Ocean, Madec & NEMO System Team) using E.U. Copernicus Marine Service Information. Accu- rately modeling SST dynamics is critical in weather forecasting or planning of coastal activities. This problem is particularly challenging as SST dynamics are only partially observed: several unobserved variables affecting the dynamics (e.g. the sea water flow) need to be estimated from data.
For this experiment, we consider trajectories collected from three geographical zones (17 to 20) following the initial train / test split of de Bézenac et al. (2018). Notably, = 9 which includes d of conditioning frames, i.c. models are tested to predict [+e[+r;]] from Vte[0,7-1]-
Incorporating consecutive time steps. To model SST which includes non-Markovian data and thus does not correspond to Initial Value Problem as in Section 2, we modify our dynamics model in similar fashion to Yıldız et al. (2019) to integrate a history of several consecutive observations 0te]o,--1] instead of only the initial observation Vo. In more details, augmented state [av, of] where a is auto-decoded state and af is an encoding of r = 4 past auto- decoded observations via neural network cę. We adjust our inference and training settings as follows:
define neural ODE over an
inference: we compute a'-1 = ce (cto..... ... a1-1) 1) and then unroll our neural ODE from the initial condition (0,-1, a,_1 to obtain (ar, a/) for all t > -- 11
d d o'] = fe a());
Vt [0, - 1]. cp(ot), a ce(av...... ................................ a,-1),
dt
training: for all f, we infer o/ Cc (o 0147-1) and fit the above neural ODE on the [a,, a²] obtained for all E [0, T + 1].
This experiment confirms that our space- and time-continuous framework can easily be extended to incorporate refined temporal models.
Results. We report in Table 10 test MSE for DINO and VarSep (Donà et al., 2021), the current state-of-the-art on SST, retrained on the same training data. DINO notably outperforms VarSep in prediction performance. This demonstrates DINO's potential to handle complex real-world spatiotemporal dynamics. We also provide some visualizations of DINo's train and test predictions Figure 10. We make two observations. First, DINo fits very accurately the train data. Second, on the test data, we observe that the dynamics on low frequencies seem to be correctly modeled while the prediction of high frequencies dynamics are less accurate. Larger scale experiments would be required to effectively evaluate the model performance on this challenging dataset. Given the complexity of the data, this out of the scope of the paper. Yet, these experiments already demonstrate that DINo behaves competitively w.r.t. the previous state-of-the-art.
Implementation choices. We choose a similar INR and dynamics architecture as for our Shallow- water experiment. We use for cę, which takes input four consecutive a,s, individual encoding of the a&s through a four-layer fully connected network which are then fed to a single linear layer.
'https://data. www.maaa....eenneee cuspernicuu..o o ANAALLLLIS SANALLYSSSPPRRCASTT PHY_001 024 www_001_024/deccrigt.
23

---

Published as a conference paper at ICLR 2023
t=
T
t= T
Ground Truth
DINO
3
Ground Truth
DINO
(a) Train predictions.
T=T
Ground Truth
DINO
Ground Truth
DINO
(b) Test predictions.
Figure 10: DINo's 's prediction examples on SST.
24


Reviews:
[{"review_id": "WU5Ovy3D96I", "reviewer": "Reviewer_eLsK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Continuous PDE Dynamics Forecasting with Implicit Neural Representations\" presents a space and time continuous model for forecasting. This is achieved by coupling an autodecoder to estimate a latent state, a hypernetwork-based emmission model that parameterizes an implicit neural network, and by modeling the devleopment of the latent via a learned flow with an ODE. The method, termed DINo, is compared to baselines on three datasets (2D wave equation, 2D Navier Stokes, and 3D spherical shallow water), and shown to outperform other methods in the flexible setting of changing sampling grids.\n", "summary_of_the_review": "The paper is well written, and proposes a novel approach in an interesting domain. The biggest weakness is the lack of comparison on a real-world task.\n", "strength_and_weaknesses": "trengths.\n- the paper is very well written, and easy to understand.\n- the topic of continuous-time, space-continous modeling is theoretically interesting, and offers more generality than standard machine learning approaches on a grid - hence, is very interesting to the community in my opinion.\n- a novel method is worked out, which makes intutive sense. In particalar the proposed separation of time & space, termed amplitude modulation, seems to improve performance.\n\nWeakenesses.\n- all datasets are somewhat synthetic. This offers a lot of possiblity to investigate the method (which the authors do), but showing performance on a real-world dataset would significantly strengthen the paper (e.g. on a weather foracasting task, or on a multi-variate time-series task, e.g. from e-Commerce).\n- limitations of the method should be worked out clearer: e.g. if I understand correctly, the method cannot easily integrate information from observing several time-points? Another one: the MSE loss on the latents implies uni-modality (a single Gaussian), and determinism - can the model handle multi-modalities and stochasticities in the forecasts?\n\nSmall things:\n- in several places you write 'extrapolation' (e.g. abstract, p.2. ...). However, while the method can perform extrapolation well in time, due to the ODE, it probably performs not well on extrapolation in space. The case of changing the grid to points within the convex hull of seen ones (e.g. superresolution etc.), I'd consider to be an interpolation task. Hence, please use 'interpolation' for the spatial tasks in the paper.\n- Figure 4: from eq. 9 it seems b and \\mu are similar - there should be also a plut between W and b then. Also, the z_ts point to b, while in the formula only W is modulated by z.\n\nUPDATE: \nThanks for the changes made to the paper, which nicely removes some of the limitations, and adds a real-world dataset (although only in the appendix). I upgraded the score accordingly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PSnzfgwwaSu", "reviewer": "Reviewer_P8h2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DINO, a model that generalizes to arbitrary spatial and temporal resolutions, beyond the spatial and temporal samples in training. This is achieved via a combination of autodecoding, dynamics model, and hypernetwork with amplitude modulation. It shows in experiment that DINO achieves in general superior performance in generalizing to novel time interval and spatial resolutions, compared to strong baselines.", "summary_of_the_review": "In summary, the paper is well-motivated, well-written, novel, and solid (through thorough experiments). The paper would be strengthened if the weaknesses is addressed.", "strength_and_weaknesses": "Strength:\nThe paper is well motivated and clearly written. The method is novel, and the experiment evaluation is thorough. The quality of the paper is very high.\n\nWeaknesses:\n1. It is not clear how the method's error accumulate in the long-term evolution for out-t scenario. The long-term evolution is a key challenge in neural-based solvers. A model with excellent short-term error does not necessarily mean a good long-term error (since the error can accumulate and the model can overfit to short-term behavior.\n\nTherefore, it would be nice and strengthen the paper, if the authors show how different models behave where the out-t is range is significantly greater than the in-t (e.g. 5 times, 10 times, 50 times the length). A figure showing the error accumulation is also nice.\n\n2. It is not clear how time consuming is the method. The auto-decoding may need multiple steps to infer the correct alpha. In addition, the latent evolution may be more time consuming that regular time intervals. It would be great if the authors can report the runtime (in seconds) of each method.\n2. The author mentioned FNO in the introduction. it would be nice to compare with the proposed method with the strong baseline of FNO.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G_siCsCF_a", "reviewer": "Reviewer_swAv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a continuous space-time data-driven model for predicting the spatiotemporal evolution of physical phenomena driven by PDE. The method embeds spatial observations independently of their discretization via Implicit Neural Representations and then models continuous-time evolution by a learned latent ODE. The paper claims it can learn from sparse irregular grids or manifolds and extrapolate at arbitrary spatial and temporal locations. ", "summary_of_the_review": "The paper proposes an encoder-decoder framework to address the continuous space-time data-driven model for predicting spatiotemporal evolution, which is important and challenging. The extensive experiments on the simulated dataset validate the model’s performance and the capability to extrapolate at arbitrary spatial and temporal locations. However, there are several concerns to be addressed:\n\n1. The ignorance of spatial autocorrelation in the framework.\n2. There is no analysis of the training convergence. How to ensure the encoded feature reflect the spatial \nReal-world spatiotemporal forecasting evaluation would be more interesting. \n", "strength_and_weaknesses": "Strength:\nThe problem is interesting and challenging.  The forecasting model for continuous space-time is important for many real-world tasks.\nThe model does not rely on fixed discretization on training and test data and can extrapolate at arbitrary spatial and temporal locations.\nThe extensive experiments on the simulated dataset validate the model’s performance in modeling the evolution of the dynamic system.  \n\nWeakness:\nThe model decoder only uses the current latent dynamic temporal feature (alpha_t) and ignores previous spatial information. On different spatial locations, the decoder prediction is also independent, which ignores the spatial autocorrelation among samples. \nSome baseline for spatiotemporal interpolation and extrapolation with common machine learning models is lacking, such as the Gaussian process. \nThe bi-level optimization training for the dynamic model and auto-decoder seems hard to converge. Some analysis on the training and convergence is needed. \nMore evaluations on some real-world spatiotemporal datasets like weather forecasting, air quality forecasting would be interesting to show the generalization capability on real dynamic problems. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4SAVa9gbtQN", "reviewer": "Reviewer_4pqH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces DINo, a continuous time and space solver for partial differential equations. The authors proposed to model the solution of the PDE as an implicit neural representation conditioned on the initial condition through an hyper-network. The dynamics is modeled as a learned latent ordinary differential equation.", "summary_of_the_review": "I really enjoyed the paper and find the overall model architecture quite clever. However, I still have some doubts about the performances of the model compared to more trivial interpolation techniques combined with existing methods. The paper provides a quite extensive comparison with SOTA, yet I am not sure that the chosen task are actually relevant to correctly assessed the model. If the authors provides satisfying answers to my questions, I will be very pleased to increase my grade.\n\n## EDIT\nThe authors provided very detailed and convincing answers, which remove my fears on this paper. I now think that this work deserves to be presented at ICLR. I still have some concern about the practical performance of the model. Although the datasets are perfectly within the standards of the community, these tasks are already relatively well mastered. I would have given the maximum score if the technical contribution (definitely respectable) had been completed with new results on more difficult datasets than those conventionally used in the state of the art.\n\nIt is nevertheless a work of excellent quality, which opens the door to new approaches for the resolution of PDE, and which, I believe, deserves the attention of the community. I increased my rating from 6 to 8.", "strength_and_weaknesses": "I really enjoyed this paper. Forecasting solutions of PDEs in a continuous manner is a very interesting topic yet very challenging. Such techniques may alleviate the need of heaving meshing procedure in standard physics solver. The proposed approach seems reasonable: while using INR for modeling continuous solutions seems the natural thing to do, conditioning its parameters to the initial condition is not straightforward. Amplitude modulation seems like a very clever and interesting method to tackle the problem. However, I may need additional precisions to fully understand the contribution. \n\n- I do understand the advantages of using FourierNet with the chosen architecture for the hyper-network in theory. However, I wonder if such gain is verified in practice. For example, could it be possible to replace FourierNet by SIREN or MFN conditioned with an hyper-network (and not directly through the input) ? Could the authors briefly discuss about the other structure they tried ? I have the same kind of question for the auto-decoder: is it actually beneficial over an explicit learned encoder model ?\n\n- I am not absolutely convinced by the \"separation of variable\" argument in section 4.4. It is not clear to me how separation of variable is relevant in this situation. This technique allows to disentangle a PDE in a set of ODEs, which are easier to solve. Yet, in this framework, the PDE seems to be directly translated into a ODE in the latent space $\\alpha_t$, where the dynamics solely depends on the time derivatives. I would appreciate if the authors could clarify this statement.\n\n- Related to my previous remark, I am not sure how the space derivatives (which are needed to solve the PDE) can be embedded in the latent vector $\\alpha_t$ such that the entire dynamics can be resumed as a temporal ODE. Does the authors have any insights about what is happening here ? May the auto-decoder somehow pre-computes spatial derivatives (even if the observation is partial) and embed them in a convenient form in $\\alpha_t$ ?\n\nI also acknowledge the release of the code, which actually helps me to answer to more technical question about the model (especially, how the partial observation is inputted to the model). However, I think that appendix section D could be more precise.\n\nMy most important questions focuses on the experiments. I do appreciate the comprehensive study conducted by the authors, with relevant and numerous baselines, and extensive analysis on the generalization capacity of DINo. Yet, I am still a bit skeptical about the chosen datasets, for three reasons, ordered below in ascending order of importance:\n\n- The main results (table 2 and 3) are computed on uniform grids. I do not see any aspect of DINo that are not compatible with uniform meshes, which are much more powerful and widely used to solve PDEs, especially Navier-Stokes equations. However, uniform grid contains a lot of regularities that both DINo and the baselines may leveraged. I wonder how the proposed method would behave on irregular triangular mesh with complex physics such as the ones introduced in [1].\n\n- Both qualitatives and quantitatives results may indicates that the task on which DINo is evaluated are actually saturated. My doubts are based on the very close MSE obtained in the standard setup $s=100$% and the merely indistinguishable difference between figure 6 and 7. This hypothesis could quickly be rejected by showing additional prediction example from both DINo and I-MP-PDE, maybe with some failure cases. This seems important: if the tasks are saturated, it is unclear how such tasks are relevant for assessing DINo generalization capacities.\n\n- I was quite amazed by the performances of DINo in the 5% subsampling scenario. According to me, these results, and actually many others, may be explained by one the three following hypothesis:\n     1. DINo can leverage knowledge for the PDE dynamics to extrapolate un-observed information. \n     2. DINo exploits biaises in the data to infer the state in the entire space.\n     3. Datasets are not sufficiently complex and basic interpolation is enough.\nIn my opinion, (3) is highly possible, as I-MP-PDE seems to perform competitively to DINo in the 5% setup. (3) may be easily rejected by comparing the reconstruction error of DINo on $v_0$ with direct bi-cubic interpolation from the partial observations. If the error on the latter is bigger than the error on the former, then DINo does leverages knowledge either from physics or from regularities in the dataset. Checking the validity of (2) is much more challenging. Maybe the easiest way could be to look at a wider set of trajectories from the dataset and make sure to observe sensible diversity among them. I would really appreciate if the authors could provide an in-depth discussion concerning these points.\n\nI also wonder why I-MP-PDE is absent from table 3, it seems to me that it could be compared with DINo in this setup as well. \n\n\n[1] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning mesh-\nbased simulation with graph networks. In International Conference on Learning Representations\n(ICLR), 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%