Title: Learning Space-Time Continuous Latent Neural PDEs from Partially Observed States
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural

PDEs

neural PDEs

partial observations

space time continuous
Abstract: We introduce a novel grid-independent model for learning partial differential equations (PDEs) from noisy and partial observations on irregular spatiotemporal grids. We propose a space-time continuous latent neural PDE model with an efficient probabilistic framework and a novel encoder design for improved data efficiency and grid independence. The latent state dynamics are governed by a PDE model that combines the collocation method and the method of lines. We employ amortized variational inference for approximate posterior estimation and utilize a multiple shooting technique for enhanced training speed and stability. Our model demonstrates state-of-the-art performance on complex synthetic and real-world datasets, overcoming limitations of previous approaches and effectively handling partially-observed data. The proposed model outperforms recent methods, showing its potential to advance data-driven PDE modeling and enabling robust, grid-independent modeling of complex partially-observed dynamic processes across various domains.

Learning Space-Time Continuous Neural PDEs from Partially Observed States

Abstract

We introduce a novel grid-independent model for learning partial differential equations (PDEs) from noisy and partial observations on irregular spatiotemporal grids. We propose a space-time continuous latent neural PDE model with an efficient probabilistic framework and a novel encoder design for improved data efficiency and grid independence. The latent state dynamics are governed by a PDE model that combines the collocation method and the method of lines. We employ amortized variational inference for approximate posterior estimation and utilize a multiple shooting technique for enhanced training speed and stability. Our model demonstrates state-of-the-art performance on complex synthetic and real-world datasets, overcoming limitations of previous approaches and effectively handling partially-observed data. The proposed model outperforms recent methods, showing its potential to advance data-driven PDE modeling and enabling robust, grid-independent modeling of complex partially-observed dynamic processes.

1 Introduction

Modeling spatiotemporal processes allows to understand and predict the behavior of complex systems that evolve over time and space (Cressie and Wikle, 2011). Partial differential equations (PDEs) are a popular tool for this task as they have a solid mathematical foundation (Evans, 2010) and can describe the dynamics of a wide range of physical, biological, and social phenomena (Murray, 2002; Hirsch, 2007). However, deriving PDEs can be challenging, especially when the system's underlying mechanisms are complex and not well understood. Data-driven methods can bypass these challenges (Brunton and Kutz, 2019). By learning the underlying system dynamics directly from data, we can develop accurate PDE models that capture the essential features of the system. This approach has changed our ability to model complex systems and make predictions about their behavior in a data-driven manner.

While current data-driven PDE models have been successful at modeling complex spatiotemporal phenomena, they often operate under various simplifying assumptions such as regularity of the spatial or temporal grids (Long et al., 2018; Kochkov et al., 2021; Pfaff et al., 2021; Li et al., 2021; Han et al., 2022; Poli et al., 2022), discreteness in space or time (Seo et al., 2020; Pfaff et al., 2021; Lienen and Günnemann, 2022; Brandstetter et al., 2022), and availability of complete and noiseless observations (Long et al., 2018; Pfaff et al., 2021; Wu et al., 2022). Such assumptions become increasingly limiting in more realistic scenarios with scarce data and irregularly spaced, noisy and partial observations.

We address the limitations of existing methods and propose a space-time continuous and gridindependent model that can learn PDE dynamics from noisy and partial observations made on irregular spatiotemporal grids. Our main contributions include:

harri.lahdesmaki}@aalto.fi

Development of an efficient generative modeling framework for learning latent neural PDE models from noisy and partially-observed data;

Novel PDE model that merges two PDE solution techniques - the collocation method and the method of lines - to achieve space-time continuity, grid-independence, and data efficiency;

Novel encoder design that operates on local spatiotemporal neighborhoods for improved data-efficiency and grid-independence.

Our model demonstrates state-of-the-art performance on complex synthetic and real-world datasets, opening up the possibility for accurate and efficient modeling of complex dynamic processes and promoting further advancements in data-driven PDE modeling.

2 Problem Setup

In this work we are concerned with modeling of spatiotemporal processes. For brevity, we present our method for a single observed trajectory, but extension to multiple trajectories is straightforward. We observe a spatiotemporal dynamical system evolving over time on a spatial domain Ω . The observations are made at M arbitrary consecutive time points t 1: M := ( t 1 , . . . , t M ) and N arbitrary observation locations x 1: N := ( x 1 , . . . , x N ) , where x i ∈ Ω . This generates a sequence of observations u 1: M := ( u 1 , . . . , u M ) , where u i ∈ R N × D contains D -dimensional observations at the N observation locations. We define u j i as the observation at time t i and location x j . The number of time points and observation locations may vary between different observed trajectories.

We assume the data is generated by a dynamical system with a latent state z ( t, x ) ∈ R d , where t is time and x ∈ Ω is spatial location. The latent state is governed by an unknown PDE and is mapped to the observed state u ( t, x ) ∈ R D by an unknown observation function g and likelihood model p :

where ∂ · x z ( t, x ) denotes partial derivatives wrt x .

In this work we make two assumptions that are highly relevant in real-world scenarios. First, we assume partial observations, that is, the observed state u ( t, x ) does not contain all information about the latent state z ( t, x ) (e.g., z ( t, x ) contains pressure and velocity, but u ( t, x ) contains information only about the pressure). Second, we assume out-of-distribution time points and observation locations, that is, their number, positions, and density can change arbitrarily at test time.

3 Model

Here we describe the model components (Sec. 3.1) which are then used to construct the generative model (Sec. 3.2).

3.1 Model components

Our model consists of four parts: space-time continuous latent state z ( t, x ) and observed state u ( t, x ) , a dynamics function F θ dyn governing the temporal evolution of the latent state, and an observation function g θ dec mapping the latent state to the observed state (see Figure 1). Next, we describe these components in detail.

Latent state. To define a space-time continuous latent state z ( t, x ) ∈ R d , we introduce z ( t ) := ( z 1 ( t ) , . . . , z N ( t )) ∈ R N × d , where each z i ( t ) ∈ R d corresponds to the observation location x i . Then, we define the latent state z ( t, x ) as a spatial interpolant of z ( t ) :

where Interpolate( · ) maps z ( t ) to an interpolant which can be evaluated at any spatial location x ∈ Ω (see Figure 2). We do not rely on a particular interpolation method, but in this work we use linear interpolation as it shows good performance and facilitates efficient implementation.

Latent state dynamics. Given a space-time continuous latent state, one can naturally define its dynamics in terms of a PDE:

where F θ dyn is a dynamics function with parameters θ dyn. This is a viable approach known as the collocation method (Kansa, 1990; Cheng, 2009), but it has several limitations. It requires us to decide which partial derivatives to include in the dynamics function, and also requires an interpolant which has all the selected partial derivatives (e.g., linear interpolant has only first order derivatives). To avoid these limitations, we combine the collocation method with

23(t)

z²(t)

21(t)

C4

3C2

c1

z(t,∞)

another PDE solution technique known as the method of lines (Schiesser, 1991; Hamdi et al., 2007), which approximates spatial derivatives ∂ · x z ( t, x ) using only evaluations of z ( t, x ) , and then let the dynamics function approximate all required derivatives in a data-driven manner. To do that, we define the spatial neighborhood of x as N S ( x ) , which is a set containing x and its spatial neighbors, and also define z ( t, N S ( x )) , which is a set of evaluations of the interpolant z ( t, x ) at points in N S ( x ) :

and assume that this information is sufficient to approximate all required spatial derivatives at x . This is a reasonable assumption since, e.g., finite differences can approximate derivatives using only function values and locations of the evaluation points. Hence, we define the dynamics of z ( t, x ) as

which is defined only in terms of the values of the latent state, but not its spatial derivatives.

One way to define the spatial neighbors for x is in terms of the observation locations x 1: N (e.g., use the nearest ones) as was done, for example, in (Long et al., 2018; Pfaff et al., 2021; Lienen and Günnemann, 2022). Instead, we utilize continuity of the latent state z ( t, x ) , and define the spatial neighbors in a grid-independent manner as a fixed number of points arranged in a predefined patter around x (see Figure 3). This allows to fix the shape and size of the spatial neighborhoods in advance, making them independent of the observation locations. In this work we use the spatial neighborhood consisting of two concentric circles of radius r and r/ 2 , each circle contains 8 evaluation points as in Figure 3. In Appendix D we compare neighborhoods of various shapes and sizes.

Equation 7 allows to simulate the temporal evolution of z ( t, x ) at any spatial location. However, since z ( t, x ) is defined only in terms of a spatial interpolant of z ( t ) (see Eq. 3), with z i ( t ) = z ( t, x i ) , it is sufficient to simulate the latent state dynamics only at the observation locations x 1: N . Hence, we can completely characterize the latent state dynamics in terms of a system of N ODEs:

2

×

×

×

For convenience, we define z ( t ; t 1 , z 1 , θ dyn ) := ODESolve( t ; t 1 , z 1 , θ dyn ) as the solution of the ODE system in Equation 8 at time t with initial state z ( t 1 ) = z 1 and parameters θ dyn. We also define z ( t, x ; t 1 , z 1 , θ dyn ) as the spatial interpolant of z ( t ; t 1 , z 1 , θ dyn ) as in Equation 3. We solve the ODEs using off the shelf differentiable ODE solvers from torchdiffeq package (Chen, 2018). Note that we solve for the state z ( t ) only at the observation locations x 1: N , so to get the neighborhood values z ( t, N S ( x i )) we perform interpolation at every step of the ODE solver.

Observation function. We define the mapping from the latent space to the observation space as a parametric function g θ dec with parameters θ dec :

where N is the Gaussian distribution, σ 2 u is noise variance, and I D is D -byD identity matrix.

3.2 Generative model

Training models of dynamic systems is often challenging due to long training times and training instabilities (Ribeiro et al., 2020; Metz et al., 2021). To alleviate these problems, various heuristics have been proposed, such as progressive lengthening and splitting of the training trajectories (Yildiz et al., 2019; Um et al., 2020). We use multiple shooting (Bock and Plitt, 1984; Voss et al., 2004), a simple and efficient technique which has demonstrated its effectiveness in ODE learning applications (Jordana et al., 2021; Hegde et al., 2022). We extent the multiple shooting framework for latent ODE models presented in (Iakovlev et al., 2023) to our PDE modeling setup by introducing spatial dimensions in the latent state and designing an encoder adapted specifically to the PDE setting (Section 4.2).

Multiple shooting splits a single trajectory { z ( t i ) } i =1 ,...,M with one initial state z 1 into B consecutive non-overlapping subtrajectories { z ( t i ) } i ∈I b , b = 1 , . . . , B with B initial states s 1: B := ( s 1 , . . . , s B ) while imposing a continuity penalty between the subtrajectories (see Figure 4). The index set I b contains time point indices for the b 'th sub-trajectory. We also denote the temporal posi-

tion of s b as t [ b ] and place s b at the first time point preceding the b 'th sub-trajectory (except s 1 which is placed at t 1 ). Note that the shooting states s b have the same dimension as the original latent state z ( t ) i.e., s b ∈ R N × d . Multiple shooting allows to parallelize the simulation over the sub-trajectories and shortens the simulation intervals thus improving the training speed and stability. In Appendix D we demonstrate the effect of multiple shooting on the model training and prediction accuracy.

We begin by defining the prior over the unknown model parameters and initial states:

where p ( θ dyn ) and p ( θ dec ) are zero-mean diagonal Gaussians, and the continuity inducing prior p ( s 1: B | θ dyn ) is defined as in (Iakovlev et al., 2023)

Intuitively, the continuity prior p ( s b | s b -1 , θ dyn ) takes the initial latent state s b -1 , simulates it forward from time t [ b -1] to t [ b ] to get µ [ b ] = ODESolve( t [ b ] ; t [ b -1] , s b -1 , θ dyn ) , and then forces µ [ b ] to approximately match the initial state s b of the next sub-trajectory, thus promoting continuity of the full trajectory. We assume the continuity inducing prior factorizes across the grid points, i.e.,

where p ( s j 1 ) is a diagonal Gaussian, and parameter σ 2 c controls the strength of the prior. Note that the term z ( t [ b ] , x j ; t [ b -1] , s b -1 , θ dyn ) in Equation 13 equals the ODE forward solution ODESolve( t [ b ] ; t [ b -1] , s b -1 , θ dyn ) at grid location x j .

Finally, we define our generative in terms of the following sampling procedure:

with the following joint distribution (see Appendix A for details about the model specification.):

4 Parameter Inference, Encoder, and Forecasting

4.1 Amortized variational inference

Weapproximate the true posterior over the model parameters and initial states p ( s 1: B , θ dyn , θ dec | u 1: M ) using variational inference (Blei et al., 2017) with the following approximate posterior:

where q ψ dyn , q ψ dec and q ψ j b are diagonal Gaussians, and ψ dyn, ψ dec and ψ j b are variational parameters. To avoid direct optimization over the local variational parameters ψ j b , we use amortized variational inference (Kingma and Welling, 2013) and train an encoder h θ enc with parameters θ enc which maps observations u 1: M to ψ j b (see Section 4.2). For brevity, we sometimes omit the dependence of approximate posteriors on variational parameters and simply write e.g., q ( s j b ) .

In variational inference the best approximation of the posterior is obtained by minimizing the Kullback-Leibler divergence: KL [ q ( θ dyn , θ dec , s 1: B ) ∥ p ( θ dyn , θ dec , s 1: B | u 1: N ) ] , which is equivalent to maximizing the evidence lower bound (ELBO), defined for our model as:

The terms ( ii ) , ( iv ) , and ( v ) are computed analytically, while terms ( i ) and ( iii ) are approximated using Monte Carlo integration for expectations, and numerical ODE solvers for initial value problems. See Appendix A and B approximate posterior details and derivation and computation of the ELBO.

4.2 Encoder

Here we describe our encoder which maps observations u 1: M to local variational parameters ψ j b required to sample the initial latent state of the sub-trajectory b at time point t [ b ] and observation location x j . Similarly to our model, the encoder should be data-efficient and grid-independent.

Similarly to our model (Section 3.1), we enable grid-independence by making the encoder operate on spatial interpolants of the observations u 1: M (even if they are noisy):

where spatial interpolation is done separately for each time point i . We then use the interpolants u i ( x ) to define the spatial neighborhoods N S ( x ) in a grid-independent manner.

ti-1

tb=ti

ti+1

ti+2

{ui-1(m)JENs(m;) {u;(α)}ENs(x)） {ui+1(α)}aeNs(ms)

hspatial

hspatial

hspatial

α²+1]

S

htemporal

hread

To improve data-efficiency, we assume ψ j b does not depend on the whole observed sequence u 1: M , but only on some local information in a spatiotemporal neighborhood of t [ b ] and x j . We define the temporal neighborhood of t [ b ] as

where δ T is a hyperparameter controlling the neighborhood size, and then define the spatiotemporal neighborhood of t [ b ] and x j as

Our encoder operates on such spatiotemporal neighborhoods u [ t [ b ] , x j ] and works in three steps (see Figure 5). First, for each time index k ∈ N T ( t [ b ] ) it aggregates the spatial information { u k ( x ) } x ∈N ( x j ) into a vector α S k . Then, it aggregates the spatial representations α S k across time into another vector α T [ b ] which is finally mapped to the variational parameters ψ j b as follows:

Spatial aggregation. Since the spatial neighborhoods are fixed and remain identical for all spatial locations (see Figure 5), we implement the spatial aggregation function h spatial as an MLP which takes elements of the set { u k ( x ) } x ∈N S ( x j ) stacked in a fixed order as the input.

Temporal aggregation. We implement h temporal as a stack of transformer layers (Vaswani et al., 2017) which allows it to operate on input sets of arbitrary size. We use time-aware attention and continuous relative positional encodings (Iakovlev et al., 2023) which were shown to be effective on data from dynamical systems observed at irregular time intervals. Each transformer layer takes a layer-specific input set { ξ in k } k ∈N T ( t [ b ] ) , where ξ in k is located at t k , and maps it to an output set { ξ out k } k ∈N T ( t [ b ] ) , where each ξ out k is computed using only the input elements within distance δ T from t k , thus promoting temporal locality. Furthermore, instead of using absolute positional encodings the model assumes the behavior of the system does not depend on time and uses relative temporal distances to inject positional information. The first layer takes { α S k } k ∈N T ( t [ b ] ) as the input, while the last layer returns a single element at time point t [ b ] , which represents the temporal aggregation α T [ b ] .

Variational parameter readout. Since α T i is a fixed-length vector, we implement h read as an MLP.

4.3 Forecasting

Given initial observations ˜ u 1: m at time points t 1: m , we predict the future observation ˜ u n at a time point t n > t m as the expected value of the approximate posterior predictive distribution:

The expected value is estimated via Monte Carlo integration (see Appendix C.4 for details).

5 Experiments

We use three challenging datasets: SHALLOW WATER, NAVIER-STOKES, and SCALAR FLOW which contain observations of spatiotemporal system at N ≈ 1100 grid points evolving over time (see Figure 6). The first two datasets are synthetic and generated using numeric PDE solvers (we use scikitfdiff (Cellier, 2019) for SHALLOW WATER, and PhiFlow (Holl et al., 2020) for NAVIER-STOKES), while the third dataset contains real-world observations (camera images) of smoke plumes raising in warm air (Eckert et al., 2019). In all cases the observations are made at irregular spatiotemporal grids and contain only partial information about the true system state. In particular, for SHALLOW WATER we observe only the wave height, for NAVIER-STOKES we observe only the concentration of the species, and for SCALAR FLOW only pixel densities are known. All datasets contain 60 / 20 / 20 training/validation/testing trajectories. See Appendix C for details.

We train our model for 20k iterations with constant learning rate of 3 e-4 and linear warmup. The latent spatiotemporal dynamics are simulated using differentiable ODE solvers from the torchdiffeq package (Chen, 2018) (we use dopri5 with rtol= 1 e-3, atol= 1 e-4, no adjoint). Training is done on a single NVIDIA Tesla V100 GPU, with a single run taking 3-4 hours. We use the mean absolute error (MAE) on the test set as the performance measure. Error bars are standard errors over 4 random seeds. For forecasting we use the expected value of the posterior predictive distribution. See Appendix C for all details about the training, validation, and testing setup.

Latent state dimension. Here we show the advantage of using latent-space models on partially observed data. We change the latent state dimension d from 1 to 5 and measure the test MAE. Note that for d = 1 we effectively have a data-space model which models the observations without trying to reconstruct the missing states. Figure 7 shows that in all cases there is improvement in performance as the latent dimension grows. For SHALLOW WATER and NAVIER-STOKES the true latent dimension is 3. Since SCALAR FLOW is a real-world process, there is no true latent dimension. As a benchmark, we provide the performance of our model trained on fully-observed versions of the synthetic datasets (we use the same architecture and hyperparameters, but fix d to 3 ). Figure 7 also shows examples of model predictions (at the final time point) for different values of d . We see a huge difference between d = 1 and d = 3 , 5 . Note how apparently small difference in MAE at d = 1 and d = 5 for SCALAR FLOW corresponds to a dramatic improvement in the prediction quality.

ShallowWaterNavier-Stokes

ScalarFlow

0.080

Data

0.060

ShallowWater

0.040

0.020

O----o----O----O

0.000

MAE

0.120

d=5

0.090

Navier-Stokes

0.060

0.030

0.000

d=3

0.070

0.060

ScalarFlow

0.050

0.040

0.030

T

T

d =1

1

2

3

4

5

Latent State Dim.

Grid independence. In this experiment we demonstrate the grid-independence property of our model by training it on grids with ≈ 1100 observation locations, and then testing on a coarser, original, and finer grids. We evaluate the effect of using different interpolation methods by repeating the experiment with linear, k-nearest neighbors, and inverse distance weighting (IDW) interpolants. For SHALLOW WATER and NAVIER-STOKES the coarser/finer grids contain 290 / 4200 nodes, while for SCALAR FLOW we have 560 / 6420 nodes, respectively. Table 1 shows the model's performance for different spatial grids and interpolation methods. We see that all interpolation methods perform rather similarly on the original grid, but linear interpolation and IDW tend to perform better on finer/coarser grids than k-NN. Performance drop on coarse grids is expected since we get less accurate information about the system's initial state and simulate the dynamics on coarse grids. Figure 8 also shows

ShallowWaterNavier-Stokes

ScalarFlow

Data

Finer

Original

Coarser

examples of model predictions (at the final time point) for different grid sizes and linear interpolant.

Comparison to other models. We test our model against recent spatiotemporal models from the literature: Finite Element Networks (FEN) (Lienen and Günnemann, 2022), Neural Stochastic PDEs (NSPDE) (Salvi et al., 2021), MAgNet (Boussif et al., 2022), and DINo (Yin et al., 2023). We also use Neural ODEs (NODE) (Chen et al., 2018) as the baseline. We use the official implementation for all models and tune their hyperparameters for the best performance (see App.

C for details). For SHALLOW WATER and NAVIER-STOKES we use the first 5 time points to infer the latent state and then predict the next 20 time points, while for SCALAR FLOW we use the first 10 points for inference and predict the next 10 points. For synthetic data, we consider two settings: one where the data is fully observed (i.e., the complete state is recorded) - a setting for which most models are designed - and one where the data is partially observed (i.e., only part of the full state

is given, as discussed at the beginning of this section). The results are shown in Table 2. We see that some of the baseline models achieve reasonably good results on the fully-observed datasets, but they fail on partially-observed data, while our model maintains strong performance in all cases. Apart from the fully observed SHALLOW WATER dataset where FEN performs slightly better, our method outperforms other methods on all other datasets by a clear margin. See Appendix C for hyperparameter details. In Appendix E we demonstrate our model's capability to learn dynamics from noisy data. In Appendix F we show model predictions on different datasets.

6 Related Work

Closest to our work is Ayed et al. (2022), where they considered the problem of learning PDEs from partial observations and proposed a discrete and grid-dependent model that is restricted to regular spatiotemporal grids. Another related work is that of Nguyen et al. (2020), where they proposed a variational inference framework for learning ODEs from noisy and partially-observed data. However, they consider only low-dimensional ODEs and are restricted to regular grids.

Other works considered learning the latent space PDE dynamics using the 'encode-process-decode' approach. Pfaff et al. (2021) use GNN-based encoder and dynamics function and map the observations to the same spatial grid in the latent space and learn the latent space dynamics. Sanchez et al. (2020) use a similar approach but with CNNs and map the observations to a coarser latent grid and learn the coarse-scale dynamics. Wu et al. (2022) use CNNs to map observations to a low-dimensional latent vector and learn the latent dynamics. However, all these approaches are grid-dependent, limited to regular spatial/temporal grids, and require fully-observed data.

Interpolation has been used in numerous studies for various applications. Works such as (Alet et al., 2019; Jiang et al., 2020; Han et al., 2022) use interpolation to map latent states on coarse grids to observations on finer grids. Hua et al. (2022) used interpolation as a post-processing step to obtain continuous predictions, while Boussif et al. (2022) used it to recover observations at missing nodes.

Another approach for achieving grid-independence was presented in neural operators (Li et al., 2021; Lu et al., 2021), which learn a mapping between infinite-dimensional function spaces and represent the mapping in a grid-independent manner.

7 Conclusion

We proposed a novel space-time continuous, grid-independent model for learning PDE dynamics from noisy and partial observations on irregular spatiotemporal grids. Our contributions include an efficient generative modeling framework, a novel latent PDE model merging collocation and method of lines, and a data-efficient, grid-independent encoder design. The model demonstrates state-of-the-art performance on complex datasets, highlighting its potential for advancing data-driven PDE modeling and enabling accurate predictions of spatiotemporal phenomena in diverse fields. However, our model and encoder operate on every spatial and temporal location which might not be the most efficient approach and hinders scaling to extremely large grids, hence research into more efficient latent state extraction and dynamics modeling methods is needed.

References

Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas LozanoPerez, and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 212-222. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/ v97/alet19a.html .

Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Modelling spatiotemporal dynamics from earth observation data with neural differential equations. Machine Learning , 111 (6):2349-2380, 2022. doi: 10.1007/s10994-022-06139-2. URL https://doi.org/10.1007/ s10994-022-06139-2 .

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, Apr 2017. ISSN 1537-274X. doi: 10.1080/01621459.2017.1285773. URL http://dx.doi.org/10.1080/01621459.2017. 1285773 .

H.G. Bock and K.J. Plitt. A multiple shooting algorithm for direct solution of optimal control problems*. IFAC Proceedings Volumes , 17(2):1603-1608, 1984. ISSN 1474-6670. doi: https://doi. org/10.1016/S1474-6670(17)61205-9. URL https://www.sciencedirect.com/science/ article/pii/S1474667017612059 . 9th IFAC World Congress: A Bridge Between Control Science and Technology, Budapest, Hungary, 2-6 July 1984.

Oussama Boussif, Yoshua Bengio, Loubna Benabbou, and Dan Assouline. MAgnet: Mesh agnostic neural PDE solver. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=bx2roi8hca8 .

Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=vSix3HPYKSU .

Steven L. Brunton and J. Nathan Kutz. Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control . Cambridge University Press, 2019. doi: 10.1017/9781108380690.

Nicolas Cellier. scikit-fdiff, 2019. URL https://gitlab.com/celliern/scikit-fdiff .

Ricky T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq .

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 2018.

Alexander H.-D. Cheng. Radial basis function collocation method. In Computational Mechanics , pages 219-219, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-540-75999-7.

Cressie and C. K. Wikle. Statistics for Spatio-Temporal Data . Wiley, 2011. ISBN 9780471692744. URL https://books.google.fi/books?id=-kOC6D0DiNYC .

Marie-Lenat Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: A large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics , 38(6):239, 2019.

C. Evans. Partial Differential Equations . American Mathematical Society, 2010. ISBN 9780821849743.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https://proceedings.mlr.press/v9/glorot10a.html .

Hamdi, W. E. Schiesser, and G. W Griffiths. Method of lines. Scholarpedia , 2(7):2859, 2007. doi: 10.4249/scholarpedia.2859. revision #26511.

Xu Han, Han Gao, Tobias Pfaff, Jian-Xun Wang, and Liping Liu. Predicting physics in mesh-reduced space with temporal attention. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=XctLdNfCmP .

Pashupati Hegde, Cagatay Yildiz, Harri Lähdesmäki, Samuel Kaski, and Markus Heinonen. Variational multiple shooting for bayesian ODEs with gaussian processes. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022. URL https://openreview.net/forum?id= r2NuhIUoceq .

Charles Hirsch. Numerical computation of internal and external flows: The fundamentals of computational fluid dynamics . Elsevier, 2007.

Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurIPS Workshop on Differentiable vision, graphics, and physics applied to machine learning , 2020. URL https: //montrealrobotics.ca/diffcvgp/assets/papers/3.pdf .

Chuanbo Hua, Federico Berto, Michael Poli, Stefano Massaroli, and Jinkyoo Park. Efficient continuous spatio-temporal simulation with graph spline networks. In ICML 2022 2nd AI for Science Workshop , 2022. URL https://openreview.net/forum?id=PBT0Vftuji .

Valerii Iakovlev, Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Latent neural ODEs with sparse bayesian multiple shooting. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=moIlFZfj_1b .

Chiyu lMaxr Jiang, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A. Tchelepi, Philip Marcus, Mr Prabhat, and Anima Anandkumar. Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , Nov 2020. doi: 10.1109/sc41405.2020.00013. URL http://dx.doi.org/10.1109/SC41405. 2020.00013 .

Armand Jordana, Justin Carpentier, and Ludovic Righetti. Learning dynamical systems from noisy sensor measurements using multiple shooting, 2021.

E.J. Kansa. Multiquadrics-a scattered data approximation scheme with applications to computational fluid-dynamics-ii solutions to parabolic, hyperbolic and elliptic partial differential equations. Computers and Mathematics with Applications , 19(8):147-161, 1990. ISSN 0898-1221. doi: https://doi.org/10.1016/0898-1221(90)90271-K. URL https://www.sciencedirect.com/ science/article/pii/089812219090271K .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.

Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning-accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences , 118(21):e2101784118, 2021. doi: 10.1073/pnas.2101784118. URL https: //www.pnas.org/doi/abs/10.1073/pnas.2101784118 .

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=c8P9NQVtmnO .

Marten Lienen and Stephan Günnemann. Learning the dynamics of physical systems from sparse observations with finite element networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=HFmAukZ-k-2 .

Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3208-3216. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/long18a.html .

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, Mar 2021. ISSN 2522-5839. doi: 10.1038/s42256-021-00302-5. URL http://dx.doi.org/10.1038/s42256-021-00302-5 .

Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz, and Tal Kachman. Gradients are not all you need, 2021.

James D. Murray. Mathematical Biology I. An Introduction , volume 17 of Interdisciplinary Applied Mathematics . Springer, New York, 3 edition, 2002. doi: 10.1007/b98868.

Duong Nguyen, Said Ouala, Lucas Drumetz, and Ronan Fablet. Variational deep learning for the identification and reconstruction of chaotic and stochastic dynamical systems from noisy and partial observations. ArXiv , abs/2009.02296, 2020.

Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=roNqYL0_XP .

Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Re, and Stefano Ermon. Transform once: Efficient operator learning in frequency domain. In ICML 2022 2nd AI for Science Workshop , 2022. URL https://openreview.net/forum?id=x1fNT5yj41N .

Antônio H. Ribeiro, Koen Tiels, Jack Umenberger, Thomas B. Schön, and Luis A. Aguirre. On the smoothness of nonlinear system identification. Automatica , 121:109158, 2020. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2020.109158. URL https://www.sciencedirect. com/science/article/pii/S0005109820303563 .

Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic pdes: Resolutioninvariant learning of continuous spatiotemporal dynamics, 2021.

Alvaro Sanchez, Dmitrii Kochkov, Jamie Alexander Smith, Michael Brenner, Peter Battaglia, and Tobias Joachim Pfaff. Learning latent field dynamics of pdes. 2020. We are submitting to Machine Learning and the Physical Sciences workshop with a submission deadline on October 2nd.

William E. Schiesser. The numerical method of lines: Integration of partial differential equations. 1991.

Sungyong Seo, Chuizheng Meng, and Yan Liu. Physics-aware difference graph networks for sparselyobserved dynamics. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=r1gelyrtwH .

Kiwon Um, Robert Brand, Yun Fei, Philipp Holl, and Nils Thuerey. Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers. Advances in Neural Information Processing Systems , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Henning Voss, J. Timmer, and Juergen Kurths. Nonlinear dynamical system identification from uncertain and indirect measurements. International Journal of Bifurcation and Chaos , 14, 01 2004.

Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=xvZtgp5wyYT .

Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Ode2vae: Deep generative second order odes with bayesian neural networks. ArXiv , abs/1905.10994, 2019.

Yuan Yin, Matthieu Kirchmeyer, Jean-Yves Franceschi, Alain Rakotomamonjy, and patrick gallinari. Continuous PDE dynamics forecasting with implicit neural representations. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=B73niNjbPs .

A Appendix A

A.1 Model specification.

Here we provide all details about our model specification. The joint distribution for our model is

Next, we specify each component in detail.

Parameter priors. The parameter priors are isotropic zero-mean multivariate normal distributions:

where N is the normal distribution, 0 is a zero vector, and I is the identity matrix, both have an appropriate dimensionality dependent on the number of encoder and dynamics parameters.

Continuity prior. We define the continuity prior as

where N is the normal distribution, 0 ∈ R d is a zero vector, I ∈ R d × d is the identity matrix, and σ c ∈ R is the parameter controlling the strength of the prior. Smaller values of σ c tend to produce smaller gaps between the sub-trajectories.

Observation model

where N is the normal distribution, σ 2 u is the observation noise variance, and I ∈ R D × D is the identity matrix. Note again that z ( t i , x j ; t [ b ] , s b , θ dyn ) above equals the ODE forward solution ODESolve( t i ; t [ b ] , s b , θ dyn ) at grid location x j .

A.2 Approximate posterior specification.

Here we provide all details about the approximate posterior. We define the approximate posterior as

Next, we specify each component in detail.

Dynamics parameters posterior. We define q ψ dyn ( θ dyn ) as

where γ dyn and τ 2 dyn are vectors with an appropriate dimension (dependent on the number of dynamics parameters), and diag( τ 2 dyn ) is a matrix with τ 2 dyn on the diagonal. We define the vector of variational parameters as ψ dyn = ( γ dyn , τ 2 dyn ) . We optimize directly over ψ dyn and initialize γ dyn using Xavier (Glorot and Bengio, 2010) initialization, while τ dyn is initialized with each element equal to 9 · 10 -4 .

Decoder parameters posterior. We define q ψ dec ( θ dec ) as

where γ dec and τ 2 dec are vectors with an appropriate dimension (dependent on the number of decoder parameters), and diag( τ 2 dec ) is a matrix with τ 2 dec on the diagonal. We define the vector of variational parameters as ψ dec = ( γ dec , τ 2 dec ) . We optimize directly over ψ dec and initialize γ dec using Xavier (Glorot and Bengio, 2010) initialization, while τ dec is initialized with each element equal to 9 · 10 -4 .

Shooting variables posterior. We define q ψ j b ( s j b ) as

where the vectors γ j b , τ j b ∈ R d are returned by the encoder h θ enc , and diag([ τ j b ] 2 ) is a matrix with [ τ j b ] 2 on the diagonal. We define the vector of variational parameters as ψ j b = ( γ j b , [ τ j b ]) . Because the variational inference for the shooting variables is amortized, our model is trained w.r.t. the parameters of the encoder network, θ enc .

B Appendix B

B.1 Derivation of ELBO.

For our model and the choice of the approximate posterior the ELBO can be written as

Next, we will look at each term L i separately.

where KL is Kullback-Leibler (KL) divergence. Both of the KL divergences above have a closed form but the expectation w.r.t. q ( θ dyn , s b -1 ) does not.

B.2 Computation of ELBO.

We compute the ELBO using the following algorithm:

Sample θ dyn , θ dec from q ψ dyn ( θ dyn ) , q ψ dec ( θ dec ) .

Sample s 1: B by sampling each s j b from q ψ j b ( s j b ) with ψ j b = h θ enc ( u [ t [ b ] , x j ]) .

Compute u 1: M from s 1: B as in Equations 14-16.

Compute ELBO L (KL terms are computed in closed form, for expectations we use Monte Carlo integration with one sample).

Sampling is done using reparametrization to allow unbiased gradients w.r.t. the model parameters.

C Appendix C

C.1 Datasets.

SHALLOW WATER. The shallow water equations are a system of partial differential equations (PDEs) that simulate the behavior of water in a shallow basin. These equations are effectively a depth-integrated version of the Navier-Stokes equations, assuming the horizontal length scale is significantly larger than the vertical length scale. Given these assumptions, they provide a model for water dynamics in a basin or similar environment, and are commonly utilized in predicting the propagation of water waves, tides, tsunamis, and coastal currents. The state of the system modeled by these equations consists of the wave height h ( t, x, y ) , velocity in the x -direction u ( t, x, y ) and velocity in the y -direction v ( t, x, y ) . Given an initial state ( h 0 , u 0 , v 0 ) , we solve the PDEs on a spatial domain Ω over time interval [0 , T ] . The shallow water equations are defined as:

where g is the gravitational constant.

The spatial domain Ω is a unit square with periodic boundary conditions. We set T = 0 . 1 sec. The solution is evaluated at randomly selected spatial locations and time points. We use 1089 spatial locations and 25 time points. The spatial end temporal grids are the same for all trajectories. Since we are dealing with partially-observed cases, we assume that we observe only the wave height h ( t, x, y ) .

For each trajectory, we start with zero initial velocities and the initial height h 0 ( x, y ) generated as:

where N = 3 and λ kl , γ kl ∼ N (0 , 1) .

The datasets used for training, validation, and testing contain 60 , 20 , and 20 trajectories, respectively. We use scikit-fdiff (Cellier, 2019) to solve the PDEs.

NAVIER-STOKES. For this dataset we model the propagation of a scalar field (e.g., smoke concentration) in a fluid (e.g., air). The modeling is done by coupling the Navier-Stokes equations with the Boussinesq buoyancy term and the transport equation to model the propagation of the scalar field. The state of the system modeled by these equations consists of the scalar field c ( t, x, y ) , velocity in x -direction u ( t, x, y ) , velocity in y -direction v ( t, x, y ) , and pressure p ( t, x, y ) . Given an initial state

( c 0 , u 0 , v 0 , p 0 ) , we solve the PDEs on a spatial domain Ω over time interval [0 , T ] . The Navier-Stokes equations with the transport equation are defined as:

where ν = 0 . 002 .

The spatial domain Ω is a unit square with periodic boundary conditions. We set T = 2 . 0 sec, but drop the first 0 . 5 seconds due to slow dynamics during this time period. The solution is evaluated at randomly selected spatial locations and time points. We use 1089 spatial locations and 25 time points. The spatial and temporal grids are the same for all trajectories. Since we are dealing with partially-observed cases, we assume that we observe only the scalar field c ( t, x, y ) .

For each trajectory, we start with zero initial velocities and pressure, and the initial scalar field c 0 ( x, y ) is generated as:

where N = 2 and λ kl , γ kl ∼ N (0 , 1) .

The datasets used for training, validation, and testing contain 60 , 20 , and 20 trajectories, respectively. We use PhiFlow (Holl et al., 2020) to solve the PDEs.

SCALAR FLOW. This dataset, proposed by Eckert et al. (2019), consists of observations of smoke plumes rising in hot air. The observations are postprocessed camera images of the smoke plumes taken from multiple views. For simplicity, we use only the front view. The dataset contains 104 trajectories, where each trajectory has 150 time points and each image has the resolution 1080 × 1920. Each trajectory was recorded for T = 2 . 5 seconds.

To reduce dimensionality of the observations we sub-sample the original spatial and temporal grids. For the temporal grid, we remove the first 50 time points, which leaves 100 time points, and then take every 4th time point, thus leaving 20 time points in total. The original 1080 × 1920 spatial grid is first downsampled by a factor of 9 giving a new grid with resolution 120 × 213, and then the new grid is further sub-sampled based on the smoke density at each node. In particular, we compute the average smoke density at each node (averaged over time), and then sample the nodes without replacement with the probability proportional to the average smoke density (thus, nodes that have zero density most of the time are not selected). See example of a final grid in Figure 9. This gives a new grid with 1089 nodes.

We further smooth the observations by applying Gaussian smoothing with the standard deviation of 1.5 (assuming domain size 120 × 213).

We use the first 60 trajectories for training, next 20 for validation and next 20 for testing.

In this case the spatial domain is non-periodic, which means that for some observation location x i some of its spatial neighbors N S ( x i ) might be outside of the domain. We found that to account for such cases it is sufficient to mark such out-of-domain neighbors by setting their value to -1 .

Time grids used for the three datasets are shown in Figure 10.

0.1

0.0

0.1

0.0

0.2

0.4

0.6

0.8

1.0

NormalizedTime

0.1

0.0

0.1

0.0

0.2

0.4

0.6

0.8

1.0

NormalizedTime

0.1

0.0

0.1

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Time

C.2 Model architecture and hyper-parameters.

Dynamics function. For all datasets we define F θ dyn as an MLP. For SHALLOW WATER/NAVIERSTOKES/SCALAR FLOW we use 1/3/3 hidden layers with the size of 1024/512/512, respectively. We use ReLU nonlinearities.

Observation function. For all datasets we define g θ dec as a selector function which takes the latent state z ( t, x ) ∈ R d and returns its first component.

Encoder. Our encoder h θ enc consists of three function: h θ spatial , h θ temporal , and h θ read . The spatial aggregation function h θ spatial is a linear mapping to R 128 . The temporal aggregation function h θ temporal is a stack of transformer layers with temporal attention and continuous relative positional encodings (Iakovlev et al., 2023). For all datasets, we set the number of transformer layers to 6. Finally, the variational parameter readout function h θ read is a mapping defined as

where Linear is a linear layer (different for each line), and γ j b and τ j b are the variational parameters discussed in Appendix A.

Spatial and temporal neighborhoods. We use the same spatial neighborhoods N S ( x ) for both the encoder and the dynamics function. We define N S ( x ) as the set of points consisting of the point x and points on two concentric circles centered at x , with radii r and r/ 2 , respectively. Each circle contains 8 points spaced 45 degrees apart (see Figure 11 (right)). The radius r is set to 0 . 1 . For SHALLOW WATER/NAVIER-STOKES/SCALAR FLOW the size of temporal neighborhood ( δ T ) is set to 0 . 1 / 0 . 1 / 0 . 2 , respectively.

Multiple Shooting. For SHALLOW WATER/NAVIER-STOKES/SCALAR FLOW we split the full training trajectories into 4 / 4 / 19 sub-trajectories, or, equivalently, have the sub-trajectory length of 6 / 6 / 2 .

C.3 Training, validation, and testing setup.

Data preprocessing. We scale the temporal grids, spatial grids, and observations to be within the interval [0 , 1] .

Training. We train our model for 20000 iterations using Adam (Kingma and Ba, 2017) optimizer with constant learning rate 3 e-4 and linear warmup for 200 iterations. The latent spatiotemporal dynamics are simulated using differentiable ODE solvers from the torchdiffeq package (Chen, 2018) (we use dopri5 with rtol= 1 e-3, atol= 1 e-4, no adjoint). The batch size is 1.

Validation. We use validation set to track the performance of our model during training and save the parameters that produce the best validation performance. As performance measure we use the mean absolute error at predicting the full validation trajectories given some number of initial observations. For SHALLOW WATER/NAVIER-STOKES/SCALAR FLOW we use the first 5 / 5 / 10 observations. The predictions are made by taking one sample from the posterior predictive distribution (see Appendix C.4 for details).

Testing. Testing is done similarly to validation, except that as the prediction we use an estimate of the expected value of the posterior predictive distribution (see Appendix C.4 for details).

C.4 Forecasting.

Given initial observations ˜ u 1: m at time points t 1: m , we predict the future observation ˜ u n at a time point t n > t m as the expected value of the approximate posterior predictive distribution:

The expected value is estimated via Monte Carlo integration, so the algorithm for predicting ˜ u n is:

Sample θ dyn , θ dec from q ( θ dyn ) , q ( θ dec ) .

Sample ˜ s m from q (˜ s m ) = ∏ N j =1 q ψ j m (˜ s j m ) , where the variational parameters ψ j m are given by the encoder h θ enc operating on the initial observations ˜ u 1: m as ψ j m = h θ enc (˜ u [ t m , x j ]) .

Compute the latent state ˜ z ( t n ) = z ( t n ; t m , ˜ s m , θ dyn ) .

Sample ˜ u n by sampling each ˜ u j n from N (˜ u j n | g θ dec (˜ z ( t n , x j ))) , σ 2 u I ) .

Repeat steps 1-4 n times and average the predictions (we use n = 10 ).

C.5 Model comparison setup.

NODE. For the NODE model the dynamics function was implemented as a fully connected feedforward neural network with 3 hidden layers, 512 neurons each, and ReLU nonlinearities.

FEN. We use the official implementation of FEN. We use FEN variant without the transport term as we found it improves results on our datasets. The dynamics were assumed to be stationary and autonomous in all cases. The dynamics function was represented by a fully connected feedforward neural network with 3 hidden layers, 512 neurons each, and ReLU nonlinearities.

NSPDE. We use the official implementation of NSPDE. We set the number of hidden channels to 16, and set modes1 and modes2 to 32.

DINo. We use the official implementation of DINo. The encoder is an MLP with 3 hidden layers, 512 neurons each, and Swish non-linearities. The code dimension is 100 . The dynamics function is an MLP with 3 hidden layers, 512 neurons each, and Swish non-linearities. The decoder has 3 layers and 64 channels.

MAgNet. We use the official implementation of MAgNet. We use the graph neural network variant of the model. The number of message-passing steps is 5. All MLPs have 4 layers with 128 neurons each in each layer. The latent state dimension is 128.

D Appendix D

D.1 Spatiotemporal neighborhood shapes and sizes.

Here we investigate the effect of changing the shape and size of spatial and temporal neighborhoods used by the encoder and dynamics functions. We use the default hyperparameters discussed in Appendix C and change only the neighborhood shape or size. A neighborhood size of zero implies no spatial/temporal aggregation.

Initially, we use the original circular neighborhood displayed in Figure 11 for both encoder and dynamics function and change only its size (radius). The results are presented in Figures 12a and 12b. In Figure 12a, it is surprising to see very little effect from changing the encoder's spatial neighborhood size. A potential explanation is that the dynamics function shares the spatial aggregation task with the encoder. However, the results in Figure 12b are more intuitive, displaying a U-shaped curve for the test MAE, indicating the importance of using spatial neighborhoods of appropriate size. Interestingly, the best results tend to be achieved with relatively large neighborhood sizes. Similarly, Figure 12c shows U-shaped curves for the encoder's temporal neighborhood size, suggesting that latent state inference benefits from utilizing local temporal information.

We then examine the effect of changing the shape of the dynamics function's spatial neighborhood. We use n circle neighborhoods, which consist of n equidistant concentric circular neighborhoods (see examples in Figure 11). Effectively, we maintain a fixed neighborhood size while altering its density. The results can be seen in Figure 13. We find that performance does not significantly improve when using denser (and presumably more informative) spatial neighborhoods, indicating that accurate predictions only require a relatively sparse neighborhood with appropriate size.

x

x

x

x

X

X

X

x

(a)

(b)

(c)

0.040

0.050

0.060

Shallow_Water

ShallowWater

ShallowWater

0.025

0.035

0.025

0.010

0.010

MAE

0.080

MAE

0.160

Navier-Stokes

0.100

Navier-Stokes

MA

Navier-Btokes

Test

0.055

0.030

Test

0.050

0.020

0.090

0.050

0.080

0.060

ScalarFlow

ScalarFlow

0.045

0.050

0.050

0.040

0.020

0.0

0.05

0.1

0.2

0.0

0.05

0.1

0.2

0.0

0.1

0.2

0.4

Spatial Neighb.Size (Enc.)

Spatial Neighb.Size (Dyn.)

Temporal Neighb. Size (Enc.)

0.05

ShallowWater

TestMAE

0.00

0.05

Navier-Stokes

0.00

0.05

ScalarFlow

0.00

1circle

2circle

4circle

8circle

Spatial Neighb.Shape (Dyn.)

D.2 Multiple shooting.

Here we demonstrate the effect of using multiple shooting for model training. In Figure 14 (left), we vary the sub-trajectory length (longer sub-trajectories imply more difficult training) and plot the test errors for each sub-trajectory length. We observe that in all cases, the best results are achieved when the sub-trajectory length is considerably smaller than the full trajectory length. In Figure 14 (right) we further show the training times, and as can be seen multiple shooting allows to noticeably reduce the training times.

Time

0.025

O-

ShallowWater

Training

(hours)

-O-ShallowWater

0.020

0.015

2

T

T

2

5

13

25

Time

2

5

13

25

Test MAE

0.075

Navier-Stokes

Training‘

(hours)

4

Navier-Stokes

0.050

3

T

T

T

2

5

13

25

Time

2

5

13

25

0.075

ScalarFlow

Training

(sinou)

3

ScalarFlow

0.050

2

2

4

10

20

2

4

10

20

Sub-trajectorylength

Sub-trajectorylength

E Appendix E

Noisy Data. Here we show the effect of observation noise on our model and compare the results against other models. We train all models with data noise of various strengths, and then compute test MAE on noiseless data (we still use noisy data to infer the initial state at test time). Figure 15 shows that our model can manage noise strength up to 0.1 without significant drops in performance. Note that all observations are in the range [0 , 1] .

MAgNet

DINo

Ours

ShallowWater

Navier-Stokes

0.30

0.40

0.30

Test MAE

Test MAE

estMAE

0.23

0.30

0.23

0.15

0.20

0.15

0.07

0.10

3

0.07

0.00

0.00

0.00

0.0

0.1

0.2

0.5

0.0

0.1

0.2

0.5

0.0

0.1

0.2

0.5

F Appendix F

F.1 Model Predictions

We show (Fig. 16) predictions of different models trained on different datasets (synthetic data is partially observed).

F.2 Visualization of Prediction Uncertainty

Figures 17, 18, and 19 demonstrate the prediction uncertainty across different samples from the posterior distribution.

ShallowWater

Navier-Stokes

ScalarFlow

Data

DINo

MAgNet

Time

Data,Mean Prediction,and Standard Deviation

0.6

Observation

0.5

0.4

0.3

0

5

10

15

20

25

Time Index

Data,Mean Prediction,and Standard Deviation

0.5

Observation

0.4

0.3

0.2

0

5

10

15

20

25

Time Index

Data,MeanPrediction,andStandard Deviation

0.6

Observation

0.2

0.0

0.0

2.5

5.0

7.5

10.012.515.017.5

Time Index

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a grid-independent generative model to learn PDEs, which supports irregular grids. Amortized variational inference is employed for posterior approximation and multiple shooting, a recent method to train neural ODEs efficiently, is used and adapted to PDEs. \nResults show a quite significant dominance over two SOTA PDE learning methods.\n\nSTRENGTHS:\n- The paper is well written and very well presented. The motivation is clear, the model formalism is well presented, the general flow is good. I enjoyed reading the paper. \n\n- The figures are well made and appreciated to have an intuition about the different elements. \n\n- The contribution of the paper is the model itself, which appears to be a natural extension to the Iakovlev et al., 2023, adpated for a PDE instead of an ODE setting. It is therefore not the most original methodologically, but a completely proper and natural contribution. \nNote that even though grid-independence and irregular grid prediction capabilities have been studied before for PDE learning, (notably by the SOTA the method is comparing to) which again restrains the novelty of the paper, the present methodology seems to significantly outshine the previous methods, which therefore justifies the contribution.\n\nWEAKNESSES:\n- Relating to the previous comment regarding the contribution on the strength section: because it is not a new gap that the method is filling, but rather an improvement over SOTA methods, the contribution of the paper, besides being an extension of the multiple shooting for Neural ODE learning, relies on the performance in order to be strong. Therefore, it would be very useful to have more extensive experiments:\n   - of SOTA methods on finer and coarser grids, in order to test super resolution capabilities of the method compared to the other SOTA methods. An experiment is done on Figure 8, but not compared to SOTA methods\n   - to test prediction on long future horizon for all models including SOTA\n   - trying multiple grid sizes when training, instead of only multiple number of trajectories, as shown in Figure 10 \n\n- It seems that it lacks some important details\n    -  In the experiments regarding time specification, specifically the time horizon and the time resolution. It is mentioned in the appendix that \"25 time points\" are used for the two first data and 20 \"time points\" for the scalar flow, but it is not clear to me what they represent in real time. Maybe I missed it somewhere? If not, please mention it, as this is important when comparing to SOTA methods, as each method seems to have its strength, regarding spatial or time capabilities. \n\n    - In the figures: please add the time resolution on Figure 6 and 9 (how much is each time step?), and the horizon predicted on all the figures, to know what the MAE correspond to! If it is the same every time, please state it clearly somewhere (again, I apologize if I missed it...!)\n\n- How is the generative aspect of the method used? The fact that it is probabilistic is partly what makes it interesting as opposed to SOTA models (MAgNet and DINo, which are not generative), so it is a pity that no probabilistic outputs are used or shown in the results, but rather mean predictions.\n\nI think in general, the authors could move a bit of the formalism of the model to the appendix (which is also taken from Iakovlev et al., 2023 anyway, so that makes it easier), to let some space for more experiments and more details on the experiments, since, once again, they are in my view key to the contribution of the pure performance of the model!\n\nQUESTIONS:\nPlease refer to the weaknesses part for recommendations or modifications. \n\nRegarding other small questions or comments: \n\n- A general figure to describe the overall method would be very nice to have! Given, that the method is quite complex and gathers many different elements, it would really help grasping the whole framework, at least at a high level. \n\n- For the spatial aggregation, you use an MLP, like for example in MAgNet. The latter actually shows in the paper, in an ablation study, that the use of the MLP interpolator, is actually not so beneficial with respect to a cubic interpolator. Have you tried other interpolators, perhaps in an ablation study? Since this is the key to the grid-independent capabilities of the model, I think it is worth a part in the paper. \n\n- I think the C.4 appendix section on Forecasting should be put in the main text as it is quite crucial to the method. \n\n- I realized from the appendix training details that the batch size is 1. Was the data so big that it was impossible to fit a higher batch size in memory? Didn't it create training stability issues?\n\n- I think it is a bit unncessary to split the contributions in 3 the way you do it in the introduction : generative model, PDE model, encoder. To my understanding, the model you are proposing is exactly the mix of the 3, so I think it is better to simply do this one strong claim: you designed one PDE learning model, with all its characteristics.\n\nLIMITATIONS:\nThe authors addressed some limitations about the scalability of the method, but not much else; perhaps there are other points, like the training speed or the lack of special geometries taken into account within the method, or super-resolution capabilities still to be compared to SOTA? I do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a new method in learning PDEs by combining interpolation and NODEs. The difference from NODEs is that the authors uses spatial derivatives in hidden dynamics as PDE, compute the loss between shooting gaps with KL divergence, and compute the initial conditions with a transformer. The authors then support their claim by experimenting with three datasets, shallow water, navier-stokes, and scalar flow.\n\nSTRENGTHS:\n- The authors present a combination of various methods as a new solution to learning PDEs. \n- The performance of the method, on the particular datasets the authors choose, are quite well.\n\nWEAKNESSES:\nOverall, the proposed method might be nice, but the paper itself is very hard to read.\n- In section 3.2 generative modeling, there are two paragraphs discussing multiple shooting. There is no discussion on how multiple shooting is used on generative modeling.\n- There are too few captions for the figures. In figure 9, which parts are the time you interpolate? Which parts are extrapolation / forecasting? \n- The notations $q_\\psi(\\theta)$ is so confusing.\n\nQUESTIONS:\n- In fig. 4 minimizing gaps, which norm do you use? Do you instead maximize the probability / likelihood in equation 17? In some commonly used PDEs, such as wave equations, derivatives of the PDE also matters. Did you take them into account? Or does your method only work on specific type of PDEs?\n- The experimental results in the paper is very different from the experimental results in DINO paper. Do you use a different experimental setup?\n- Are figure 8 and figure 9 partially the same?\n- What is a true latent dimension in line 225?\n- What are $h$ in figure 5? Are they intemediate states? But in equation 22 they look like functions. Are they the same $h$?\n- Also in figure 5, did you add extra observations around $x_j$? Or are they within the dataset? Did you assume all observations are on the same position but not grid-like? Then why don't we use NODEs with finite element meshes?\n- What types of problems are these experiments? Why are they interesting? Are they experiments from the SOTA / benchmark you compare with, or are they associated with any practical problems in application?\n- For those synthetic experiments, what are the noise level? What is the hidden equation? Navier-stokes might be self evident, but what about shallow water?\n- Do the models rely on initialization? How much runs did you run your experiments?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a latent variable model for PDEs with an encoder-decoder architecture that evolves the latent variables in time with a neural ODE. The model achieves independence of the locations of the evaluation points by linearly interpolating the data onto points distributed in a fixed pattern around each evaluation point.\n\nSTRENGTHS:\n- The paper's main contribution is the grid-independence of the model by interpolating observations and latent states onto fixed neighboring points around each observation point\n- The experimental section investigates the effect of the latent state dimension and the size of the training dataset\n\nWEAKNESSES:\n- The collocation method is a function space method that finds solutions that solve the PDE at a given set of collocation points. I don't see how this is used in this paper. The authors refer to the general form of first-order-in-time PDE in Eq (4) as the collocation method, but Eq (4) is just a general form of a PDE and not a method.\n- The method of lines is only about eliminating spatial derivatives by discretizing the solution for each $t$ in a function space and does not contain any notion of data-driven models or being restricted to evaluations of $z(t, x)$ as claimed in lines 88-89\n- Generative modeling of PDE solutions (section 3.2) is neither motivated nor evaluated in experiments\n- Use of bayesian deep learning (variational inference of model parameters, section 3.2) is not motivated\n- Overall, the approach is quite similar to the one presented in (Lienen and Günnemann, 2022) in terms of method-of-lines and interpolation of data, though the authors only mention but don't compare to them. In contrast to the author's writing in their related work section, I would even see this as the most closely related work.\n- Application of method of lines is not novel as claimed in line 37-38, see (Iakovlev et al., 2020) and (Lienen and Günnemann, 2022)\n- This paper does not actually propose/apply a collocation method as claimed in line 37-38\n- As a result, novelty and significance of this work are limited\n\n(Iakovlev et al., 2020): https://arxiv.org/abs/2006.08956\n(Lienen and Günnemann, 2022): https://arxiv.org/abs/2203.08852\n\nQUESTIONS:\n- What are the advantages of modeling PDEs in a latent space instead of data space? Do you have any experimental insight into this question?\n\nLIMITATIONS:\nThe authors did not discuss any limitations of their model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method for learning time-dependent PDE solutions with noisy, partially-observed data on irregular grids. This setting is quite challenging and is aligned with real-world data acquisition. They adopt a generative framework and combines two techniques for solving PDEs: the collocation method, which is used to approximate spatial derivatives, and the method of lines to propagate the PDE solution forward in time. At the core of their architecture lies a spatio-temporal encoder, that aggregates for every spatial/ temporal coordinate pair $(x_j, t)$ information over the spatial neighborhood $\\mathcal{N}_S(x_j)$ and the temporal neighborhood $\\mathcal{N}_T(t)$. They tackle the bayesian problem with a variational formulation, and learn the overall model by maximizing an ELBO objective. They use sparse bayesian multiple shooting to stabilize and accelerate the training. They validate their framework on three different datasets: Shallow-Water (SW), Navier-Stokes (NS) and Scalar Flow (SF).\n\nSTRENGTHS:\nThis is overall a good technical contribution. The paper shows solid experimental results, outperforming DINo and MAgNet on the three different datasets. The idea of using a generative model for solving PDEs is interesting, and the combination of all the different blocks is far from trivial to train. The explanation of the architecture is clear, though the notations are sometimes a bit hard to follow. The authors also evaluate the capabilities of the model to adapt to coarser and finer grids to support their claims.\n\nWEAKNESSES:\n* The paper is quite difficult to read overall. The introduction and related work are very brief, especially for such a difficult topic. It would help the reader to add some background from the PDE Deep Learning literature in those sections. For instance, there is no mention of Neural Operators which has been a popular topic recently, and there is no clear explanation of the limitations of existing state-of-the-art methods. __N.B.__: DINo which tackles a very similar problem (except for noiseless data), is not cited in the introduction nor the related work but is still used as the main baseline. __N.B.2__: The Section 3.2 with the explanation of the multiple shooting framework is a bit confusing and does not add much in terms of explanation of the model. On the other hand, inference is only exposed in the Appendix and should be in the core of the paper. \n\n* Though the idea of spatiotemporal encoding is sound, its realization does not appear to be very elegant and some limitations that could contradict the original claim of spatial continuity come to mind. What if the interpolation in the spatial neighborhood is not applicable ? Let's say you have an obstacle in the domain or you want to know the solution at the boundaries of your system, how can this work ? In all the figures the frames seem to contain only the convex hull of the set of points. This is worrying in terms of possible application to domains with complex geometries which are of main interest for people working in computational dynamics.\n\n* The results shown in Figure 10 are very impressive, especially for Navier-Stokes (NS) and Shallow-Water (SW). However I wonder, how can the model generalize this well on new test initial conditions with only 2 training trajectories? In terms of MAE, the method is already at 0.03 and 0.07 with two training samples for SW and NS while it reaches 0.015 and 0.041 with 60 samples. Do you have a possible explanation for this phenomenon ?\n\n* I understand that the main motivation of the paper is to develop a framework for partially-observed, noisy, irregularly-spaced observations, and therefore there are not a lot of suitable candidate baselines. Still, I think it would help readers understand the difficulty of the tasks to have a comparison with more classical neural operators and pde solvers on regular settings.\n\nQUESTIONS:\n* Did you try interpolant functions other than linear interpolation ? Similarly, have you tried the model with spherical data to see how the interpolation on a sphere behaves ? \n* How does the model scale with the number of grid points ? ~1100 points for a 2D domain does not appear to be of really high-resolution. Is the method still applicable for instance with standard grids used in the community such as 64x64 or 64x128 points ?\n* Is the method able to capture high-frequency patterns that occur in turbulent flows ? \n* Is it possible that the proposed model overfitted the training horizon and as a result obtained better results than DINo and MAgNet ? This would be in line with the training-size analysis. How does the model extrapolate over the training horizon ?\n\nLIMITATIONS:\nThere is no limitation section. The authors discuss briefly the difficulties for the method to scale with the numbers of points.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a grid independent model for learning PDEs from noisy experimental data. The proposed framework is probabilistic with an encoder that handles data efficiency and makes the solution grid independent. I think better differentiating from Ayed et al. and DINO would be helpful for the novelty contribution even though experimentally in these settings, the proposed model performs better.\n\nSTRENGTHS:\n- The problem setting is important to be able to learn PDEs from noisy and sparse observation data.\n- The paper is well-written with a good introduction of PDEs in the intro section.\n- Good overview of the related data-driven methods, such as Neural Operators (Li et al., 2021) and MeshGraphNets (Pfaff et al., 2021).\n- Grid independent solvers are advantages of deep learning models over numerical methods. It is a nice idea to define a neighborhood rather than use the adjacent gridpoints.\n- The proposed method is a nice application of the methods of lines.\n- The problem statement is well-defined. \n- It is good that the authors are considering the real-world setting of extrapolation in time.\n- Adding probabilistic output is also important contribution.\n- Synthetic and real-world datasets are both tested including a nice set of benchmark challenging datasets, e.g., Shallow water and Navier stokes. The scalar flow real-world camera dataset is also interesting.\n- Nice experimental setting going from a given grid to a coarser one and vice vera to finer.\n- The data efficiency results are nice.\n\nWEAKNESSES:\n- There is no guarantee that the PDEs learned from data will be better than the classical numerical PDE modeling so the first paragraph of the introduction could be modified a bit.\n- The authors should better motivate the irregularly spaced noisy and partial observations in this context. In this case, Gaussian Processes or Attentive Neural Processes could also be considered as baselines. In particular \"Learning Physical Models that Can Respect Conservation Laws\" [Hansen et al., ICML 2023] uses a constrained generative ANP model to perform the interpolation task from noisy sparse data to a fine grid. This work also uses a Gaussian observation model as in Eqn. 9.\n- There are advantages to operator-based methods vs. interpolation. I think a comparison to those methods, which are resolution-independent would be beneficial as well and should also be added to the related work section rather than just considering interpolation based methods.\n- Small number of comparisons to only DINo and MAgNet as baselines. Neural Operator and MeshGraphNets could also be considered as baselines.\n- The use of Neural ODEs is similar to the approach in \"Continuous PDE Dynamics Forecasting with Implicit Neural Representations\" [Yin et al., ICLR 2023]. In this work, the authors parameterize the latent space using implicit neural representations and then solve the ODE using Neural ODEs. It is good that the authors benchmark against this work in DiNO\n- The GNN based models such as MeshGraphNet also assume the derivatives are computed using the neighboring points as mentioned on lines 97-99. While this is important to highlight, since the approach here uses a neighborhood rather than grid points, it may be better to move this to a related work section so the novelty and design of the proposed method is only discussed in 3.1.\n- The model seems to be connecting various methods from different papers such as Neural ODEs and Iakovlev et al., 2023. It may be good to highlight and clarify the novelty in particular with respect to Iakovlev et al., 2023.\n- It looks like several heuristics need to be applied to make Neural ODEs work here. Also see \"Learning continuous models for continuous physics\" [Krishnapriyan, 2022]. It may be simpler to use a discrete numerical time-stepping method as done in \"CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations\" [Chen et al., ICLR 2023]. This is also used in the MAgNet baseline and I don't think the only changes in the time-stepping scheme from discrete Euler to Neural ODEs (also done in DiNO) is sufficient for the novelty for publication. Forward Euler is also unstable in many cases as shown in the aforementioned Krishnapriyan, 2022 and higher order schemes like Runge-Kutta 4 (RK4) may be beneficial.\n- The overall model has many components and is quite complicated. I was wondering if any ablations were run to see which component is the most important.\n- 3-4 hours depending on the problem size and architecture is a bit slow. I think cost-accuracy comparisons to numerical solvers in the experimental section would be beneficial.\n- Figures 7-8 take significant space and may be moved to the appendix as ablations especially since they also show the results of the proposed method. The error plots between the predicted and exact may be easier to visualize.The scalar flow results look diffused.\n- I think stating 'very high accurate predictions\" on line 254 is a bit strong. The magnitude of the error is still high 1e-2 in comparison to numerical solvers\nMinor\n- For notation $u_i^j$ in numerical methods, the superscript typically notates time and subscript notates spatial coordinate.\n- I would give Section 3 a more descriptive title than Method and also the name the proposed method as well.\n- Can clarify on line 226 that the true latent space dimension for Navier-Stokes and Shallow water is 3 due to the x,y components of the velocity and the scalar pressure.\n- Capital A on line 236\n- \"Similarly\" to similar on line 240\n- No parathesis in references on line 272.\n\nQUESTIONS:\n1. I know the authors mention that they use linear interpolation in the results in this paper and I was wondering if they performed any ablations with different types of interpolation.\n2. It is a bit of strong assumption that the neighboring points are sufficient to compute all the derivatives. The authors mention finite differences here but higher order finite difference methods can use more grid points in the stencils than just the neighbors. Can higher-order stencils be considered here? It would also be good to add a reference for finite difference methods, e.g. LeVeque's numerical analysis book.\n3. The authors mention that they test neighborhoods of various shapes and sizes. I think this is an important ablation. How ere two concentric circles chosen? The shapes are very important in numerical methods such as finite elements and those that use a Voronoi tessellation. In this case, it's a bit hard for me to see how it is purely grid independent. The number of evaluation points must also be carefully chosen.\n4. Also in Equation 8, the ODEs are defined only at the gridpoints and then interpolation is performed at every time step in the ODE solver - how expensive is this?\n5. Why is the distribution modeled as Gaussian and can the model be extended to support other distributions?\n6. How is the temporal neighborhood size delta_T determined? Since the model performs extrapolation in time shouldn't this neighborhood only depend on the prior time steps?\n7. Were simpler architectures than Transformers tested for the temporal encodings?\n8. I understand that latent space dimension for the synthetic 2D datasets but for the scalar flow, how do we interpret the improvement in prediction quality for d=5 instead of d=1 despite similar metrics? Can you prove any relation between latent dimension d and the error since it seems experimentally to decay monotonically?\n\nLIMITATIONS:\nLimitations on the expense of the proposed method and future work are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nModeling systems with spatiotemporal evolutions, such as the ones arise in problems governed by PDEs, is challenging. This is more pronounced when the system's underlying mechanisms are too complex or unknown. The authors propose a grid-independent generative model from noisy and partial observations on irregular domains. The latent state dynamics is constructed by merging the ideas inspired by the classical numerical PDE analysis such as collocation method and the method of lines, which is discussed in section 3.1 and 3.2. They also propose a novel encoder design that operates on local spatiotemporal neighborhoods for improved data-efficiency and grid-independence. The authors apply that their model three use cases from two synthetic analysis and one from real-world datasets.\n\nSTRENGTHS:\nMultiple shooting analysis for training datasets from dynamical systems with long time simulations is a creative idea to enable reasonable cost of learning and avoid learning instabilities. \nIn Appendix D, they analyzed the impact of radius and multiple shooting on the overall performance of the model to further complete the analysis.\n\nThe examples provided are sophisticated enough to show that the framework can be applied to even more realistic scenarios.\n\nWEAKNESSES:\nI have two major concerns about this work: \n1.\tThere are various heuristic choices/assumptions with no concrete proofs. For example, the choice of linear interpolant to map the values at the grid points of z(t) to z(t,x) is arbitrary. Moreover, when authors write the dynamical system in form of equation 7, while they are inspired by collocation methods, they are not actually utilizing such method. Instead, they argue that the functions and its (sufficiently smooth) derivatives are a function of neighborhood locations. This is another heuristic assumption which prevents a careful convergence analysis. Can authors prove if the radius tends to zero, such solution converges to the “real” solution? In what order the error decreases? The analysis in Appendix D is in fact anecdotal and does not mathematically guarantee a desired performance. Finally, regarding multiple shooting method, while I think this is a very nice idea, it’s not novel. For example, see \"50 years of time parallel time integration.\" By Martin, where several other methods other than multiple shooting are proposed to accelerate ODE integration. While I don’t see any fundamental error, I am not convinced that the analysis is mathematically sounds for computational adoption.\n2.\tThe topic of the paper is closely related to model discovery, which has been investigated in PDE community for quite some time now. A more appropriate comparison would be the one with SINDy (\"Discovering governing equations from data by sparse identification of nonlinear dynamical systems.\" By Brunton et al), Neural ODE (Neural Ordinary Differential Equations by Chen et al), or auto-encoder-based discussions (Data-driven discovery of coordinates and governing equations by Champion et al). Please also note, recently people also used neural ODE with collocation points (Physics-informed neural ODE (PINODE): embedding physics into models using collocation points by Sholokhov et al). To better position this work in the larger community, a much more detailed comparison is needed.\n\nQUESTIONS:\nThe fact that the number and positions of sensor (observation locations) change with time may actually help the problem at the time of inference (Reconstruction). Have authors considered a fixed setting for which the observation locations are the same? Is there a minimum number of sensors required for a certain accuracy? Are there any insights about using this approach for sensor placement problems?\n\nIn equation 3, authors propose using a linear interpolation: what about other choices of interpolant? Any studies show performance of other choices? Seems to me that a linear interpolant may require a larger number of observation locations? Any insights?\n\nIn equation 7, what is the order of such analysis? Can authors prove this converges to actual dynamics? Seems heuristic. By order, I mean the order of accuracy of the approximation of the derivative in terms of the distance to neighborhood points (big O s^n).\n\nIn equations 12 and 13, what is the implication of assuming that the continuity inducing prior is Gaussian? Such assumption makes the solution of the problem computationally tractable but what if the actual process is non-Gaussian? What are limitations?\n\nIn the results section, for shallow water the model for d=5 (and maybe even 4) outperforms fully observed MAE which is based on true d=3. Are there any insight for this?\n\nMinor comment: the notation for $\\partial z/\\partialx_x$ on line 56 and elsewhere with a dot is a bit obscure. Maybe more common choice for notation is to use n as the degree of differentiation. \nAlso, A is capitalized on line 236.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}