$\Phi$-DVAE: Learning Physically Interpretable Representations with Nonlinear Filtering
Authors: Alex John Glyn-Davies
Connor Duffin
Omer Deniz Akyildiz
Mark Girolami
Venue: Submitted to ICLR 2023 (2023)
Keywords: variational autoencoder
nonlinear filter
physics-informed
parameter estimation
variational inference
Bayesian inverse problems

Abstract: Incorporating unstructured data into physical models is a challenging problem that is emerging in data assimilation. Traditional approaches focus on well-defined observation operators whose functional forms are typically assumed to be known. This prevents these methods from achieving a consistent model-data synthesis in configurations where the mapping from data-space to model-space is unknown. To address these shortcomings, in this paper we develop a physics-informed dynamical variational autoencoder ($\Phi$-DVAE) for embedding diverse data streams into time-evolving physical systems described by differential equations. Our approach combines a standard (possibly nonlinear) filter for the latent state-space model and a VAE, to embed the unstructured data stream into the latent dynamical system. A variational Bayesian framework is used for the joint estimation of the embedding, latent states, and unknown system parameters. To demonstrate the method, we look at three examples: video datasets generated by the advection and Korteweg-de Vries partial differential equations, and a velocity field generated by the Lorenz-63 system. Comparisons with relevant baselines show that the $\Phi$-DVAE provides a data efficient dynamics encoding methodology that is competitive with standard approaches, with the added benefit of incorporating a physically interpretable latent space.

Under review as a conference paper at ICLR 2023
Φ- Φ-DVAE: LEARNING PHYSICALLY INTERPRETABLE REPRESENTATIONS WITH NONLINEAR FILTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Incorporating unstructured data into physical models is challenging problem that is emerging in data assimilation. Traditional approaches focus on well-defined observation operators whose functional forms are typically assumed to be known. This prevents these methods from achieving a consistent model-data synthesis contigurations where the mapping from data-space model-space is unknown, To address these shortcomings, in this paper we develop physics-informe dynamical variational autoencoder (Ø-DVAE) for embedding diverse data streams into time- evolving physical systems described by differential equations. Our approach combines standard (possibly nonlinear) filter for the latent state-space model and a VAE, to embed the unstructured data stream into the latent dynamical system. A variational Bayesian framework is used for the joint estimation of the embedding. latent states, and unknown system parameters, To demonstrate the method, we look three examples: video datasets generated by the advection and Korteweg-de Vries partial differential equations, and a velocity field generated by the Lorenz-63 system. Comparisons with relevant baselines show that the 4-DVAE provides a data efficient dynamics encoding methodology that is competitive with standard approaches, with the added benefit incorporating a physically interpretable latent space.
INTRODUCTION
Physical models represented by ordinary, stochastic, or partial differential equations are ubiquitous throughout engineering and the physical sciences. These differential equations are the synthesis of scientific knowledge into mathematical form. However, as description of reality they are imperfect (Judd& Smith, 2004), leading to the well-known problem model misspecification (Box) 1979). At least since Kalman (1960) physical modellersls with observations (Anderson & Moore 1979). Such approaches are usually either solving the inverse problem of attempting to recover model parameters from data, and/or, the data assimilation (DA) problem of conducting state inference based on a time-evolving process.
For the inverse problem, Bayesian methods common Tarantola 2005 Stuart 2010). In this, prior belief in model parameters is updated with data y to give a posterior distribution, p( (|ly) This describes uncertainty with parameters given the data and modelling assumptions. DA can also proceed from a Bayesian viewpoint, where inference is cast as nonlinear state-space model (SSM) (Law al. 2013; Reich & Cotter 2015). The SSM typically the combination of time-discretised differential equation and an observation process: uncertainty enters the model through extrusive, additive errors. For latent state variable u, representing some discretised system time n. with observations y., the object of interest is the filtering distribution p(u,.lY::)) where yı.z == {yx )= Additionally, the joint filtering and estimation problem, which estimates pl [u,, A 1 has received significant attention in the literature (see, e.g., Kantas et al. (2015) and references therein). This has been well studied in, c.g., electrical engincering (Storvik 2002), geophysics (Bocquet & Sakov 2013), neuroscience (Ditlevsen & Samson 2014). chemical engineering (Kravaris et al. 2013)). biochemistry (Dochain 2003), and hydrology (Moradkhani et al., 2005). to name a few.
Typically in data assimilation tasks, while parameters of observation model may be unknown, the observation model itself is assumed known (Kantas et al. 2015). This assumption breaks down in settings where data arrives in various modalities, such videos, images, audio, hindering the

---

Under review as a conference paper at ICLR 2023
ability to perform inference. However, in such cases often the underlying variation in the data stream is due to a latent physical process, which is typically at least partially known.
In this work, these data streams video data and velocity fields. We develop variational Bayes (VB) (Blei et al. 2017) methodology which jointly solves the inverse and filtering problems for the case in which the observation operator is unknown. We model this unknown mapping with a variational autoencoder (V (VAE) (Kingma & Welling, (014), which encodes the assumed time- dependent observations VI.N into pseudo-data XI:N latent space. On this latent space, we stipulate that the pseudo-obserrations are taken from a known dynamical system, given by stochastic ordinary differential equation (ODE) or partial differential equation (PDE) with possibly unknown coefficients. The differential equation is also assumed to have stochastic forcing, which accounts for possible model misspecification. The stipulated system gives a structured prior p( (N [A) which acts physics-informed regulariser whilst also enabling inference over the unknown A. This prior approximated using classical nonlinear filtering algorithms. Our framework is fully probabilistic: inference proceeds from a derived evidence lower bound (ELBO), enabling joint estimation of unknown network parameters and unknown dynamical coefficients via VB. To set the scene for this work, now review the relevant literature.
2 RELATED WORK
As introduced above, VAEs (Kingma& & Welling 2014) are popular high-dimensional encoder. A VAE defines a generative model that learns low-dimensional representations, x, of high-dimensional data, y, using VB. To perform efficient inference, variational approximation 4s (x y) is made to the intractable posterior p(xly). Variational parameters Φ are estimated via optimisation of the ELBO. This unsupervised learning approach infers latent representations of high -igh-dimensional data. Recent works have extended the VAE to high- digh-dimensional time-series data YI:N. indexed by time n, with the aim of jointly learning latent representations XI:N, and a dynamical system that evolves them. These dynamical variational autoencoder (DVAF) methods (Girin et al. 2021) enforce the dynamics with a structured prior p(X1.N) on the latent space.
Various DVAE methods have been proposed. The Kalman variational autoencoder (K VAE) of Frac- caro et al. (2017) is popular approach, which encodes yı.N into latent variables x X1:N that are assumed to be observations of a linear Gaussian state-space model (LGSSM), driven by latent dynamic states uI.N. Assumed linear dynamics are jointly learnt with the encoder and decoder, via Kalman fittring/smoothingg Another approach is the Gaussian process variational autoencoder (GPVAE) (Pearce 2020, Jazbee et al., 2021 Fortuin et al., 2020), which models X1:V as a tempo- rally correlated Gaussian process (GP). The Markovian variant of Zhu et al. (2022) allows for similar Kalman procedure as in the KVAE, except, in this instance, the dynamics are known and are given by an stochastic differential equation (SDE) approximation to the GP (Hartikainen & Sarkka 2010). A related approach is provided for control applications in Watter et al. (2013); Hafner et al. (2019). where locally linear embeddings are estimated. Yiildiz et al. (2019) also propose the so-called ODE which encodes the data to an initial condition which is integrated through time using a Bayesian neural ODE (Chen al. 2018). This trajectory, only, is used to generate the reconstructions
via the decoder network.
A related class of methods are deep SSMs (Bayer & Ösendorfer 2014 Krishnan et al. 2015 Karl cet al. (2017). These works assume that the parametric form of the SSM is unknown, and replace the transition and emission distributions with neural network (NN) models, which are trained based on an ELBO. They harness the representational power of deep NNs to directly model transitions between fiigh-immnnsinnal states. More emphasis is placed on generative modelling and prediction than representation learning, or system identification. We also note the related VAE works of Wu jt al. (202T): Franceschi et al. (2020); Babaeizadeh et al. (2022), which use VAE-type architectures for similar video prediction tasks. In Chung et al. (2015) the variational recurrent neural networks (VRNN) attempt to capture variation in highly structured time-series data, by pairing a recurrent NN for learning nonlinear state-transition transitinniiiinss with sequential latent random variable model.
Methods to include physical information inside of autoencoders have been studied in the physics community. A popular approach uses SINDy (Brunton et al.; (016)) for discovery of dinedinenai latent dynamical systems using autoencoders (Champion et al. 2019). A predictive framework is given in Lopez& & Atzherger (202T), which aims to learn nonlinear dynamics by jointly optimizing
2

---

Under review as a conference paper at ICLR 2023
an ELBO. Following notation, this learns some function which maps un Un+ for some k. via a VAE. Lusch et al. 2018) use a physics- informed autoencoder to linearise nonlinear dynamical systems via a Koopman approach; inference is regularised through incorporating the Koopman structure in the loss function. (Oto & Rowley (019) present a similar method, and an extension of these approaches to PDE systems is given Gin al. (202T). MMorton al. (2018) use the linear regression methods of Takeishi et al. (2017) within a standard autoencoder to similarly compute (2019) derive an autoencoder which incorporates linear Markovian prediction operator, similar to Koopman operator, which uses
the Koopman observables. Erichson al.
regulariser to promote Lyapunov stability. Hernández et al. (2018) studies VAE methods to encode high-dimensional dynamical systems. Finally, we note the related work which studies the estimation of dynamical parameters within so-called "gray-box" systems, blending NN methods with known
physical laws (Lu et al. 2020:
Yin et al. 2021
Long et al. 2018 de Bézenac et al. 2019).
Our contribution. In this paper we propose a physics-informed dynamical variational autoeneoder ( DVAE approach which imposes the additional structure of known physics the latent space. We assume that there is a low-dimensional dynamical system generating the high-dimensional observed time-series: a NN is used to learn the unknown embedding this lower dimensional space. On the diweeddimessooal space, the embedded data are pseudo-observationn obseevationss a latent dynamical system, which is, in general, derived from numerical discretisation of nonlinear PDE. However, the methodology is suitably generic, allowing for ODE latent systems. Inference on this latent system is done with efficient nonlinear stochastic filtering methods, enabling the use of mature DA algorithms within our framework. Our approach follows probabilistically coherent VB construction and allows for joint learning of both the embedding and unknown dynamical parameters.
Instead of learnt, linear dynamics with the KVAE (Fraccaro et al. 2017). DDDVAE assumes a possibly misspecified nonlinear differential equation is driving the variation in the latent space, with possibly unknown parameters. This is in contrast to incorporating generic physical structure in the latent space (such as Koopman structure (Lusch et al. 2018), or Lyapunov stability (Erichson et al. 2019)), or generic temporal structure (such as in the GPVAE (Pearce 2020 Jazbec et al.) 2021 1
Fortuin et al.) ). Our latent dynamical systems give a known functionnl form of the latent transition density, instead of the learnt, NN- -N-paaameterised. transition and emission densities in deep SSMs (Bayer & Osendorfer 2014; Krishnan et al. 2015 Karl et al. 2017). Similarly, whilst we share commonality with latent differential equation, the & DVDE differs with the VDE VAE YYildiz al. 2019) as we perform inference with this ODE/PDE, instead of learning it and leveraging it to deterministically evolve the latents. Our approach trades off the generality of latent system discovery against the ability to infer physical quantities of interest relating to particular latent system. -DDVAE can infer physical parameters and states, solving the joint filtering and parameter estimation problem in scenarios where the observation model is unknown.
THE PROBABILISTIC MODEL
In this section we define our probabilistic model; our presentation roughly follows the structure of the generative model. We first give an overview of the dependencies between variables, as described by conditional probabilities. We then cover the latent differential equation model used to describe the underlying physics. Then, the pseudo-observation model is covered, followed by the decoder and the encoder. To be precise, we assume general SSM:
A p(A), un un/Uu,-1,4 A ~ p(u, A,,, X,, un ~ /n
YnlXa ~ (x));
evolving w.r.t. the dynamic model p( u, 12- 1, A), v describes the parameters of the likelihood denoted by pu Pxxx/un, and 0 describes NN parameters for the decoder pe (y Xx.). The conditional independence structure imposed by the model gives
where A describes the parameters of the Markov process (u, 1.
P(YEN, XI:N, UI:N A) po(yi.N(xxi:N)>./<a>> (ow.wwlunnppuuu (A)((())).)....))))))))))))))(
(1)
Intuitively, {yn N is the sequence of nigh-dimensional video frames, {x, is its embedding (or the data), and {u, IN is the latent physics process. For each n. we assume that yn e ) (with dim()) = ny)) . X,, E R"* u, E R²,, and A E Ra,, In what follows, we describe the components of our probabilistic model in detail.
3

---

Under review as a conference paper at ICLR 2023
YI:N
X1:N
V1:V
Decoder
Encoder
Pseubo-ohervatioss (x,, u.,))
Latent Dynamics p(u, U, A)
11
Dynamic Parameters p(A)
Figure 1: An illustration of the +-DVAE model. On the left, the frames of video be seen which are denoted Y1.N. These are converted into physically interpretable oow-dimessional encodings X1:N using am encoder. This learning process informed by the physics driven state-space model which treats X1:N as psendo-obserrations,, which can be seen on the bottom right.
3.1 DYNAMIC MODEL
The first component of the generative model is the latent dynamical system p(u, ((u,(U,-1,, A)). In general, model this latent physics process {u, W ODE latent physics is admissible within our framework (see Section 5.1). We discretise this process with the statistical finite element method (STATFEM) (Girolami
discretised stochastic PDE, however
2021 Duffin et al. [021 2022 Akyildiz et al. 2022). forming the basis of the physics-informed prior on the latent space. Stochastic additive forcing inside the PDE represents additive model error, which results from possibly misspecified physics. Full details, including the ODE case, are given in Appendix
In general, we assume that the model has possibly unknown coefficients A; on these we place the Bayesian prior A p(A) (Stuart 2010), describing our a priori knowledge on the model parameters before observing any data. We also assume that un, the initial condition, known up to measurement noise, with prior p(uo) ) set accordingly. For pedagogical purposes, derive the discrete-time dynamic model using the Korteweg-de Vries (KDV) equation as running example, which used in later sections an example PDE:
O.u + audLuu + B É gP((0 t') k(ss
(2)
L) [0, and = fa, 3}. Informally { a GP, with delta correlations in time, and spatial correlations given by the covariance kernel k(:,)): R R - (Williams & Rasmussen] 2006). This an uncertain term the PDE, representing possible model misspecification. Note that assume all GP hyperparameters are known in this work. The KDV equation is used to model nonlinear internal waves (see, e.g., Drazin & Johnson 1989), and describes the balance between nonlinear advection and dispersion. Note that although the KDV equation defines a scalar field, u(s, t), the approach similarly holds for vector fields.
where u t) E R == {(s
Following STATFEM, the equations are first spatially discretised with the finite element method (FEM), then discretised in time. Thus, equation 2 is multiplied with a smooth test function u(s) V, where V is appropriate function space, and integrated over the domain Ω to give the weak form (Brnnner & Scott 2008
Thomée 2006)
(8,u, v) + a(ud,u, v) + B(0 -E,
where (..-) the (2( inner product.
The domain is discretised to give the mesh Ω4, with vertices [5 In this case, we take the sj to be a uniformly spaced set of points, so that = -ih (h gives the spacing between trid-points). On the mesh set of polynomial basis functions (d)(s) is defined, such that approximation the
4

---

Under review as a conference paper at ICLR 2023
PDE. Letting (s, -E= u(())o((s) the weak form is now rewritten with these basis functions
(i.un, Φ5) + a(uθ, un, Φ3) + u(0 u 03) = (É. Q5); j = y.... Tu-
This gives a finite-dimensional SDE over the FEM coefficients u(() = (ui (!)...., un. (1))
du
M
+ AAu aF = E. E(t) (O 6(t - -t) G),
dt
where Mij ( Φg), , (0) 63). F(u); - (un Un Q3) and Gúj ( (k((-.)) ,z)). Time discretisation eventually gives the transition density pl p(u, 11, A), for un = u(n whose form dependent on the discretisation used (see Appendix C ffor all details).
3.2 LIKELIHOOD
The second component of the generative model is the likelihood (x./uu)). This density acts as data likelihood for pseudo-data {x, --11 This middle layer in the model is usually necessary. as high-dimensional observations (y, n= may only be generated by some observed dimensions of {u,) For example, perhaps it is known a priori that only a single component a latent coupled differential equation generates the observations (see also Section 5.T). This explicit likelihood is introduced to separate the encoding process from the state-space inference; in practice we compute the pseudo-data via the encoding 4s (x1NN (....NNy..N] ), then condition with standard nonlinear filtering algorithms (Fraccaro et al. 2017).
The latent states U, mapped discrete times pseudo -oservvobserstiions via x,, = Hu,, + En, with En ~ (O, R). We parameterise this observation density as pv (X,, un) where y/ = (H, Both the pseudo-observationn operator H *nx and the noise covariance R R assumed be known in this work. An additional noise process is assumed, r,, to represent extraneous uncertainty associated with the secudoobseervation. Observations Yn are related to pseudo-observation. X,, the decoder, represented with the conditional density pe (y Xx.). The combination of the transition and observation densities provides the nonlinear Gaussian SSM:
~ Transition: U, (M 11) + On-1, 6n-1 1 (0, (0,Q). Q):
Pseudo-observation: X, Hu,, + rn, Fa ~ (O R).
Inference is performed with the extended Kalman filter (ExKF) (Jazwinski 1970
Law et al. 2013).
which computes p( u, Xim, A). Further details are contained in Section 4
3.3
DECODER
The last component of our generative model is the decoder pp(y,,ze ///x), which describes the unknown mapping between the aseudo-observatiin data should model as closely possible the true data generation process. Prior knowledge about this process can be used to select an appropriate po (yy//x)). No temporal structure is assumed on 0, so the decoder is shared across all times pe II ()(x)(x,) For more details specific architectures see Appendix
Xa, and the observed data yn. The decoding of latents to
VARIATIONAL INFERENCE
In this section, introduce the variational family and the ELBO. Denote by q(uIN ((UI,N,XINN X1:N Alyn.)
the variational posterior, which, similar to Fraccaro et al. (2017). factorises as
q(uI,N, X1:N, A Aly:.) = q (u.NN XI:N, A (x (Tx)))
Note here that we do not make a variational approximation g (1..,, X1:N. Aly1:N): ); this is taken to
be the exact posterior p(u X1:N. A). We derive the ELBO to be (see also Appendix B
p MUI:N, X1:N. , A. VI-N)
log((y):N) 2
log
quEIN XI:N, A Aly1:N )dx
(uII,, X1:N. A
log pe pe(N1:N XI.N)
p(A)
Eng
Ao (Xi:Nly::)) + E
logp(x AA) + log ya (A)
Typically this expectation is not analytically tractable and Monte Carlo is used to compute an approximation.
5

---

Under review as a conference paper at ICLR 2023
Nonlinear Filtering. In the ELBO of equation ] estimating log p(x (A) requires marginalising over U1:N. the latent physics process. We per- form this via the EXKF (Jazwinski, 1970; Law et al.] 2015). which recursively computes a Gaussian approximation to the filtering poste- rior p(u,, X1:n: A) R. N(m,, C.,). We note however that this can also be realised with other nonlinear filters, c.g., ensemble Kalman fil- ters (Chen et al., 2022) or particle filters (Corenflos al., (02T). The factorisation of the pseudo-observation marginal likelihood, P(X1:N (A) p(x||A I [X A), enables computation, the filter can com- pute log pgg((x,, X. 11:=-1, A)), at each prediction step, via p(x, X1:n-1, A)
Latent Dynamics: MEN
Traeedir
Sonditon
Pseudo- Observations: XL.V
p(x,,) u, A) ()ppp,u,,,, X1mm-1 A) du,.
Variational approximations. As with the decoder, encoder parameters é are shared between variational distributions (go(xxlyw))>z to give an amortized approach (Kingma et al.] 2019). Unless otherwise specified, for each n the encoding has the form go (x..ly)) Functions Ho (yn): 12y + - and (y)): 1 + R"= are NNs, with parameters @ to be learnt. Specific encoding architectures are given in
(yn)) (V.)).
Velocity Field: yv
Appendix A As for 4T, we set it to a Gaussian with mean Yr and variance diag(ox).
EXPERIMENTS
We present three examples with different dynamical systems. To demon- strate the generality of the method, the first uses the stochastic Lorenz-63 system (Lorenz, 1963), a highly nonlinear stochastic ODE. In this case, high-dimensional observations are of velocity field being modulated by the chaotic system. For the final two examples, we use video data. We consider the advection and KDV PDEs, and, in these examples, we indirectly observe high-dimensional representations in the form of video data. This mimics the experimental setup of the various DVAE papers (c.g.. Fraccaro et al., 2017
Figure 2: Lorenz-63: latent states U1:N, pscudo-observations X1:N. and velocity fieldyy. yN-
Pearce et al., 2018; Jazbec et al. 202T Fortuin et al.) 2020 Zhu et al., [0222 Girin et al. (021). For these two examples, video datasets are generated in similar fashion.
both cases, simulated data emulate the scenario where noisy video of an internal wave profile is captured. Internal waves arise as waves of depression or elevation flowing within a density-stratified fluid at regions of maximum density gradient (Gerkema & Zimmerman, 2008). Our experiment setup thus emulates idealised setup where a -ook--nn-whitee. side-on, video of a laboratory experiment has been obtained, and is inspired by scenarios where the high-resolution use of classical measurement devices is not feasible, yet the use of commonplace vidco-capturing devices is (see, c.g., HHrr et al. 2001 2002).
The advection equation example is motivated by internal wave propagating undisturbed through some medium. For this linear case, comparisons with the KVAE reveal that after training, the Φ-DVAE outperforms both in terms of the estimated ELBO and in terms of the -uenn-squard-errrr (MSE). The KDV example is more complex case, and extends into the nonlinear PDE setting, while also being classical model for internal waves (Drazin & Johnson| 1989). For this example, comparisons are made with VRNNs, the GPVAE, and the standard VAE. We demonstrate that the MSE of the &VDVAE comparable or better than these approaches. Furthermore, for joint state and parameter inference, we verify the methodology and demonstrate contraction of the posterior about the truth.
5.1 LORENZ-63 EXAMPLE
In our first example, the latent dynamical model p(u,/um, A) is given by an Maruyama discretisation (Kloeden & Platen 1992) of the stochastic Lorenz-63 system,
duj (-ou + oug)dt + dw1,
duz = (-U14y + rui - U2 dt + dwz.
(4)
dug (u1 U2 bugg)d dwss

---

Under review as a conference paper at ICLR 2023
0.5
0.1 5.0
1801
40-
(
Epocts
(a) Parameter estimation
(b) Inference
Figure 3: Lorenz-63 system: (a) the results of parameter estimation for (top) estimation, (center) b estimation, and (bottom) joint estimation of A = {o, r. b). In (b) we show an example result of state estimation. Left column is the true Lorenz states ExKF means, center and right columns show the distribution of the estimate of the final states.
where utt) := [wi (2))), (a(()) 10, 6). u, = [ur u2(nA,). un (nA, fo, and W1, W2. and wy are independent Brownian motion processes (Oksendal 2003 Särkkä & Solin 2019). For full details we refer to Appendices [A [ The Lorenz-63 system is classical system widely used to benchmark filtering and data assimilation methods (see, e.g., Akyildiz & Miguez 2020). It was popularised in Lorenz (1963) through its characterisation of "deterministic nonperiodic flow*. and is common example of chaotic dynamics.
We observe synthetic 2D velocity fields, Y1:N. of convective fluid flow, and we use our method to embed these synthetic data into the stochastic Lorenz-63 system. The Lorenz-63 system is related to the velocity fields through a truncated spectral expansion (see, e.g., [Wouters] (013)) In brief, is assumed that the velocity fields have vertical velocity, the 3D velocity field is realised in 2D. The velocity field can be described by the stream function v := ( $2, s2,t), t), where $1 and s2 are the spatial coordinates of variation, respectively, and thus ytt (-O,, v. 0, +). A truncated spectral approximation and a transform to non-dimensional equations yields 16(81, 82, 1) x "n (t) (in(Ts1/ /I) sin(T22/ d),, where u1 (t) is governed by the Lorenz-63 ODE (i.e., equation 4 with w; 0).
To generate the synthetic data y I.N, we generate a trajectory true from the deterministic version 4 at discrete timepoints nA,, and use the generated uirue to compute the two-dimensional velocity
field, Yn, via (81,, 82, t). This corresponds to having middle layer = 11.n + wn where ~ NO, R2), with likelihood p(r, u,) = -Nhh" un R²) where h 0, The synthetic data of u truue() (1), XI.N, and yI.N are visualised Figure 2 the full trajectory u trr)(()) shown in 3D, and the velocity field yn is shown in 2D, for single The decoding is assumed to be of the form Pw (w(yn//,)) = = ((w,,, I2I,, with unknown coefficients w E RR"v, The variational encoding qw (In yn) is determined via pseudo-inverse, as detailed in Appendix A along with the relevant hyperparameters and numerical details.
Figure 3a displays parameter estimation results. For individual and b estimation (Figure 3a top and center), we initialise the variational posterior randomly, and visualise each across training epochs. Both results show the posteriors contracting about the true value, with b demonstrating more rapid convergence, visually. For joint estimation (Figure 3a bottom), similar behaviour is observed, with the parameter not identified by the final epoch. We conjecture this is due to identifiability with other parameters when estimating jointly. Note, however, that the true values are all contained within the confidence bands of the final variational posteriors.
We also investigate the posterior inference achieved by &-DVAE. We visualise the filtering posterior (s) through time, conditioned sample from the trained encoder x x go(: (V:w)) with fixed,
7

---

Under review as a conference paper at ICLR 2023
known A. Figure 3b (left) shows clear agreement between the filter mean and the latent states truee Marginalising over the encoding (Figure 3b (centre)) targets the filtering posterior directly L:N.
conditioned on observed data YI:N, and demonstrates unbiased mean estimates of the true state. This is particularly clear for the first latent dimension, where the posterior conditioned on an individual sample 'iNy often has poor posterior coverage of the true value (cf. Figure 3b (right)).
(3)
5.2 ADVECTION PDE
As second example, we consider the advection equation with periodic bound- ary conditions. In this example, we derive the transition density p(u,, / 11 A)
from a STATFEM discretisation of stochastic advection equation:
ELOC: lo atouttaaon
cu,u
GP(O, 5(1 (1) k((,))))
(5)
where = u(s,t), t), & == E(s.t), t). E 0. [0, 10,40], and t(s,t) u(s + 1,t). Recall that, as in Section [.2] the FEM coefficients u,
(un (nA.),, ...,.... Mn, (nAr)) are the latent variables. These are related to the discretised solution via un (8, nA,) = Ei= C.d[
i= (((n(,)ox(s)) (see Appendix
MSE burrhii
Video data VLN is generated from the deterministic version of equation (i.e. equation with { 0). A trajectory trre simulated and the correspond- ing FEM solutions u ndr)) are imposed onto a 2D grid. On the grid, pixels below u/ u (s, nA,) are lit-up in a binary fashion, with salt-and- peper noise (Gonzalez & Woods, 2007). In this experiment we use fixed parameters, setting A = c 0.5. We set the decoder to po(yn| xx,) Bern (Ho(x.)) and the encoder (x,, y.) = (u))()))) pendix for full details.
Easch
Figure 4:
Ad-
vection: (lop) and recon- struction (bottom). against ground truth, for (D-DVAE) against KVAE.
ELBO
As previous, see Ap-
MSE
Due to linearity of the underlying dynamical system, we compare the 4-DVAE to the KVAE for a set of video data generated from the advection equation, for various dimensions of the VVAE latent space. Specifying particular form of latent dynamics on the latent states increases the inductive bias imposed on the latent space, and should provide faster learning and more likely representations - than with learnt dynamics.
To explore this, compare our method to KVAE for the linear advection example in terms of the ELBO and normalised MSE, over training epochs (all methods use Adam (Kingma & Baj (017)). These are plotted in Figure 4 For the MSE, VVA quickly learns to reconstruct the images, with Φ- DVAE taking longer to reconstruct with similar accuracy but eventually producing better reconstructions (final MSEs 0.0221 vs. 0.0533 for the &-DVAE, KVAE-64, respectively). The ELBO for &-DVAR is rapidly optimised in comparison to the VAE models, and is greater by the end of training. The trained @-DVAE gives better evidence for the data (greater ELBO), whilst providing more accurate reconstructions (lower MSE).
Normulaed Recostrazion MEE
0173
015
5.3 KORTEWEG-DE VRIES PDE
412215
sar.
Our final example uses the KDV equation as the underlying dynamical system. As previously, the latent transition density p(u,/um-1, -. A) defines the evolution of the FEM coefficients, as given by a STATFEM discretisation of stochastic KDV equation:
Figure 5: KDV:
normalised recon- struction MSE after 200 epochs, for 30 where u := u(s,t), 5 = ((s,t), & E [0,2], t E [o, 1]. and u(s,t) = independent simula- u(s + 2,t). Parameters are A (a,
0,u + and,uu + 383u E. E~
8(1 ((s(-()k((ss))))
k(s,,(()))
tions.
true Data is simulated in the same fashion as in the advection equation: we simulate a trajectory u INN using FEM discretisation of the deterministic KDV equation and we impose FEM solutions uf
true (8, nA))
8

---

Under review as a conference paper at ICLR 2023
Y+
Mi
2.5
Ma
+2r,
2.0
true a
1.5
1.0
0.5
Epoch
(a) Parameter estimation.
(b) Image data.
(c) Latent inference.
(d) Reconstructions.
Figure 6: KDV joint inference results; frames shown for (10,50, 75).
on a 2D grid. We light up pixels below the solution, and add salt-and-pepper noise. For the data-
generating process, we use the parameters A {a L. 3 0.0222). and the initial condition of u (s, 0) = (TS) as in the classical work of Zabusky & Kruskal (965).. This regime is characterised by the steepening of the initial condition and the generation soliton;; nonlinear waves which have particle-lik lite interactions (Drazin & Johnson [189). As for the advection example, we set the decoder as pa (y,, xx)) = Bern (Po( and the encoder as qo (x, (Noo(yn), ox(n)))
again, see Appendix for details.
We test the VRNN, GPVAE and the standard VAE approaches alongside Φ-DVAE with equal encoding dimension n2 = 40. In Figure 5 we report the normalised MSE after fixed number of epochs, using Adam (Kingma & Ba 2017) with the same learning rate for each method. @-DVAE outperforms both GPVAE and the standard VAE in terms of median MSE. Note the variation, in MSE, of the standard VAE is also greater than the other models, suggesting that the dynamical structure provides more consistent learning. Both &-DVAE and VRNN perform well, producing visually similar reconstructions, with the VRNN having lower median MSE (0.0239 vs 0.0105, respectively). A predictive MSE is also computed by giving the model single image frame to encode, sampling from the generative model forward in time, and comparing the decoded samples against the ground truth images. Here, 4-DVAE outperforms VRNN with median predictive MSEs 0.0415 vs 0.0806 respectively.
We report joint estimation results for the partially known KDV PDE, where we fix 3 0.022 and estimate a. The prior over a. p(a) (1.5,, 0.3²), is semi- -nnoimiformetve....m Joint inference of a and latent states u, is shown Figure 6 The Gaussian variational posterior, (a) N (Mx, af) (initialised at the prior), contracts about the true value 1.0 (see Figure 6a). The filtering posterior, p( (u,,yyno 1m)), shown in Figure 6c| This is obtained via Monte Carlo approximation, marginalising the variational posteriors 4d. Qx to account for uncertainty in the encoding and parameter estimates. Including structured prior on the latent space has forced the encoding to be representative of observations taken from the KDV system, clearly capturing the latent dynamics causing the variation the image data. Figures and display the image data and reconstructions respectively. showing the ability of Φ-DVAE to both accurately reconstruct and de-noise the data.
CONCLUSION
In this paper developed @-DVAE, methodology that allows for the incorporation of unstructured data into physical models, in settings where the model-data mapping may be unknown. The proposed approach utilizes variational autoencoders and nonlinear filtering algorithms for PDEs, learn physically interpretable latent spaces where analysis and prediction can be performed straightfor- wardly. Our framework connects traditional nonlinear filtering techniques and VAEs, opening up the possibility of further combinations of these methods. Future work will focus on more challenging PDE systems, as well as more complex, and higher-dimensional,, observational data.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We have included full numerical details to reproduce all results in our paper in Appendix A The code we have developed for this paper will be made publicly available after the decision, enabling the generation of all datasets and figures.
REFERENCES
Ömer Deniz Akyildiz and Joaquín Miguez. Nudging the particle filter. Statistics and Computing, 30 (2):305- -300, 2020.
Ömer Deniz Akyildiz, Connor Duffin, Sotirios Sabanis, and Mark Girolami. Statistical Finite Elements via Langevin Dynamics. SIAM/ASA Journal of Uncertainty Quantificatio,, 2022.
Brian D.O. Anderson and John B. Moore. Optimal filterinn.. Englewood Cliffs, N.J. Prentice Hall, 1979.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic Variational Video Prediction. In International Conference on Leurning Representations, March 2022.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610, 2014.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518)859--77,, April 2017. ISSN 1459. doi: 2111:00066214552211.112888
Mare Bocquet and Pavel Sakov. Joint state and parameter estimation with an iterative ensemble kalman smoother. Nonlinear Processes in Geophysics, 20(5):803-81) 2013.
G. E. P. Box. Robustness in the Strategy of Scientific Model Building. ROBERT L. Launer and GRAHAM N. Wilkinson (eds.), Robustness in Statisties, pp. 201-236. Academic Press, January 1979. ISBN 978-0-12-4381 10066. doi: 0.1016/8978-0012-4933 12-55455-000-000000118
Susanne C. Brenner and L. Ridgway Scott. The Mathematical Theory of Finite Element Methods, volume 15 of Texts in Applied Mathematics. Springer New York, New York, NY, 2008. ISBN 978-0-387-75933-- 978-0-387-7593442
doi: -0.10077978-03387--799----
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-33) -9337, 2016.
Kathleen Champion, Bethany Lusch, Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45): 22445 -2451, 2019.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Yuming Chen, Daniel Sanz-Alonso, and Rebecca Willett. Autodifferentiable ensemble kalman filters. SIAM Journal on Mathematics of Data Science, 4(2):801-833, 2022.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
Adrien Corenflos, James Thornoo,, George Deligiannidis, and Amaud Doucet. Differentiable particle filtering via entropy-regularizee optimal transport. In International Conference on Machine Learning, pp. 2100-2111 2.. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes. Incorporating prior scientific knowledge. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124099 December 2019. ISSN 1742-5468. doi: 11188//142-546//ab55
Amaud Debussche and Jacques Printems. Numerical simulation of the stochastic Korteweg de Vries equation. Physica D: Nonlinear Phenomena, 134(2)200-2266 October 1999. ISSN 0167-2789.
doi: 111101660016--27899990000000
Susanne Ditlevsen and Adeline Samson. Estimation in the partially observed stochastic morris-lecar neuronal model with particle filter and stochastic approximation methods. The annals of applied statistics, 8(2):674-72,, 2014.
Denis Dochain. State and parameter estimation in chemical and biochemical processes: a tutorial. Journal process control. 13(8):801-818, 2003.
Arnaud Doucet, Simon Godsill, and Christophe Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing, 10(3):19--008, July 2000. ISSN 1573-1375. doi: 10. 100100//:11009954411000
P. G. Drazin and R. S. Johnson. Solitons: An Introduction. Cambridge Texts in Applied Mathematics. Cambridge University Press, Cambridge, second edition, 1989. ISBN 978-0-521-33655-1 doi: 10.1017/CB097811311220 139172099.
Connor Duffin, Edward Cripps, Thomas Stemler, and Mark Girolami. Statistical finite elements for misspecified models. Proceedings of the National Academy of Sciences, 1 118(2), 2021.
Connor Duffin, Edward Cripps, Thomas Stemler, and Mark Girolami. Low-rank statistical finite elements for scalable model-data synthesis. Journal of Computational Physics, 463, August 2022.
ISSN 0021-9991. doi: 10.1016/j..cp.22 jep.0....pp.222..111111 20...1002..1111.. 112611
N. Benjamin Erichson, Michael Muchlebach, and Michael W. Mahoney. Physics-informed Autoen- coders for Lyapunov-stable Fluid Flow Prediction, May 2019.
Lawrence Evans. Partial Differential Equations, volume 19 of Graduate Studies in Mathematics. American Mathematical Society, Providence, Rhode Island, second edition, March 2010. ISBN 178-0-8218-49742 978-1-4404-1144 11447044-144- 114-2.. doi: 10. 00900//00//
Geir Evensen. The Ensemble Kalman Filter: Theoretical formulation and practical implemen- tation. Ocean Dynamics, 53(4):343-36) November 2003. ISSN 1616-7228. doi: 10. 10.1007/
$10236-003-0036-9,
Vincent Fortuin, Dmitry Baranchuk. Gunnar Rätsch. and Stephan Mandt. Gp-vae: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pp. 1651-1661. PMLR, 2020.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jean-Yver Franceschi, Edouard Delasalles, Mickael Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic Latent Residual Video Prediction. In Proceedings of the 37th International Conference on Machine Learning, pp. 3233-3246. PMLR, November 2020.
T Gerkema and J T F Zimmerman. An introduction to internal waves. Lecture Notes, Royal NIOZ, Texel, 207:207, 2008
Craig Gin, Bethany Lusch, Steven L. Brunton, and J. Nathan Kutz. Deep learning models for global coordinate transformations that linearise PDEs. European Journal of Applied Mathematics. 32(3): 515-539, June 2021. ISSN 0956-7925, 1469-4425. doi: 10. 10.1017780956995520000377
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical variational autoencoders: A comprehensive review. Foundations and Trends in Machine Learning, 15(1-2):1-175, 2021.
11

---

Under review as a conference paper at ICLR 2023
Mark Girolami. Eky Febrianto, Ge Yin, and Fehmi Cirak. The statistical finite element method (statFEM) for coherent synthesis of observation data and model predictions. Computer Methods in Applied Mechanics and Engineering, 375:113533, March 2021. ISSN 0045-7825. doi: 10. 10.1016/j. cma.2020. 13533.
Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing. Pearson, Upper Saddle River, N.J, 3rd edition edition, August 2007. ISBN 978-0-13-168728-8
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In Proceedings of the 36th International Conference on Machine Learning. pp. 2555 2565, PMLR, May 2019.
Jouni Hartikainen and Simo Sarkka. Kalman filtering and smoothing solutions temporal Gaussian for
process regression models. In 2010 JEEE International Workshop on Machine Learning for Signal Processing, pp. 379-384, Kittila, Finland, August 2010. IEEE. ISBN 978-1-4244-7875-0 doi: 10.1 00.10000LSSP2200.55891.
Carlos X. Hernández, Hannah K. Wayment-Steclee Mohammad M. Sultan, Brooke E. Husic, and Vijay S. Pande. Variational encoding of complex dynamics. Physical Review E, 9666)22 June 2018. ISSN 2470-0045, 2470-0053.
doi: 10. 1003/PhysReE..97.062444
D. A. Horn, J. Imberger, and G. N. Ivey. The degeneration of large-scale interfacial gravity waves in lakes. Journal of Fluid Mechanics, 434:1 81 81-207, May 2001. ISSN 0022-1120 1120, 1469-7645. doi: 10.1017/500221 120000031200
D. A. Horn, J. Imberger, G. N. Ivey, and L. G. Redekopp. A weakly nonlinear model of long internal waves closed basins. Journal of Fluid Mechanics, 467:269-281 September 2002. ISSN 1469-7645, 0022-1 1120. doi: 10. 10.1017/000221 12000001362.
Metod Jazbec, Matt Ashman, Vincent Fortuin, Michael Pearce, Stephan Mandt, and Gunnar Rätsch. Scalable Gaussian Process Variational Autoencoders. In Proceedings of The 24th International Conference on Artificial Intelligence and Statisties, pp. 3511-3519. PMLR, March 2021.
Andrew H Jazwinski. Stochastic Processes and Filtering Theory. Academic Press, 1970.
K Judd and L Smith. Indistinguishable states II. The imperfect model scenario. Physica D: Nonlinear Phenomena, 196(3-4):224-25 September 2004. ISSN 01672789, doi: 10. 10.1016/5067--289000 00182-4.
R. E. Kalman. New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35 45, March 1960. ISSN 0021-9223. doi: 10.1 11661666..55...
Nikolas Kantas, Arnaud Doucet, Sumeetpal S Singh, Jan Maciejowski, and Nicolas Chopin. On particle methods for parameter estimation in state-space models. Statistical science, 30(3):328-351, 2015.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In International Conference on Learning Representations, 2017. URL httttt:://ppeeeeiiiiie htt.ooruettfootttttoo
id-HyTqHLLxq
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes, May 2014.
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends Machine Learning, 3074)330733 2019.
Peter Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Applica- tions of Mathematics. Springer Berlin Heidelberg, 1992.
Costas Kravaris, Juergen Hahn, and Yunfei Chu. Advances and selected recent developments state and parameter estimation. Computers & chemical engineering, 51:1 11:111:: 11-123, 2013.
12

---

Under review as a conference paper at ICLR 2023
Rahul G Krishnan, Uri Shalit, and David Sontag. arX1v:1511.51121.000
Deep kalman filters.
arXiv preprint
Kody Law, Andrew Stuart, and Konstantinos Zygalakis. Data Assimilation: A Mathematical Introduction, volume 62. Springer, Cham, Switerland, 2015, ISBN 978-3-319-20325-6.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-Net: Learning PDEs from Data. In Proceedings of the 35th International Conference on Machine Learning, pp. 3208-3216. PMLR, July 2018.
Ryan Lopez and Paul J. Atzherger. Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems, March 2021.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130-1444 1963.
Peter Y. Lu, Samuel Kim, and Marin Soljačić. Extracting Interpretable Physical Parameters from Spa- tiotemporal Systems Using Unsupervised Learning. Physical Review X, 10(3):031056, September 2020. doi: 10.1 10.1103/hyyReev...00 10.0331000
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9()):1-10, 2018.
Hamid Moradkhani, Soroosh Sorooshian, Hoshin V Gupta, and Paul R Houser. Dual state parameter estimation of hydrological models using ensemble kalman filter. Advances in water resources, 28 (2):135-147, 2005.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep Dynamical Modeling and Control of Unsteady Fluid Flows. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Bernt Øksendal. Stochastic differential equations. In Stochastic Differential Equations, pp. 65 84. Springer, 2003.
Samuel E. Otto and Clarence W. Rowley. Linearly Recurrent Autoencoder Networks for Learning Dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558-59) January 2019. doi:
10.1137/18M1177888
Michael Pearce. The gaussian process prior for interpretable latent dynamics from pixels. In Symposium on advances in approximate bayesian inference, pp. 1-12, 2020. tex.organization: PMLR.
Michael Pearce, Silvia Chiappa, and Ulrich Paquet. Comparing interpretable inference models for videos of physical motion. In 1st symposium advances in approximate bayesian inference, 2018.
Sebastian Reich and Colin Cotter. Probabilistic forecasting and Bayesian data assimilation, Cam- bridge University Press, 2015.
Simo Särkkä and Arno Solin. Applied Stochastic Differential Equations, volume 10. Cambridge University Press, 2019.
W. E. Schiesser. The Numerical Method of Lines: Integration of Partial Differential Equations. Academic Press, 1991. ISBN 978-0- 32-62-13--2413
Geir Storvik. Particle filters for state-space models with the presence of unknown static parameters. IEEE Transactions on signal Processing, 50(2):281-289, 2002.
A. M. Stuart. Inverse problems: Bayesian perspective.. Acta Numerica, 19:451-559, May 2010. ISSN 0962-4929, 1474-0508. doi: 10. 10.1017//006249991000000
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
13

---

Under review as a conference paper at ICLR 2023
Albert Tarantola. Inverse Problem Theory and Methods for Model Parameter Estimation. Society for Industrial and Applied Mathematics, January 2005. ISBN 778-0-99877--72-9 978---98777------99 doi: 10.1 19.11377197788898777911
Vidar Thomée. Galerkin Finite Element Methods for Parabolic Problems. Number 25 in Springer Series in Computational Mathematics. Springer, Berlin New York, 2nd ed edition, 2006. ISBN
978-3-540-33121-6.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Christopher KI Williams and Carl Edward Rasmussen, Gaussian Processes for Machine Learning. volume 2. MIT press Cambridge, MA, 2006.
Jeroen Wouters. brief introduction to the Lorenz'66 63 system. 2013.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2318-2328, Nashville, TN, USA, June 2021. IEEE. ISBN 978- 1-66544-500-2. 2. doi: 10. 1LL109//VVPR444772222000022
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics fore- casting*. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): (121(12):124013) 14012, December 2021. ISSN 1742-5468. doi: 1..1088/14225548/ac3320
N. J. Zabusky and M. D. Kruskal. Interaction of "Solitons" in Collisionless Plasma and the Recurrence of Initial States. Physical Review Letters, 15(6):240-2 24324,,,, August 1965. doi: 10. 10.11 PhysRevI.ett. 15.240.
Harrison Zhu, Carles Balsells Rodas, and Yingzhen Li. Markovian Gaussian Process Variational Autoencoders, July 2022.
14

---

Under review as a conference paper at ICLR 2023
A
NUMERICAL DETAILS AND NETWORK ARCHITECTURES
Lorenz-63 experiment.
Time-series length: N 150.
Input: Fy = 200.
Pseudo-obserrattins:: 72
Latent n. 3.
Decoder: Pw (yn(2n)) -N(Wn,, 17 11, 0.005.
Encoder: Sw (fnnyn)) -N((w' - y,, 17 w) 11
Latent initial condition: Ug = [-3.7777 -3.839. 21 15015
Latent noise processes: L = diag(0.22)) R = d0.4..
Latent discretisation: Euler-Maruyama, dt = 0.001.
Joint parameter prior: p((o, r, b)) = (30, 20, 5] diag( 102,,,,0,3300 )))
Optimiser: Adam, learning rate 10
To generate the data, simulate the Lorenz SDE with 0.001 and take pseudo-observations In every 40 time-steps of the latent system, for total of N 150 with A, 0.04. Velocity measurements are taken in the 81 and 82 directions over regular 10 x 10 grid on the domain 82,(). l). These measurements are flattened to the data $1,82 E [-4,4] 4]. via the streamfunction 82, vector yn E Rky, My 200. Parameters for data generation are = {a = 10, r = 28, 8/3).
Advection equation experiment.
Time-series length: N = 200.
Input: By = 784.
Pseudo-observationn: ng = 64.
Latent n, = 64.
Decoder: po (yn (x,)) = (xxo())(
(+):): MLP. two fully connected hidden layers with dimension 128
Encoder: qo (x,, ,.) oo
p+(+),, a-)): MLP, fully connected hidden layers with dimension 128
Neural Network activations: LeakyReLU, negative slope 0.01
Latent initial condition: u(s, 0) exp(- - 2.5) /0.1)
Latent noise processes: 0.02, ( - 0.1, R diag(
Latent discretisation: FEM ( 1]) polynomial trial/test functions, Crank-Nicolson time
discretisation, dt 0.02.
Optimiser: Adam, learning rate 0.001.
To generate the data, simulate the advection equation with dt 0.02, observing every 10 time- steps for A, 0.2 and N = 200. Latent dimensions are Th2 = 64 and f2. 64, with each image 28 28 pixels. These images then flattened vectors yn [0, 1 with ny 784.
KDV equation experiment.
Time-serics length: N 100.
Input: ny 1792.
Pseudo-observations:: nh 40.
Latent n. 600.
Decoder: po (y,, (x.) (ern()ooxx..)
15

---

Under review as a conference paper at ICLR 2023
po(+): MLP, two fully connected hidden layers with dimension 128
Encoder: (x (x.(yn) = N
(s()),a()) MLP, two fully connected hidden layers with dimension 128
Neural Network activations: LeakyReLU, negative slope 0.01
Latent initial condition: w(s, 0) cos( (TS).
Latent noise processes: p 0.01, f 0.2, R diag(0.05').
Latent discretisation:
approach of Debussche & (rintems(1999)) (²( polynomial trial functions, Crank-Nicolson time discretisation, dt 0.01.
Parameter prior: p(a) (1.5,, 0.3²).
Optimiser: Adam, learning rate 0.005.
To generate the data we simulate the KnV equation with 0.01, observing every timestep for A, 0.01, and we take N 100 observations Yn, with y. [0, 1 Each yn is flattened image dimension ny = 64 x 28 = 1792, and we encode to pscudo-observations X,, of dimension Th 40. The latent state dimension is na 600. Predictive MSE is calculated by encoding Ye: = 0.1 to Xx, 0.1, sampling x from the generative model forward in time for 10 time-steps (up to 0.2), and computing the normalised MSE of the decoded samples y: compared to the ground truth.
Linear decoding/encoding. If linear data generation is assumed from X1.N YI:N, then this structure can inform decoding. With a linear decoder of the form:
(y(y x) = (Ax,, v/ T).
In this case, we use the "inverted" linear decoder given by:
A (xly) (VA( -N y. By selecting the encoder appropriately, the space of parameterised variational distributions can be restricted to align with our beliefs about the data generation process.
(ATAA)
B FULL VARIATIONAL FRAMEWORK
We derive the approximate ELBO for joint estimation of dynamic parameters A, and autoencoder parameters o, 0. We start by writing the evidence
p(y = /
XI,N, A. A
We maximize plyplyy))) as
log pgy((y)))) = log
p( p(UENN XI,N, A, (I:N)dxI.Ndee Ix
p( NIN AENN)
log
XINN, Aww.
((IINN XINN AlyuN)
(UI:N X1:N. V1:N)
log
( (UI:N, X1:N; Aly
ELBO,
where the third line follows from the application of Jensen's inequality. Our generative model determines the factorisation of the joint distribution, given in equation [
p(uENN XINN A, VI.N) = p(y..N X ui,N A ( unnN A))p(A
( X1.N )p X1:Nv U1.,, A)p(uI-N/ (A)p(A).
Next, plug this factorised distribution into the ELBO and obtain
Po(y bx::) (xw.::)p(xxx////l / A)p u A)p(A)
ELBO log
quIIN XINN Aly I.N) qu XI.N Aly1:)) dx dAA.
16

---

Under review as a conference paper at ICLR 2023
The family of distributions which use to approximate the posterior is described below. We assume
a factorisation based on the model into variational encoding 4o (o(.). ).. a full latent state posterior qu (-),
and the variational approximation to the parameter posterior (x))):
QuE:N, X1:N, Aly1.n) = (u XI:N, A ))xx (x1.
(6)
N
= (u XI,N A) (A)
II 4s (xnnxyyn))
==
The second line demonstrates the amortized structure of the autoencoder, where the same encoding parameters are shared across datapoints. We can then substitute this expression into our ELBO and obtain
/ log
p XI.N )p X |ui,,, (pp(unN(A))p)
ELBO =
qu (UIN XINN, A) ())x(A)))))))) (a))))))s( (x (X::N\V.:))
x q(uLNN X x
Assuming the variational posterior is the exact filtering posterior, i.e., qu (uENN/X = p(u1:N/ (I:N, then applying Bayes rule
p[x |n:N ()p( A) = p(x1.N|A)
gu (u X1:N, A)
leads a simplification of ELBO in terms of the marginal likelihood p(x1:N|A) of the state-space model. Substituting this expression leads to
/
pe ( ())
ELBO
log
(x VI.N )ax (A)
x qe (u1.N/ XI:N, A A)ga (A)qa(X1.)) wd
/ log
pa (y XIEN)
4oh (xI:N1 S11:N/VV.N VI.N ddx...
9s (XI:N VI:N)
p(A)
logp(xLNNI (A) + log
4A (A)
pa (VINN
logp(xENN A) log p(A)
Egg log as X1:N YI:N)
+ Egg
Enyx
(A(
Using single MC sample from do (XI:N to approximate the expectation, we can write:
T(0.6, log (y( (1:N)
Ey, [log [logp(xi:N/AN] A)] KC(()(A),P(A)).
Ts (x YI.N)
We can sample 4x (A) to approximate the expectation of logp(x1:N|AM Note that this requires the reparameterisation trick that is used for also used when sampling X1.N- This allows for backpropaga- tion of errors through the sampling step. The KL-divergence can be calculated analytically for the case of Gaussian prior and posterior:
M Σ
T(0, 6, A) = log
M [ogg(()nNA((0) -K(()(A),P(A)),,
1
go (x(X (1:N)
and approximated via Monte-Carlo otherwise
T(0, log (x)N) go (x(NNNN
[og (oEp())NAA() (lgg((A()) log 9x (A
+
M
C FURTHER DETAILS ON THE DYNAMIC MODEL
In this work we take the latent dynamical model be a stochastic ODE PDE. For an ODE this follows from standard SDE Särkkä & Solin 2019 Øksendal 2003), given by
du Sx(u, 1; A dt + L(I)dW(!)) (1),
17

---

Under review as a conference paper at ICLR 2023
R" The noise process (()) is a standard vector Brownian motion. The diffusion term L(t) can be used to describe any priori correlation in the error process dimensions, As stated in the main text, this error process is taken to represent possibly mispecified/unknowi physics, which may have been omitted when specifying the model. Discretisation with an explicit Fuler-Maruyama scheme (Kloeden & Platen 1992) gives,
where == u(t) E 1R",,tE [0, fA: R x [0. + R",, L: [0.
Up 1n-1 + (un-i= + - AW,-1 ~ .V(O, A,I,
where U1 u(nd,), (n(::)) (x(,,n))) and This gives transition density
p(u, [12 A) = (un-1 + A A); L LL,1),
defining Markov model the now discretised state vector u,, To align with the notation introduced in the main text, this gives:
P(un Un 1n =
M(un-1) = un-1 + A. ( A, -
Due to the structure of the STATFEM discretisation, the fully- dully-discretised underlying model is of the same mathematical form this ODE case. The difference lies in the dynamics being defined from either PDE or ODE system. common cases, lower dimensional state vector, Un typically results for the ODE case in comparison to the PDE case. For the PDE case, entries of the state vector Un are coefficients of the finite element basis functions.
For the PDE case, the derivation is similar, with additional step pre-time-discretiatton to spatially discretise the system. This yields a method-of-linee approach (Schiesser 1991). As in the main text, consider generic nonlinear PDE system of the form
Ouu TAU + Fx = f & E~ GP(0, 1') (ss,s)),.
where = uls, t). 5 = Es,,t). f f(s). Ω Rd, and t [0. The operators LA and (-) are linear and nonlinear differential operators, respectively. The process & is the Wiener process, whose increments are given by Gaussian process
derivative of a function-valuee with the covariance kernel k(:.)) In our examples, we use the squared--exponentall covariance function WWiliamss & Rasmussen 2006)
|s s 112
k(s,s') = p² exp
2
Hyperparameters {p, f} are always assumed to be known, being set priori. Further work investigat- ing inference of these hyperparameters is of interest.
As stated in the main text we discretise the linear time-evolving PDE following the STATFEM as in Duffin et al. (2021), for which refer to for the full details of this approach. In brief, we
discretise spatially with finite elements (see, e.g.. Brenner & Scott (208): Thomée (006)), for standard references) then temporally via finite differences. We first multiply equation g with sufficiently smooth test function e E V, where V is an appropriate function space (e.g. the H ( Sobolev space (Evans] 2010 and integrate over the domain S2 to give the weak form (Brenner
Scott 2008)
(0.u v) + AA(,,)) + (Pa(u), e) = (f,u) + (E. Ve E Recall that Ax(ss) is the bilinear form generated from the linear operator LA, and
(f.g) J fs)g((s ds,
the (?)) inner product.
Next we introduce discrete approximation to the domain, 2r C Ω having vertices {s, {s,)72n This is parameterised by h which indicates the degree of mesh-refinement. We now introduce a fiite-dimensional set of polynomial basis functions {s such that (s(s) = 8gy In this work these are exclusively the (22 linear polynomial "hat" basis functions. This gives the fnite-dimensional function space Vn, span ( (s) [= which is the space look for solutions
18

---

Under review as a conference paper at ICLR 2023
in. Next, rewrite and u in terms of these basis functions: un (s, t) Σ}= () and (s. 1) = Σ}=1 My (s). As the weak form must hold for all oy Vn, this is equivalent to holding for all Φ.. Thus, the weak form can now be rewritten in terms of this set of basis functions
(0.un, +5) + (un. hy) + (Fx (un), و5) = (5 + (É) J = ,,..., Ru.
Note that, in general, and 24 do not necessarily have to be defined on the same function space, but as we use the linear basis functions in this work we stick with this here.
As stated in the main text, this is an SDE over the FEM coefficients u(t) = (t)(t)....... Una (t)) given by
du M
+ Au + F(u) = b
E(t) ~ NN - -
dt
where M, = (0r, Qg), Aij = (A( F(u),
b, = (f. Oz), and G.j = (os, (k(,))), 0z))). Letting G = LL we can then write this in the familiar notation as above
Mdu + Audt + F(u)dt bdt + LdW()),
from which an Euler-Maruyama time discretisation gives
Un (I - A.MM -A) - A,M (.M--F(un--i)) + A,M 'b + M n-1: n-1: where AW,- (0, A, eventually defining transition model of the form
Pu(u,/un-11 = N ((I A,M AA),,1 A,M A,M--((u"")) A,M b, A.MM-GG.M GM-
Note that also that the STATFEM methodology also allows for implicit discretisations which may be desirable for time-integrator stability. The transition equations for these approaches can be written in closed form, yet although they give Markovian transition models, the resultant transition densities necessarily Gaussian due the nonlinear dynamics being applied to the current state u, Letting en = LAW,- n-1 A,G,, then the implicit Euler is
p( u, U2 A)
M (un, u, A,A + A.F(u,) A,b 1n-1:
(10)
and the Crank-Nicolson is
M (u, Un-1) + A,Au, + (,F(u,)1)2) A,b en-1
(11)
where 4 = uu, + u. /2. Furthermore, to compute the marginal measure p(u,, this also requires integrating over the previous solution u again due nonlinear dynamics this will not necessarily be Gaussian.
In each of these cases, therefore, the transition equation is
M(Un, Un-1) = 1n-1
where we take, for the implicit Fuler,
M(u,, un-1) = M (u,, - up-1) + A,uu, + A,F(un) + A.b
and, for the Crank-Nicolson,
M(u,, un-1) M(u, un-1) + A,Au, A,F(u, A.b.
In practice due to conservative properties of the Crank-Nicolson discretisation we use this for all our models in this work.
Discretised solutions u, are mapped at time n to "pseudo-observations"" via the observation process
X, Hu, Fas r, ...
This observation process has the density (x,, (u,) where v = (H. R). As stated in the main text, the oseuctoobservation operator H and observational covariance R are assumed known in this work. We typically use a diagonal covariance, setting R I In the PDE case, for a given STATFEM discretisation above, these pseudo-observationn given by Xobs E "= The pseudo-observation operator thus acts as an interpolant, such that
assumed to be taken user-specified grid,
Hu, [ n.d)), uh (X n.d)) ... Rx n nA)))
19

---

Under review as a conference paper at ICLR 2023
For the ODE case, have worked with observation operators that extract relevant entries from the state vector. The pseudo-observation are mapped to high-dimensional observed data through possibly nonlinear observation model which has the probability density Po (x,). Recall that in this, / are neural network parameters. This defines the decoding component of our model (see Figure
Nonlinear Filtering for Latent State Estimation. To perform state inference given set of pseudo- observations we use the ExKF. The ExKF constructs an approximate Gaussian posterior distribution via linearising about the nonlinear model M(-). The action of the nonlinear MM-) is approximated via tangent linear approximation. We will derive our filter in the general context of nonlinear Gaussian SSM given by
Transition:
M(u Uz-1) = en-1,
NO Q),
Observation:
X, Hu,, + In: Fn ~ NNO,
This allows for the use of implicit ime-integrators and subsumes the derivation for the explicit case. We assume that at the previous timestep an approximate Gaussian posterior has been obtained, p(u,- - 1, (mm,, Cn-1). For each the ExKF thus proceeds as:
1. Prediction step. Solve M(m,, m,-1) = 0 for m.. Calculate the tangent linear covariance update:
C₂ J (Jn-ICn-1d n-1 JQJ
where Jn aM/eau.au i.. m and J a.M/Ou,-- ma-1" This gives pl u, 11:-1,, A) MMm,, C..)
2. Update step. Compute the posterior mean m, and covariance C,::
m. m, + (HC, Ry, Hmn), Cn = Cn H² (HĆ, + R) -R)'CHH... This gives pu,, [xmm A) = ((m,,, C,).
The log- marginal likelihood can be caleulated recursively, with each term of the log-likelihood computed after each prediction step:
logp(xiN|A) Σ
log pxn, / (1,
pix,/XIm-11 A) (Hm, HC,HH" +R).
Note that although we focus on the ExKF other nonlinear filters could be used; two popular alternatives the ensemble Kalman filter (Evensen] (2003). the particle filter (Doucet et al. 2000).
For linear dynamical model, such as the advection equation considered in Section 5.2 the EXKF reduces to the standard Kalman filter (Kalman 1960).
As mentioned the main text we can marginalise over the uncertainty in the encoder, via a Monte Carlo approximation:
puu,yyim: A) z / p(un A (x1 (Vin)) dxi:n
(12)
M
( A),
x qas--Nisn))
(13)
M
The intractable integral is approximated using samples from the encoder, which provides an approx- imate posterior in the form of a mixture of Gaussians distribution, where each p(u,, (x A)
(( c(O)) A similar marginalisation procedure can proceed over the parameters
p(unlyin)) 2
p(u, XIm (x. dAdx
(14)
Mr MA
p ( (i)), Xin ~ go(-)) A gx(-).
(0)
(15)
M i=1 j=1
20


Reviews:
[{"review_id": "k0_4dE90Nf", "reviewer": "Reviewer_91wk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper combines standard VAE with filter algorithms (for example extended Kalman filter). The VAE is used to encode and decode from observation space to the latent space (in the paper often referred to as pseudo-observation space). While the filter algorithm is used to estimate the marginal log likelihood over the latent physics process introduced via additional data likelihood term in the model. The parameters of the model are trained jointly using variational inference (ELBO). The model is tested on video and velocity field datasets. The model obtains similar or better performance compared to previous works in the field w.r.t MSE. ", "summary_of_the_review": "Overall the paper is very well-written and places itself nicely among the existing work. Acknowledging this is a subjective judgment, I find the proposed approach incremental as (i) many latent ODE/SDE/PDE works are already proposed, and (ii) the model does not achieve groundbreaking results on very interesting datasets. I also think knowing the functional form of the dynamics as well as the pseudo-observation operator and the noise covariance is slightly too restrictive. Certain claims are not well-supported (see \"Weaknesses\" question). I also have concerns about the datasets, comparison partners. and fairness of the comparisons. I recommend a reject but would be happy to update my review based on the author reply.\n", "strength_and_weaknesses": "Strengths:\n- The paper is very well-written. Most concepts are communicated clearly and the related work section was thorough.\n- It touches upon a timely topic, i.e., latent differential equation systems with physical information.\n\nWeaknesses:\n- Both the pseudo-observation operator and the noise covariance are assumed to be known in this work. Isn't this too strong of an assumption?\n- Since the model and inference heavily resemble KVAE (Fraccaro et al., 2017), the differences should be discussed. In my understanding, the linear dynamical system formulation in KVAE is generalized here. Prior dynamics parameters $\\Lambda$ also resemble the coefficients of the linear transition model, i.e., $A$, and $b$ in KVAE; yet, the functional form of the transitions is assumed to be known here (with unknown parameters). Anything else? \n - Certain claims are not well-supported:\n    - The authors mention that they form a ‘physics-informed prior on the latent space.’ However, I would argue that the latent space is the exact physical model rather than just a prior on it. They have argued that the noise term captures model misspecification, however, none of the experiments truly supports this claim: all of your experiments are with the true underlying physical model (with known/partially known or unknown parameters) of the system.\n    - The following claim made is not well supported: 'this explicit likelihood is introduced to obtain a well-defined formulation for embedding’. The learned embedding for $x$ is given by the encoder $p(x|y)$. From the formulation, it is not immediately clear how eq7 affects the encoder in order to have ‘well-defined embeddings’. Based on the loss I see that we have a marginal likelihood term for $p(x|\\lambda)$ in which we use the data likelihood term defined here. But at the moment I do not see in what way this would affect the variational encoding distribution $q(x|y)$. This should be (at least empirically) demonstrated.\n- I have major concerns about the experiments, please see the \"quality\" point in the next question.\n\nQuestions:\n- I'm not able to fully follow between eq5 and sec3.2. Is there any novelty here or is it an adaptation of STATFEM into this framework? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W97yfSVB5V4", "reviewer": "Reviewer_8UxT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work combines VAEs and nonlinear state space models to model high dimensional time series data of physical systems.\n\nThe proposed $\\Phi$-DVAE uses a variational autoencoder to map the high dimensional data in a latent representation (latent observations) whose dynamics can be modeled by a non linear state space model derived from the ODEs or PDEs governing the physical system. Joint estimation of latent states and model parameters is obtained by maximizing the ELBO,  using the extended Kalman filter for state estimation of the SSM.\n", "summary_of_the_review": "The paper is interesting and novel, but for me to argue for acceptance its exposition needs to be improved and simplified.", "strength_and_weaknesses": "_STRENGHTS_\n\n1. This work in an interesting extension of the  StatFEM framework for modelling high dimensional data derived from an underlying low dimensional physical system with known dynamics.\n2. While most of the previous work has focused on modelling simpler dynamics, the proposed model can handle highly nonlinear systems governed by ODEs or PDEs.\n3. The added stochasticity of the StatFEM model allows to deal with potential model misspecifications.\n4. The model performs well in the chosen experiments, outperforming or performing comparably to competing methods. \n\n_WEAKNESSES_\n\n1. The StatFEM framework of the dynamic model presented in section 3.1 is quite complex for the average reader in the ICLR community. In its current state i therefore think that this paper would not have the impact it could.\nThere are a number of improvements that the authors could do to make the paper more accessible, for example:\n\n    1. A simple running example could be used throughout the paper to provide more intuition to the reader.\n    2. It would be good to have the full derivation of the dynamic model for the lorenz-63 or advection PDE example (even in the appendix).\n    3. You could add pseudo code for the method.\n\n2. In the model you assume the emission distribution of the SSM to be known. How come did you do that? Couldn't you have learnt those parameters as well? \n\n3. I was surprised to see that the VRNN is slightly better than your model in the Korteweg-de Vries PDE example. Can you provide more intuition on why this is the case? I would have expected a model like the VRNN, which has no inductive bias of physical knowledge, to have a hard time learning to model these compex dynamics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Fgb3-kxpAa", "reviewer": "Reviewer_66rK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Bayesian, probabilistic approach to data assimilation and parameter estimation for physical systems with unknown observation process. It takes the form of a dynamic VAE integrating in its temporal model the parameterized differential equation that drives the physical phenomenon, allowing the model trained with an ELBO to jointly estimate the unknown parameters and the physical state. The empirical performance of the model is then assessed on three differential equations where its ability to reconstruct the signal and estimate its parameters are demonstrated.", "summary_of_the_review": "This paper tackles a well motivated problem and introduces a well designed model to solve it, by integrating physical dynamics prior into a Bayesian, VAE-based model. However, I have concerns regarding the experimental results which I believe are not sufficient to show the relevance of the proposed model w.r.t. prior work, and some clarity issues should be fixed to improve the quality and accessibility of the paper to the ICLR audience. Therefore, I find this paper to lie below the acceptance threshold.\n\nNonetheless, I believe that my concerns can be addressed during the discussion phase with the authors and the other reviewers, and would be glad to increase my score in this case.", "strength_and_weaknesses": "### Strengths\n\nTo the best of my knowledge, this paper tackles the **relevant problem** of data assimilation and parameter estimation in the ML x physics community. In this regard, it is **well motivated** as it is designed to operate not directly on the true physical state like many prior approaches, but on an indirect observation of this state whose specifications are unknown. This setting, as well as the designed model, **are novel in the ML x physics domain**. Models and results are, for the most part, **clearly presented and well designed**, with explicit motivations for each of the model components accounting for known and unknown information in the studied phenomenon. **Encouraging experimental results** show the capacity of the model to correctly reconstruct the physical system.\n\n### Weaknesses\n\nThere are two weaknesses that limit, to the best of my understanding, the significance of the presented contributions.\n\nMost importantly, **the empirical evaluation of the model does not suffice to properly evaluate the contributions**, because of three main issues.\n - There is a **lack of relevant baselines** in the comparison. Section 5.1 includes no comparison, hence results can hardly be used to draw conclusions. Section 5.2 only considers KVAE, and Section 5.3 only VRNN, GP-VAE and a weak baseline (a non-dynamic VAE). This is insufficient to draw informed conclusions on the relevance of the proposed model. To the very least, VRNN, GP-VAE and KVAE should be included in all experiments. Moreover, other recent baselines could be considered, e.g. Karl et al. (2017, cited in the paper), Krishnan et al. (2017), Li et al. (2018), Yıldız et al. (2019, cited in the paper), de Bézenac et al. (2020), Lu et al. (2020, cited in the paper), to cite probabilistic methods only; note that many ODE/PDE baselines may be considered as well, cf. the Minor Issues section. Note that I am not asking for all these methods to be included but I would expect some of them to be in the paper; alternatively, I would be interested in a discussion from the authors about their relevance in our context.\n - The experiments **fail to prove the relevance of the proposed model w.r.t. the state of the art**. VRNN outperforms the proposed model in reconstruction MSE, and it appears that increasing the dimensionality of KVAE could outperform it as well -- higher values than $n_x = 64$ should be tested. In contrast with the proposed model which requires strong prior knowledge on the physical phenomenon, these baselines are generic and they would be expected to underperform, which is not the case.\n - Even though motivated in Section 3, **main model components are not properly justified with experimental results**. As highlighted in Section 3.1, the introduced dynamics model is stochastic to circumvent uncertainties in the state estimation, but this property is never assessed in the experiments, especially as all considered differential equations are in practice studied in their deterministic version. Moreover, the benefits of adding prior knowledge and inferring the true parameters of the system are not demonstrated in the experiments either: this could be solved by including an ablation study.\n\nSecondly, there are **some clarity issues in the description of the model and its training**, in particular for the non-experts in physical data assimilation like in the ICLR audience. The discretization and implementation of the dynamics model in Section 3.1 are obscure and could benefit from more intuitive explanations. Moreover, the marginalization over $\\mathbf{u}$ in Section 4 (paragraph \"Nonlinear Filtering\") would require further explanations. Finally, **the derivation of the ELBO in Section 4 should be clarified**. Indeed, the proof in Appendix B involves the assumption that the variational posterior is the exact filtering posterior, which is neither discussed nor apparent elsewhere in the paper.\n\nKrishnan et al. Structured Inference Networks for Nonlinear State Space Models. AAAI 2017.\\\nLi et al. Disentangled Sequential Autoencoder. ICML 2018.\\\nde Bézenac et al. Normalizing Kalman Filters for Multivariate Time Series Analysis. NeurIPS 2020.\n\n### Minor Issues\n\n - By design, the proposed model cannot handle non-Markovian data, as the latent dynamics model is a first-order differential equation and the encoder $q_{\\phi}$ is factorized as $q_\\phi (\\mathbf{x} \\mid \\mathbf{y}) = \\prod_n q_\\phi (\\mathbf{x}_n \\mid \\mathbf{y}_n)$. This is a limitation of the model that could have been easily avoided like in many deep SSM using filtering or smoothing strategies -- cf. Fraccaro (2018, Section 3.3).\n - The related work misses two relevant lines of work.\n   - Some aspects of ML for physics were not addressed. One could refer for examples to the introduction and related work of Yin et al. (2021) containing missing references for parameter estimation and incorporation of physical priors in ML systems.\n   - Since the paper deals with video-like input data, it would be beneficial to discuss the relations of the model with existing VAE-based approaches for video modeling, which have already been approached with latent and state-space methods. Cf. for instance the related work of Wu et al. (2021).\n\nYin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting. ICLR 2021.\\\nWu et al. Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction. CVPR 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Vjt7Z2BjD", "reviewer": "Reviewer_FFuK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a VAE-like framework with a sequential latent variable following a (discretized) PDE. They test it for three different systems.", "summary_of_the_review": "While the paper is nicely written basically, the description of the relation between the proposed methods and other VAE-like sequential models remains unclear. Also, the experiments are only with very artificial data and thus are not so strong.", "strength_and_weaknesses": "### Strengths\n\nThe paper is basically nicely written and easy to follow.\n\n### Weaknesses\n\nW1.\nThe relation to the related work is not clearly stated. Given the variety of VAE-like models with sequential structures, it is hard to evaluate the particular design choice of the proposed method unless it is manifested with a clear comparison to the most relevant models.\n\nW2.\nAlthough the authors motivate themselves by referring to the \"traditional approaches\" that \"focus on well-defined observation operators whose functional forms are typically assumed to be known,\" (in the abstract), in my understanding, the proposed method also stands on an observation operator whose functional form is assumed to be known; the map from $u$ to $x$ is linear (Eq. (7)), and the map from $x$ to $y$ is supposed to be selected with prior knowledge (Section 3.3).\n\nW3.\nExperiments are quite artificial and no real data appears.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject