Physics-aware, probabilistic model order reduction with guaranteed stability
Authors: Sebastian Kaltenbach
Phaedon Stelios Koutsourelakis
Venue: ICLR 2021 Poster (2021)
Keywords: inductive bias
probabilistic generative models
state-space models
model order reduction
slowness
long-term stability

Abstract: Given (small amounts of) time-series' data from  a high-dimensional, fine-grained, multiscale dynamical system, we propose a generative framework for learning an effective, lower-dimensional, coarse-grained dynamical model that is predictive of the fine-grained system's long-term evolution but also of its behavior under different initial conditions.
We target fine-grained models as they arise in physical applications (e.g. molecular dynamics, agent-based models), the dynamics  of which are strongly non-stationary but their transition to equilibrium is governed by unknown slow processes which are largely inaccessible by brute-force simulations.
Approaches based on domain knowledge heavily rely on physical insight in identifying temporally slow features and fail to enforce the long-term stability of the learned dynamics. On the other hand, purely statistical frameworks lack interpretability and rely on large amounts of expensive simulation data (long and multiple trajectories) as they cannot infuse domain knowledge. 
The generative framework proposed achieves  the aforementioned desiderata by  employing a flexible prior on the complex plane for the latent, slow processes, and  an intermediate layer of physics-motivated latent variables that reduces reliance on data and imbues inductive bias. In contrast to existing schemes, it does not require  the a priori definition of projection operators from the fine-grained description and addresses simultaneously the tasks of dimensionality reduction and model estimation.
We demonstrate its efficacy and accuracy in multiscale physical systems of particle dynamics where probabilistic, long-term predictions of phenomena not contained in the training data are produced.

Published as a conference paper at ICLR 2021
PHYSICS-AWARE, PROBABILISTIC MODEL ORDER RE- DUCTION WITH GUARANTEED STABILITY
Sebastian Kaltenbach, Phacdon-Stelios Koutsourelakis
Professorship of Continuum Mechanics
Technical University of Munich
(sebastian, ian. ka tenbach, p.s. htt.aaaca.pp...toooourrraaaaaaa. koutsourellaklssjtumm.do
ABSTRACT
Given (small amounts of) time-series' data from a high-dimensional, fine-grained, multiscale dynamical system, we propose generative framework for learning effective, lower-dimensional, coarse-grained dynamical model that is predictive of the fine-grained system's long-term evolution but also of its behavior under different initial conditions. We target fine-grained models as they arise in physi- cal applications (e.g. molecular dynamics, agent-based models), the dynamics of which are strongly non-stationary but their transition to equilibrium is governed by unknown slow processes which are largely inaccessible by brute-force simu- lations. Approaches based domain knowledge heavily rely on physical insight in identifying temporally slow features and fail to enforce the long-term stability of the learned dynamics. On the other hand, purely statistical frameworks lack interpretability and rely on large amounts of expensive simulation data (long and multiple trajectories) as they cannot infuse domain knowledge. The generative framework proposed achieves the aforementioned desiderata by employing flex- ible prior the complex plane for the latent, slow processes, and intermediate layer of physics-motivater motivated latent variables that reduces reliance on data and im- bues inductive bias. In contrast existing schemes, it does not require the priori definition of projection operators or encoders and addresses simultaneously the tasks of dimensionality reduction and model estimation. We demonstrate its ef- ficacy and accuracy in multiscale physical systems of particle dynamics where probabilistic, long-term predictions of phenomena not contained the training data are produced.
INTRODUCTION
nigh-dimmessonal, nonlinear systems are ubiquitous in engineering and computational physics. Their nature is in general multi-scald]] E.g. in materials, defects and cracks occur scales of millimeters to centimeters whereas the atomic processes responsible for such defects take place much finer scales (Belytschko & Song 2010). Local oscillations due to bonded interactions of atoms (Smit 1996) take place at time scales of femtoseconds (10-15g), whereas protein folding pro- cesses which can be relevant for e.g. drug discovery happen at time scales larger than milliseconds (10-3s). ). In Fluid Mechanics, turbulence phenomena are characterized by fine-scale spatiotemporal fluctuations which affect the coarse-scall response (Laizet & Vassilicos) (009). In all of these cases, macroscopic observables are the result of microscopic phenomena and better understanding of the interactions between the different scales would be highly beneficial for predicting the system's evo- lution (Givon et al. 2004). The identification of the different scales, their dynamics and connections however is a non-trine non-trivial task and is challenging from the perspective statistical as well as physical modeling.
With the term multiscale we refer to systems whose behavior arises from the synergy of two or more processes occurring different (spatio))emporal scales. Very often these processes involve different physical descriptions and models (i.e. they are also multi-physics). We refer to the description/model at the finer scale fine-grained and the description/modell the scale as coarse-grained.

---

Published as a conference paper at ICLR 2021
High-dimensional System a:
Dunsity
Velocity
Latent Physical variables X
G
Latent variables
Figure Visual summary of proposed framework. The low-dimensional variables act via probabilistic map G' as generators of an intermediate layer of latent, physicallymootivaeed variables X that are able reconstruct the high-dimensional system a with another probabilistic map F.
In this paper we propose novel physics-aware, probabilistic model order reduction framework with guaranteed stability that combines recent advances statistical learning with a hierarchical archi- tecture that promotes the discovery of interpretable, low-dimensional representations. We employ a generative state-space model with two layers of latent variables. The first describes the latent dynamics using novel prior the complex plane that guarantees stability and yields clear dis- tinction between fast and slow processes, the latter being responsible for the system's 's long-term evolution. The second layer involves thysically-motivatee latent variables which infuse inductive bias, enable connections with the very high- high-dimensional observables and reduce the data require- ments for training. The probabilistic formulation adopted enables the quantification of a crucial, and often neglected, component in any model compression process, i.e. the predictive uncertainty due information loss. We finally want to emphasize that the problems of interest are Small Data ones due to the computational expense of the physical simulators. Hence the number of time-steps as well as the number of time-series used for training small as compared to the dimension of the system and to the time-horizon over which predictions are sought.
PHYSICS-AWARE PROBABILISTIC MODEL ORDER REDUCTION
Our data consists of N times-series { over T time-steps generated by computational physics simulator. This can represent positions and velocities of each particle in a fluid or those of atoms in molecular dynamics. Their dimension is generally very high i.e. M C CR ( >>>>>>>>>>> 1). In the context of state-space models, the goal is to find a lower-dimensional set of collective variables latent generators 2, and their associated dynamics. Given the difficulties associated with these tasks and the solutions that have been proposed in statistics and computational physics literature, dimer-dimensional variables
we advocate the use of an intermediate layer of physically-motivate X, (e.g. density or velocity fields), the meaning of which will become precise in the next sections. These variables provide a coarse-graines description of the high-dimensional observables and imbue interpretability in the learned dynamics. Using X, alone (without 2,) would make it extremely difficult to enforce long-term stability (see Appendix H.2p while ensuring sufficient complexity in the learned dynamics (Felsberger & Koutsourelakis 2019 Champion et al.) 2019). Furthermore and even if the dynamics of D; are first-order Markovian, this is not necessarily the case for X, (Chorin & Stinis 2007). The latent variables 2, therefore effectively correspond to a nonlinear coordinate transformation that yields not only Markovian but also stable dynamics (Gin et al.) 2019). The general framework is summarized in Figure I and provide details in the next section.
2

---

Published as a conference paper at ICLR 2021
2.1 MODEL STRUCTURE
Our model consists of three levels. At the first level, we have the latent variables 2, which are connected with X, in the second layer through probabilistic map G. The physical variables X, are finally connected to the high-dimensional observables through another probabilistic map F. We parametrize F, G with deep neural networks and denote by 0 and 0g the corresponding parameters (see Appendix D). In particular, we postulate the following relations:
jt.j 1t- exp(),) Jyri A3 C, a CN(O,)) 1), j = 1,2,
(1)
X: G(z,,01)
(2)
F(,,
(3)
We assume that the latent variables 2, are complex-valued and priori independent. Complex vari- ables were chosen as their evolution includes a harmonic components which are observed in many physical systems. In Appendix H.1 we present results with real-valued latent variables 2t,j and illustrate their limitations. We model their dynamics with a discretized on the complex plane with initial conditions 20.j CO o (0,000 The parameters associated with this level are denoted summarily by 0n = {oū o². > These, along with 0,, O2 mentioned earlier, and the state variables X, and 2, have to be inferred from the data Z₁. We explain each of the aforementioned components in the sequel.
UaattinnUhleneeckk process
2.1.1 STABLE LOW-DIMENSIONAL DYNAMICS
While the physical systems (e.g. particle dynamies) of interest are highly non-stationary. they gen- erally converge to cquilibrium in the long-term. We enforce long-term stability here by ensuring that the real-part of the Xy's Equation (T negative, i.e.:
X1 R(2,) i (A2)) with R(A,) < 0
which guarantees first and -order-order stability i.e. the mean as well as the variance are bounded at all time steps.
The transition density each process Zt,j is given by:
(R((2 (4,5)
R (1(2-11))
p (Zt.j zt-1,)) = .V
1 85 R2
(5)
(21,5)
(21-11))
2
where the orthogonal matrix R, depends the imaginary part of Ay:
cos( (aA))
-sin(3()) (3())) cos (Ss())))))
R =
(6)
sin (3()))
and the decay rate 8j depends on the real part of Ay:
8j = ex)()
(7)
ie. the closer to zero the latter is, the "slower" the evolution of the corresponding process is. As in probabilistic Slow Feature Analysis (SFA) (Turner Sahani 2007, Zafeiriou et al. 2013). we set of = = e2 R(Ay)) = 1- 83 and oū. = 1. As consequence, priori, the latent dynamics are stationary and ordering of the processes 2t,j is possible on the basis of R(())) Henee the only independent parameters are the Aj. the imaginary part of which can account for periodic effects in the latent dynamics (see Appendix B).
The joint density of z, can finally be expressed as:
p(ZO:T)
M 2-11j 0o)p((0.5 (0o) J=1
II
The transition density between states at non-neighhouring time-instants also available analytically and is useful for training on longer trajectories or cases of missing data. Details can be found in Appendix B
'A short review of complex normal distributions, denoted by C.,, can be found in Appendix More details can be found in Appendix
3

---

Published as a conference paper at ICLR 2021
2.1.2 PROBABILISTIC GENERATIVE MAPPING
We employ fully probabilistic maps between the different layers which involve two conditional densities based on Equations 2 and (3) i.e.:
p 02) and p(X; 21, 01)
In contrast to the majority of physics-motivated papers (Chorin & Stinis 2007 Champion et al,
2019) as well as those based on transfer-operatte Klus et al. (2018), we note that the generative structure adopted does not require the prescription of restriction operator (or encoder) and the reduced variables need not be selected priori but rather are adapted to best reconstruct the observ- ables.
The splitring of the generative mapping into two parts through the introduction of the intermediate variables X; has several advantages. Firstly, known physical dependencies between the data æ and the physical variables X can be taken into account, which reduces the complexity of the associated maps and the total number of parameters. For instance, in the case particle simulations where X represents density velocity field, i.e. it provides coarsened or averaged description of the fine-scale observables, it can be used to (probabilistically) reconstruct the positions or velocities of the particles. This physical information can be used to compensate for the lack of data when only few training sequences are available (Small data) and can seen as a strong prior to the model order reduction framework. Due to the lower dimension of associated variables, the generative map between and X, can be more easily learned even with few training samples. Lastly, the inferred physical variables X can provide insight and interpretability the analysis of the physical system.
2.2 INFERENCE AND LEARNING
Given the probabilistic relations above, our goal is infer the state variables X x!177) as all model parameters O. We follow a hybrid Bayesian approach in which the posterior of the state variables is approximated using structured Stochastic Variational Inference (Hoffman et al., 2013) and MAP point estimates for = (0,, 0,, 02} are computed.
(1:n) well =
The application of Bayes' rule leads to the following posterior:
p(Xoo
(1:n) (1:n) 0.T FoT
0 00000
((rr))
p(
(1:m)
0:T
(1:m)
ZorT
(1mm)
0) p( X
0:7
(1m)
*0::T"
(1:n)
(10)
(1:n)
p
p(ap-r
(1m)
D:T
(1=))
0)
pl
x(1n)
0:1
(1m)
0:1
(1:n)
0) pl 20-
(1:m)
() p(0)
(11)
(1m)
p(
where p(0) denotes the prior on the model parameters. In the context of variational inference, we use the following factorization of the approximate posterio
g0 (x (1x)
(1:)) Fo.
(1:n),
II
II 1=
9
(=77 (1)
x go (x(01))
(12)
We approximate the conditional posterior of given X with complex multivariate normal which is parameterized using a tridiagonal precision matrix as proposed in Archer et al. (2013); Bamler & Mandt (2017). This retains dependencies between temporally neighbouring but the number of parameters grows linearly with the dimension of z which leads to a highly scalable algorithm. For the variational posterior of X employ Gaussian with diagonal covariance, i.e.:
O:T) = CN CN ()) B&
(1) 0:7
0:7 = Nu
(1
Is
(13)
We denote summarily with Φ the parameters involved and note that deep neural networks are used for the mean pa X (0) as well as the upper bidiagonal matrix B& X (i))). Details on the neural net architectures employed are provided in Section 4 and in Appendix D
We note that this factorization does not introduce any error due to the conditional independence of given X.
4

---

Published as a conference paper at ICLR 2021
It can be readily shown that the optimal parameter values are found by maximizing the Evidence Lower Bound (ELBO) F(ga (x (1:1)). 0) which is derived in Appendix C We compute Monte Carlo estimates of the gradient of the ELBO with respect to and 0 with the help of the reparametrization trick (Kingma & Welling 2013) and carry out stochastic optimization with the
0:7 2b.
ADAM algorithm (Kingma & Ba 2014).
2.3 PREDICTIONS
Once state variables have been inferred and MAP estimates MAP for the model parameters have been obtained, the reduced model can be used for probabilistic future predictions. In order do so for a time sequence used in training, we employ the following Monte Carlo scheme to generate a sample TT+p, i.e. P time-steps into the future:
1. Sample Xy and 2r from the inferred posterior (s(20:r Xo:1 (4 (Xo:r)
2. Propagate 2y for P time steps forward by using the conditional density in Equation (
3. Sample X and TT+P from PXT-PP *T+P; Of and p(æT+P XT+P, 0g AMA)) respectively.
More importantly perhaps, the trained model can be used for predictions under new initial con-
ditions, e.g. To- To achieve this, first the posterior p(zo| x J p(zo Xx, 0M p(Xo
0 OMAP) d.Xo must be found before the Monte Carlo steps above can be employed starting at T= 0.
RELATED WORK
The main theme of our work is the learning of low-dimensional dynamical representations that stable, interpretable and make use of physical knowledge.
Linear latent dynamics: In this context, the line of work that most closely resembles pertains
to the use of Koopman-operator theory (Koopman, 1931) which attempts to identify appropriate transformations of the original coordinates that yield linear dynamics (Klus et 2018). We note
that these approaches (Lusch et al., 2018 Champion et al., 2019 Gin et al. 2019 Lee & Carlberg 2020) require additionally the specification of an encoder i.c. map from the original description to the reduced coordinates which we avoid in the generative formulation adopted. Furthermore only small fraction are probabilistic and can quantify predictive uncertainties but very often employ restrictive parametrizations for the Koopman matrix in order to ensure long-term stability (Pan & Duraisamy 2020). To the best of our knowledge, none of the works along these lines employ variables and as result have demonstrated their applicability only in dimer-dimensional problems and require very large amounts training data or some ad hoc pre- processing. We provide comparative results with Koopman-basee deterministic and probabilistic models in Appendix H.3
additional,
Data-driven discovery of nonlinear dynamics: The data-driven discovery of governing dynamics has received tremendous attention in recent years. Efforts based on the Mori-Zwanzig formalism can accurately identify dynamics of pre-defined variables, which also account for memory effects, but cannot reconstruct the full fine-grained picture or make predictions about other quantities of interest (Chorin & Stinis 2007; Kondrashov al. 2015 Ma et al., 2019). Similar restrictions apply when nuuual- based models are employed as e.g. (Chen et al. 2018: Li et al. 2020). Efforts based on the popular SINDy algorithm (Brunton al., 2016) require additionally data of the time- derivatives of the variables of interest which when estimated with finitedifffrrencs introduce errors and reduce robustness. Sparse Bayesian learning tools in combination with ressicallymmoivvate variables and generative models have been employed by (Felsberger & Koutsourelakis (019) but cannot guarantee the long-term stability of the learned dynamics as we also show in Appendix H.2 and in the context of the systems investigated in section
Infusing domain knowledge from physics: Several efforts have been directed in endowing neural networks with invariances equivariances arising from physical principles. Usually those pertain to translation or rotation invariance and are domain--seciin as in SShhitt et al. (2017). More gen- eral formulations such as Hamiltonian (Greydanus et al., 2019: Toth al. 2019) and Lagrangian

---

Given this, the probabilistic map Published as a conference paper at ICLR 2021
Dynamies (Lutter al. 2019)) are currently restricted in terms of the dimension of the dynamical system. Physical knowledge has been exploited in conjunction with Gaussian Processes in (Camps-) Valls et al., (2118) well as in the context of PDEs for constructing reduced-order models in (Grigo & Koutsourelakis) 2019) or for learning modulated derivatives using Graph Neural Networks in (Seo al. 2020). Another approach involves using physical laws regularization terms or for augmenting the loss function as in (Raissi et al., 2019 Lusch et al.) 2018 Zhu et al., [219 Kaltenbach & Koutsourelakis/ 2020). In the context of molecular dynamics multiple schemes for coarse-grasd grrssnggganngggg which also guarantee long-term stability have been proposed by Noë (2018) and Wu al. (017 2018). our formulation,
eessicallymmoivvated latent variables are used to facilitate generative maps very high-dimensional data and serve the requisite information bottleneck in order to reduce the amount of training data needed.
Slowness and interpretability: Finally, in contrast to general state-space models for analyzing time-series data such as Karl et al.] (2016); [aanaauram et al. (2018); Li et al. (2019), the prior proposed on the complex-valus dicting the evolution of multiscale systems and in combination with the variables X₁ can provide interpretability and insight into the underlying physical processes.
2, enable the discovery of slow features which are crucial in pre-
4 EXPERIMENTS
The high-dimensional, fine-grained model considered consists of identical particles which can move in the bounded one-dimensional domain 8 E [-1, 1] (under periodic boundary conditions). The variables æ, consist therefore of the coordinates of the particles at each time instant and the di- mension of the system of is equal to the number of particles. We consider two types of stochastic par- wwrection-diffusio---e
ticle dynamics that correspond to type (section 4.2) behavior. In all experiments, the pessically-motiateee variables X, relate to a discretization of the partiele density into d 25 equally-sized bins for advection-diifisiinn dynamics and into d = 64 equally-sized bins for viscous riscous-Burgers -type dynamics. In order to auto- matically enforce the conservation of mass at each time instant, we make use of the softmax function exp( Xx,))
(section 4.1 and a viscous -orrours
ie. the particle density at bin & is expressed as
Eí exp(X.,) (F in Equation corresponds to a multinomial density, i.e.:
f!
exp(Xe.k) II
d
mx(3.)
p( 1 X₂)
(14)
mi (xx)) m2(x,)) my (x+)
Σd E exp(X.,,)
where (x:) is the number of partieles in bin k. The underlying assumption that, given X., the coordinates of the particles a; are conditionally independent. This does not imply that they move independently nor that they cannot exhibit coherent behavior (Felsberger & Koutsourelakis 2019). Furthermore, the aforementioned model automatically satisfies permutation-invaariane which cen- tral feature of the fine-grained dynamics. This is another advantage of the physically motivated intermediate variables X as enforcing such symmetries/invrrianees is not a trivial task even when highly expressive models (e.g. neural networks) are used (Rezende et al., 2019). The practical consequence of Equation (14) is that no parameters 02 need to be inferred.
The second map pertains to p(X, Zr, 01) which we represent with a multivariate normal distribu- tion with a mean and a diagonal covariance matrix modeled by neural network with parameters 0,. Details of the parameterization can be found in Appendix
We assess the performance of the method by computing first- and second-order statistics illus- trated in the sequel as well as in Appendix[] We provide comparative results on the same examples in Appendix H where report on the performance of various alternatives, such formulation without the latent variables 2, as well as deterministic and probabilistic Koopman- based models. Moreover short study the effect of the amount of training data is included in Appendix G
4.1 PARTICLE DYNAMICS: ADVECTION-DIFFUSION
We train the model on N = 64 time-series of the positions of f 250 x 10 particles over T 40 time-steps which were simulated as described in Appendix D.I Furthermore, made

---

Published as a conference paper at ICLR 2021
of h. = complex, latent processes 2t,-- Details regarding the variational posteriors and neural network architectures involved can be found the Appendix D.1
In Figure 2 the estimates of the complex-valucd parameters Xj plotted well as the inferred associated processes žt, on the complex plane. We note the clear
and predicted time-cvolution separation of time-scales in the first plot with two slow processes, ones. This is also evident the indicative trajectories on the complex plane. A detailed discussion of the map learned (Equation ( between 2₁ and X can be found in the Appendix F
intermediate and two fast
Prod
ctrd Mrann
0.1
363
0.00
0.0
-0.55
-0.10
-015
-10
10
15
-0.2
-0.1
0.0
0.1
R(A
RR22)
91(2)
Figure 2: Estimated Ay (left) and the time evolution of two 2t,j processes where one is slow (middle) and the other fast (right).
In Figure we compare the true particle density with the one predicted by the trained reduced model. We note that the latter is computed by reconstructing the æ, futures. We observe that the model able to accurately track first-order statisties well into the future.
005
-1.0
6.5
0.0
0.5
-.0
75
100
125
150
200
T Timesstps in Ar
Figure 3: Particle density: Inferred and predicted posterior mean (bottom) in comparison with the ground truth (top). The red line divides inferred quantities from predicted ones. The horizontal axis corresponds to time-steps and the vertical to the one-dimensional spatial domain of the problem E 1-
A more detailed view of the predictive estimates with snapshots of the particle density at selected time instances is presented in Figure 4 Here, not only the posterior mean but also the associated uncertainty is displayed. We want to emphasize the last Figure at = 1000 when the steady state has been reached which clearly shows that our model is capable of converging to stable equilibrium.
5.300
2.801
1.311
23111
20110
1.000
33007
2002-
1401
Figure 4: Predicted particle density profiles at t 80, 120, 160, 1000 (from left right).
Since the proposed model is capable of probabilistically reconstructing the whole fine-grained pic- ture. i.e. 2a, predictions with regards to any observable be obtained. In Appendix [.I we assess the accuracy of predictions in terms recond-order statistics, and in particular for the probability of finding simultaneously pair of particles two specified positions.

---

Published as a conference paper at ICLR 2021
Finally, we demonstrate the accuracy of the trained in model in producing predictions under unseen initial conditions as described in section 2.3] Figure 5 depicts a new initial condition in terms of the particle density according to which particle positions Do drawn. The posterior the corresponding Zo (see section 2.3 Figure.
can be used to reconstruct the density as shown also on the same
0.0
4000 -500
Figure 5: New initial condition (not used in training) in terms of the particle density. Reference shows the actual initial condition, whereas the posterior mean and uncertainty bounds correspond to the reconstruction of the initial condition based the inferred latent variables 20-
Figure shows predictions of the particle density (i.e. first-order statistics) at various timesteps. We want to emphasise the frame on the right, for t = 500 which shows the steady state of the system. We note that even though the initial condition was not contained in the training data, our framework is able to correctly track the system's 's evolution and to predict the correct steady state.
Arlentee
473
138
119
53) 058
Figure 6: Predictions of the particle density at = 25, 75, 125, 500 (left to right) for on the new initial condition in Figure 5
4.2 PARTICLE DYNAMICS: VISCOUS BURGERS' EQUATION
In this example, we made use of N 64 sequences of f 500 x 10 particles over T = 40 time- steps. Details regarding the physical simulator, the stochastic interactions between particles as well as the associated network architectures are contained in Appendix D.2 As in the previous example, we employed the particle density with the softmax transformation for X, and complex-valued processes 2, the lowest model level.
In Figure 7 the estimated values for xy are shown where the clear separation of time-scales with three slow and two fast processes can be observed.
0.25
-0.33
-5
3((A)
Figure 7: Estimated Xy for the viscous Burgers' system.
In Figure g we compare the evolution of the true particle density with the (posterior mean of) the model-ppediccee one. We point out the sharp front at the lower left corner which is characteristic of the Burgers' equation and which eventually dissipates due to the viscosity. This is captured in the inferred as well in the predicted solution.
8

---

Published as a conference paper at ICLR 2021
0.0
005
-1.0
50
75
100
125
150
175
200
Timesteps In A
1.0
6.5
0.0
-0.5
25
50
75
1.00
125
150
175
200
Timesteps In At
Figure 8: Particle density: Inferred and predicted posterior mean (bottom) in comparison with the ground truth (top). The red line divides inferred quantities from predicted ones. The horizontal axis corresponds to time-steps and the vertical to the one-dimensional spatial domain of the problem s E 1,1
A more detailed view on the predictive results with snapshots of the particle density at selected time instances is presented in Figure We emphasize again the stable convergence of the learned dynamics to the steady state as well as the accuracy in capturing, propagating and dissipating the shock front.
Figure 9: Predicted particle density profiles t 40, 80, 120, 160, 1000 (from left to right).
We compare the accuracy of the predictions for second-order statistics of the fine- ane-grained system in terms of the two- two-particle probability in Appendix F.2 where excellent agreement with the ground truth, i.e. the one computed by simulating the fine- one-grained system, is observed.
CONCLUSIONS
We presented a framework for efficiently learning a lower-dimensional, dynamical representation of high-dimensional, fine-grained system that is predictive of its long-term evolution and whose stability is guaranteed. We infuse domain knowledge with the help of an additional layer of latent variables. The latent variables at the lowest level provide interpretabable separation of the time- scales and ensure the long-term stability of the learned dynamics. We employed scalable variational inference techniques and applied the proposed model in data generated from very large systems of interacting particles. In all cases accurate probabilistic predictions were obtained both in terms of first- and second-order statistics ability of the trained model to produce predictions under new, unseen initial conditions was demon- strated. An obvious limitation for the applicability of the proposed method to general dynamical systems pertains to the physically motivated variables X. While such variables are available for several classes of physical systems, they need be re-defined when moving to new problems and expert elicitation might be necessary. Furthermore, if an incomplete list of such variables is avail- able from physical insight, this would need to be complemented by additional variables discovered from data. To that end, one could envision that some of the abstract latent variables
a very long time range into the future. More importantly, the
functions
thereof, e.g. fx1), could be employed a generative map the form I; = F(X,, f(z)) instead of Equation (3). A final deficiency of the proposed model the lack of automated procedure for determining the appropriate number of z. We believe that the ELBO, which provides a lower bound on the model evidence, could be used for this purpose.

---

Published as a conference paper at ICLR 2021
REFERENCES
H. H. Andersen, M. Højbjerre. D. Sørensen, and P. S. Eriksen. The Multivariate Complex Normal Distribution, pp. 15-37. Springer New York, New York, NY, 1995. ISBN 978-1- 4612-4240-6. doi:
URL https://cot http://000..0g11.110000000
978 978-1-4612 4
Evan Archer, Il Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. Black box variational inference for state space models. arXiv preprint arXiv:1511.07300 2015.
Robert Bamler and Stephan Mandt. Structured black box variational inference for latent time series 2017.
models. arXiv preprint
Ted Belytschko and Jeong-Hoon Song.
Coarse-graining of multiscale crack propagation. Interna- tional journal for numerical methods in engineering, 81(5)3377-563 2010.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113(15):3932-399) 2016.
Gustau Camps-Valls, Luca Martino, Daniel H Svendsen, Manuel Campos-Taberner Jordi Muñoz- Marí, Valero Laparra, David Luengo, and Francisco Javier Garcfa- Haro. Physics-aware gaussian processes in remote sensing. Applied Soft Computing, 68:69-8 82, 2018.
Kathleen P Champion, Steven L Brunton, and J Nathan Kutz. Discovery nonlinear multiscale systems: Sampling strategies and embeddings. SIAM Journal on Applied Dynamical Systems, 18 (1):312-333, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Alina Chertock and Doron Levy. Partiele Methods for Dispersive Equations. Journal of
Computational Physics, 171(2):708-730, August 2001. ISSN 0021-9991. doi: 10.1006/
jeph.2001 6803. URL http:: http::/www.scieee wwtp:wwwwww..c.....ccce// rect comm//ccienc///rtiile//...... S002 0102219910111 968032
Alexandre Chorin and Panagiotis Stinis. Problem reduction, renormalization, and memory. Com-
munications in Applied Mathematics and Computational Science, 1(1):1-27, 2007.
Georges-Hene Henri Cottet and Petros D. Koumoutsakos. Vortex Methods: Theory and Practice. Cam- bridge University Press, Cambridge New York, 2 edition edition, March 2000. ISBN 978-0-521- 62186-1.
L Felsberger and PS Koutsourelakis, Physics-constrained, data-driven discovery of coarse-grained dynamics. Communications in Computational Physics, 2555):125--1301, 2019. doi: 10.4208/ cicp.0A-2018-0174
Craig Gin, Bethany Lusch, Steven Brunton, and Nathan Kutz. Deep learning models for global coordinate transformations that linearize pdes. arXiv preprint arXiv:1911.027100 2019.
D. Givon, R. Kupferman, and A. Stuart. Extracting Macroscopic Dynamics: Model Problems and Algorithms. Nonlinearity, 2004.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15379-15389, 2019.
Constantin Grigo and Phaedon-Stelios Koutsourelakis. A physics-aware, probabilistic machine learning framework for coarse-graining figh-dimensional systems in the small data regime. Jour- nal of Computational Physics, 397:108842, 2019.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. The Journal of Machine Learning Research, 14(1):1303-137,, 2013.
10

---

Published as a conference paper at ICLR 2021
Sebastian Kaltenbach and Phaedon-Stelle Koutsourelakis. Incorporating physical constraints in deep probabilistic machine learning framework for coarse-gradd gaassnggaining dynamical systems. Journal of Computational Physics, 419:109673, 2020.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep varia- tional bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06433 064322 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling, arXiv:1312.611,,22 2013.
Auto-encoding
variational bayes. arXiv preprint
Stefan Klus, Feliks Nüske, Péter Koltai, Hao Wu, loannis Kevrekidis, Christof Schütte, and Frank Noé. Data-Driven Model Reduction and Transfer Operator Approximation. Jourmal of Nonlinear Science, 28(3):855-010, June 2018. ISSN 1432-1467. doi: 10.10077:00033-00--943771 URL
fetps://doi.orgge 10070/00073/3322 017 9437-7
Dmitri Kondrashov, Mickaël D Chekroun, and Michael Ghil. Data-driven non-markovian closure models. Physica D: Nonlinear Phenomena, 297:33-55, 2015.
B. O. Koopman. Hamiltonian Systems and Transformations in Hilbert Space. Proceedings of the National Academy of Sciences of the United States of America, 17(5):315-118, 1931. ISSN 0027- 8424. URL Balhttss: www. jstor 32gg/saabee/B611
S Laizet and JC Vassilicos. Multiscale generation of turbulence. Journal of Multiscale Modelling, (01):177-196, 2009.
Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404: 404:108973, 2020.
Ju Li, Panayotis G. Kevrekidis, C. William Gear, and Ioannis G. Kevrekidis. Deciding the Nature of the Coarse Equation Through Microscopic Simulations: The Baby-Bathwate Scheme. SIAM Rev., (933):49-4477 July 2007. ISSN 0036-1445. doi: 10.113770700992003. URL http:
fddx.do1 org. 11w.1111370000000
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning interpretable deep state space model for probabilistic time series forecasting. In IJCAI, pp. 2901-2908, 2019.
Xuechen Li, Kim Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. arXiv preprint arXiv:2001.00 0228, 2020.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 91):1-10,, 2018.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint 1eX.iv:907.044000 2019.
Chao Ma. Jianchun Wang, and Weinan E. Model Reduction with Memory and the Machine Learning of Dynamical Systems. Communications in Computational Physics, 25(4):947-962, April 2019. doi: 00.4400//iicp.00-2111--222 Press WOS:00045596300000
ISSN 1815-2406.
Place: Wanchai Publisher: Global Science
Frank Noé. Machine learning for molecular dynamics on long timescales. arXiv preprint
arXiv:1812.07669, 2018. 2018.
Shaowu Pan and Karthik Duraisamy. Physics-informed probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19(1):80-550, 2020.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.

---

Published as a conference paper at ICLR 2021
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in neural information processing systems, pp. 7785-7794, 2018.
Danilo Jimenez Rezende, Sébastien Racanière, Irina Higgins, and Peter Toth. Equivariant Hamilto- nian Flows. arXiv: 13739 (cs, stat), September 2019. URL
org. abs/
1909. 13739
arXiv: 1909.13739.
Stephen Roberts. Convergence of Random Walk Method for the Burgers Equation. Mathematics of Computation, 52(116):647-673, 1989. ISSN 0025-5718. doi: 10.2307/2008486. URL URLhttp: / //www. jstor wtt.too..rr//table//20000/
Kristof T Schütt,, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8(1):1-8, 2017.
Sungyong Seo, Chuizheng Meng, and Yan Liu. Physics-aware difference graph networks for sparsely-observed dynamics. In International Conference on Learning Representations, 2020.
Berend Smit. Understanding molecular simulation: from algorithms to applications. Academic Press, 1996.
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. arXiv preprini arXiv: /909. J 13789, 2019.
Richard Turner and Maneesh Sahani. A Maximum-Likelihodd Interpretation for Slow Feature Anal- ysis. Neural Computation, 19(4):1022 -038 April 2007. ISSN (899-7667, doi: 10.1162/neco. 9.222.2.2
2007.19.44 1022. URL http::tttsstttcttccttt.ll doi org 10. 11621122222 62/ neco 2007
Hao Wu, Feliks Nüske, Fabian Paul, Stefan Klus, Péter Koltai, and Frank Noé. Variational koopman models: slow collective variables and molecular kinetics from short off-equilibrium simulations. The Journal of Chemical Physics, 146(15):154100 2017.
Hao Wu, Andreas Mardt, Luca Pasquali, and Frank Noe. Deep generative markov state models. In Advances in Neural Information Processing Systems, pp. 3975-3984, 2018.
Lazaros Zafeiriou, Mihalis A Nicolaou, Stefanos Zafciriou, Symeon Nikitidis, and Maja Pantic. Probabilistic slow features for behavior analysis. IEEE transactions on neural networks and learning systems, 27(5):1034-100 2015.
Yinhao Zhu, Nicholas Zabaras, Shaacdnn-Stelios Koutsourclakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data, Journal of Computational Physics, 394:56-81 2019.
12

---

Published as a conference paper at ICLR 2021
A COMPLEX NORMAL DISTRIBUTION
In this Appendix, the complex random normal distribution is reviewed. The mathematical definitions introduced follow Andersen al. (1955):
A y-variate complex normal random variable Y E CP with Y ~ CN Ec) is defined by complex mean vector E C' and a complex Covariance Matrix Ep E C4 The density with respect Lesbegue measures on C can be stated as:
(
det (ec) exp (- Ec)'EE (y uc))
(15)
where * indicates the conjugate transpose of a matrix.
This complex normal random variable has similar properties the well-known, real-valued coun- terpart. For instance, linear transformations of complex random normal variables are again complex random normal variables.
These properties directly follow from the fact, that for a complex random normal variable there exists an isomorphic transformation to a real valued 2p-variate normal random variable W RR This random normal variable is defined with mean
R(ee) ((ec)
PR
(16)
and covariance
R(EE) (
-(EE) R(EC))
Eg
(17)
Therefore: W ( ER)
As an example the real valued isomorphic counterpart of the standard complex random normal distribution CO, 1) is the bivariate normal distribution (0, IT.
13

---

Published as a conference paper at ICLR 2021
B CHOICE OF VARIANCE FOR A-PRIORI STEADY STATE
We derive transient and stationary properties of the complex- valued latent processes 2t.j and justify our choices for the model parameters. Based on Equation ① and for each process j, the dynamics can be written in terms of the real and imaginary parts in two dimensions as follows:
M (2.,)) 3 (2t.))
R(=
R(E,,))
= Aj
((1
+ tog
3(Et.))
(18)
or:
R(z5,) (イ,]
R(,00) ((1,0)
t-1
R R((e-ks)
A
+ to
A
(19)
S ((
(((tkk.))
k=
where M(E,,)), (EL,g) ~ (0, 1/2). The matrix Aj is given by (see also Equations (6
cos (3(A))
(in()
A, = s, R, c R(AA)
(20)
sin (32g))
cos cos( (32)))
and can be diagonalized as A = VP, V
where:
1 1
Te 0
V v2 i
P==
(21)
V is the conjugate transpose of V and X, denotes the complex conjugate of Xy. Due to the linearity of the model and the Gaussian initial conditions, i.e. R(zo,)), 3(20,) NN /,, the marginal
will remain Gaussian at all times t. A direct consequence of the above is that the mean of real and
imaginary parts is always zero, i.e.:
E EEBR(2 [1 [[3((t))]]
0
(22)
Furthermore, the covariance c(O) is given by:
c =
R(
MZ)s) 3(z,5)]
0b.1 A3
+ oz Σ k=
AA())**
(23)
(((,))
2
which upon use of the diagonalized form of A yields:
2 0o.1 V
I-e (3g)
c
VT
VT
(24)
(1)))a)
1-c 2 27:07
As mentioned earlier, a necessary condition for the long-term stability of the processes is that R((A)) < 0, in which case and as f - oo it leads to:
c.e0 -
a²
o
v
VT
(25)
2
2(1 22(().))
This implies that real and imaginary parts are asymptotically uncorrelated with a variance
2(1 2 By setting of 1 22((A, we enable direct comparisons between žr, solely on
the basis of Re(A,) i.e. the degree of slowness (Turner& & Sahani 2007: Zafeiriou et al., 2015 In this case the asymptotic variance of real and imaginary parts becomes 1/2. Finally by setting
σō. 1 we ensure that, priori, the processes žt.j are stationary, with c(i) = -(u) I i.e. no a-priori bias is introduced with regards to their transient characteristics.
We finally note that the autocovariance D(u) of each of these stationary processes j is given by:
D ()) E
R (Zttns) [3(2449))
(R()))) 3((,))] AJE
((t.))
(R(z))) (((N))] A] c.I) (3(2))
(26)
where A, and the covariance c(u) are given above. By exploiting the diagonalization of A, and
that c(u) = =(i = I (for the parameter values discussed earlier), we obtain that:
DD e" R(2,) (os((T((A)) (in())(A,))
sin(r3(A,)) cos (A,,))
(27)
One can clearly observe harmonic (cross- ccorelltion terms which depend on the imaginary part of the >j and can capture persistent periodic effects of the dynamical system in the long-time range.
14

---

Published as a conference paper at ICLR 2021
C DERIVATION OF THE ELBO
This section contains details of the derivation of the Sridence-Lower-Boune (ELBO) which serves the objective function for the determination of the parameters & and 0 during training. In partic- ular:
log pl F0:1
(1:n) (0)
= log p(
O.T
(1m))
X
0:T
(lan]
O.T
(1am)
①) d.x
O.T
(lin) (1m))
0:7
log
p(
(12z)
D:
(15)
PoT
(1:n)
0 O)p( X
0:T FoTT
(15n)
(1xm)
0)
Ts (x
(1:m)
=0(r)) dx((r))
d
( (1:n)
((()))
(1:n]
qsl
0.T
flog
p(=u=
(1m))
X ((1m)
0.1
(
(x
D.T
(1mm)
(1:x)
0 0)p(2 O)p(X0z
(1:1)
0.7
)
(1:m) (1m)
o:r
0)
q
X
0:T
(1:n)
=
dx
o:T
(1:n)
(1:1)
0:T
(1:n)
(()))) ).0)
(28)
15

---

Published as a conference paper at ICLR 2021
D DETAILS FOR EXPERIMENTS
This appendix contains details for our experiments involving moving particles that have stochas- tic interactions corresponding to either an Advection-Diffusion behaviour or viscous Burgers' type behaviour.
D.I PARTICLE DYNAMICS: ADVECTION-DIFUSIION
For the simulations presented
250 10 particles were used, which, at each microscopic time step öt = 2.5 x 10 performed random, non-interacting, jumps of size 8s I either to the left with probability Pteft 0.1875 or the right with probability Pright 0.2125. The positions were restricted in |-1, with periodic boundary conditions. It is well-known (Cottet & Koumoutsakos, 2000) that in the limit (i.e. ʃ + oo) the particle density p(x, () can be modeled with an advection-
840
diffusion PDE with diffusion constant D = (Pleft + Pright) 25t
65² and velocity v = (Pright Pieft) 8t
6s.
0p + 0t
0p Øs
Pp 052
D
8 E-
(29)
From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system. Sample initial conditions are shown in Figure T0:
Figure 10: Sample initial conditions for the advection-diffusion type dynamics.
The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure [1] The neural network used for the generative mapping between the low-dimensional states 2, and the mean and covariance for X, consists of only one dense layer, whereas the variational posterior on zo.r is parameterized using a dense Layer with ReLu activation followed by another dense layer.
X Xo.
Z
walh and acciatius
Dess laya
Px11
Σx.
Bo(
Figure 11: Neural Net architecture used for the particle dynamics corresponding to an advection- diffusion equation.
D.2 PARTICLE DYNAMICS: VISCOUS BURGERS' EQUATION
The second test-case involved a fine- tne-ggained system of f = 500 x 10 particles which perform interactive random walks i.e. the jump performed each fine-scale time-step 8l 2.5 10-3 depends on the positions of the other walkers. In particular we adopted interactions as described in Roberts
& Levy (200T); Li et al. (2007) so as, in the limit (i.e. when f co, öt + 0, 0s + 0), the particle density p(s,t) follows viscous Burgers' equation with 0.0005:
0p 1 10p
82p
+
V
$ SE (-1,1).
(30)
Øt 2 0s 012'
16

---

Published as a conference paper at ICLR 2021
From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system. Sample initial conditions are shown in Figure [2]
Figure 12: Sample initial conditions for the Burger's type dynamics.
The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure [3] The neural network used for the generative mapping between the low- oinedimesioonal states and the mean and covariance for X, consists of several dense layers with ReLu activation and Dropout layers avoid overfitting, whereas the variational posterior on #0.7 is parameterized using two dense layers with ReLu activa- tion followed by another dense layer.
X 0:T
2
Dexx Leyer with tooe Neanes and EdUU ecciebous
Dosa Leye with 2t Neaen and BrLl wntinatee
y
Deew with
Nearoes
Denac Lagez wish 100 Rezzoer
Deapn (p-83))
Pg(XoT) Bo(Yar)
Laper Nazzeu laed? actiatiie
Doperi (w01))
Ixsa Noaes
Px,11
Σx.
Figure 13: Neural Net architecture used for the particle dynamics corresponding to viscous Burg- ers' equation.
17

---

Published as a conference paper at ICLR 2021
E DETAILED ANALYSIS OF THE GENERATIVE MAPPING AND THE SLOW LATENT VARIABLES
In this Appendix take closer look at the generative mapping and the (slow) latent variables learned. For the Advection-Diffusin example, we discovered two slow processes (Section[4.) (.1ioon/41.... and the marginally faster process #2- The rest of the processes were very fast in comparison and took values close to the zero point of the complex plane during the inference as well as during the prediction phase.
In order to visualize the influence through the generative mapping of these two slow processes, we the value of all other processes to zero and then reconstructed the fine-gres greeiera state based dif- ferent absolute values of 21 and Z2. The result are shown in Figure [4]in in terms of the reconstructed particle density. It is clearly visible that those two processes are responsible for variety of density profiles. In accordance with their slowness, 21 (the slightly slower process) is responsible for the most striking changes, whereas the other slow process generates some smaller scale fluctuations.
#2
Figure 14: Reconstruction of the particle density profiles for various values of the two slowest processes 21 and 22 identified. All other latent variables ž1, were set to zero.
18

---

Published as a conference paper at ICLR 2021
F TWO-POINT PROBABILITY
This appendix contains the predictive estimates for the two-point probability, i.e. the probability of finding two particles simultancously in lwo bins (61, 62). This two-point probability can be com- puted based on the reconstructed fine-grained system and corresponds to second order statistic.
F.1 PARTICLE DYNAMICS: ADVECTION- DIFFUSSON
The estimated two-point probability as well as the comparison to test data is shown for two indicative time-instants in Figure 15 and 16
We note a very good agreement with the ground truth.
XC09
12-
2314
b1
Figure 15: Two-point probability at time step 90: On the left the two-point probability of the data shown as reference, the figure in the middle contains the predictive posterior mean whereas the figure on the right contains the standard deviation. The figure on the left and the figure in the middle share the same colorbar.
1932
1892
23
193
D.
Figure 16: Two-point probability at time step 140: On the left the two-point probability of the data shown reference, the figure in the middle contains the predictive posterior whereas the figure on the right contains the standard deviation. The figure on the left and the figure in the middle share the same colorbar.
F.2 PARTICLE DYNAMICS: VISCOUS BURGERS' EQUATION
The estimated two-point probability well the comparison test data is shown for two indicative time-instants in Figure 17 and T8 We note a very good agreement with the ground truth.
0.0000E
10
CO0E325
DJ000S
X
COC925
0,0000
8
24
C00925
44
C00289
25
CO0923
46
0,00000
b.
Figure 17: Two-point probability at time step 90: On the left the two-point probability of the data shown reference, the figure in the middle contains the predictive posterior whereas the figure on the right contains the standard deviation. The figure on the left and the figure in the middle share the same colorbar.
19

---

Published as a conference paper at ICLR 2021
COC325
CZC0R
8
CA0229
5,0000
6,000
Figure 18: Two-point probability time step 140: On the left the two-point probability of the data shown as reference, the figure in the middle contains the predictive posterior mean whereas the figure on the right contains the standard deviation. The figure on the left and the figure in the middle share the same colorbar.
20

---

Published as a conference paper at ICLR 2021
G EFFECT OF THE AMOUNT OF TRAINING DATA
This section contains a study on the influence of the amount of training data. We illustrate this in the context of the Advection-Diffusiin example (section 4 by using 16 time sequences instead of the 64 employed earlier. In the figures below we note that the proposed model is capable capturing the main features of the system's dynamics as well as the correct steady state even with fewer training data. We believe that this is due to the intermediate layer of which introduces an information bottleneck. Finally, and as one would expect, fewer training data leads to increased predictive uncertainty.
variables X
0.500
Aarkrance
66466
3 Desatta
00456
40-3 Steddad Devirian
60450
51425
514.3
514.3
60015
65015
600
60025
60025
60322
235
2.50
2.25
025 050 8.75 1.50
BORCE
3335 -050 -825 0:0 935 030 0.75
1.38
-1
0.25
0.01
929 8:50 0.75 155
Figure 19: Predicted particle density profiles at t = 80, 160, 1000 (from left to right) with 64 sam- ples.
60500
00500
6,500
6.0115
6:415
4 Sardard
6045
6/425
6/425
pm
64025
0:025
6.022
60000
-000-17 -175
036
B35 030
6.75
1.08
60000
-320
-335 -050 -0.25 036 035 6.5 6.75 100
6,000
-3.00
030
125 3.50 6.75 138
Figure 20: Predicted particle density profiles at t = 80, 160, 1000 (from left to right) with 16 sam- ples.
21

---

Published as a conference paper at ICLR 2021
H COMPARISON WITH OTHER APPROACHES
This appendix contains results obtained by other methods for the test cases discussed in the main text. The simulation data is identical to the one used for the proposed method and details of the specific algorithms are described in the following.
H.1 REAL-VALURE LATENT SPACE
To demonstrate the utility a complex-valued latent space, the two examples were also solved with real-valued Zt.j. The only difference here is the restriction of the latent variables Rt.j and the Aş to real values.
A model with real-valued latent space is also capable of ensuring the stability but it is not capable of capturing periodic components of the dynamics. As most physical systems (including the two examples) contain such components, the algorithm is not able accurately model the dynamics and fails in generating reliable predictions. This can be readily observed in the extrapolative predictions of Figures 21 and 22 where, apart for the biased results, one also note increased predictive
uncertainty.
Hur Dardin
Ellver Deniiie
Decdion
130
2301
2401
2001
138
33C
2000
Figure 21: Ddvection-Diffusiii system: Predictions at 0, 80, 160 and 1000 obtained with real- valued latent variables 2t,j-
130
saherncc
Figure 22: Burgers' system: Predictions at / = 0, 80, 160 and 1000 obtained with real-valued latent variables 2t,j-
22

---

Published as a conference paper at ICLR 2021
H.2 No STABLE LATENT SPACE
Another possibility is to employ only the physically motivated latent variables X, and remove com- pletely the latent variables 2, in the first layer. This approach similar the one investigated in Felsberger & Koutsourelakin iakis (2019) and Kaltenbach & Koutsourelakis (2020) as well as to the idea of neural ODEs (Chen et al.] 2018). In this case, one must learn directly the dynamics of X; and for this purpose we employed three-layer. fully-connected neural network NN N as follows:
X (N)) + tE,
e~ C~
(31)
The learned dynamics are in general non-linear and stability is not guaranteed. As can be observed in Figures 23 and 24 the trained model is capable of producing accurate predictions for some time- steps but eventually both cases predictions become unstable. This could also be problematic when the trained model is used to make predictions with new initial conditions the chaotic nature of the nonlinear dynamics can lead to significant errors even for shorter time horizons.
340
Figure 23: Advection- ddvection-Diffusson system: Predictions t 0, 80, 160 and 1000 obtained without 21 and with the model of Equation (3I.
410
4.0
530
5.11
X.03
130 00-471 26
330
Figure 24: Burgers' system: Predictions at t = 80, 160 and 1000 obtained without 2N and with the model of Equation (3..
23

---

Published as a conference paper at ICLR 2021
H.3 KOOPMAN-BASED MODELS
The final alternative explored involved probabilistic and deterministic Koopman-based models for the latent dynamics. We kept the generative framework of our model and did not use encoder as for instance in Gin et al. (2019) in order to remove the effect of the associated model choice. For the same reason, we retained the intermediate variables X, even though these do not appear in any known Koopman-operale implementations. We replaced our complex-valued dynamics of the latent processes 2t.j with the models described in the sequel.
H.3.1 PROBABILISTIC KOOPMAN CROPMAN-BASED MODEL
We used real-valued latent variables 2, which are not a-priori independent and whose dynamics are parameterized with a Koopman matrix K and a diagonal noise matrix W:
2++1 Kz WE, E~,(O.
(32)
The learned matrix K is not guaranteed be stable in the absence of additional constraints but in both cases examined the eigenvalues of the learned K were real smaller than one. Long-term term predictions were stable and for the Burgers' case Figure 26 we were also able to reach the true steady state. For the (simpler) Advection-Diffusion example in Figure 25 an incorrect steady state was reached and the predictive quality started to deteriorate after some time-steps. In comparison to framework, the probabilistic Koopman-based model does not provide direct separation between slow and fast processes and therefore its interpretability reduced.
1,20
1071
139
1000
440
340
Figure 25: ddveetion-Diffusion system: Predictions at 0, 80, 160 and 1000 obtained with the probabilistic Koopman- based model of Equation (32)
Deration
km0
330
320
330
0114
Figure 26: Burgers' system: Predictions at t = 80,0, 160 and 1000 obtained with the probabilistic Koopman-based model of Equation 32)).
H.3.2 NON-PROBABILISTIC KOOPMAN LEARNING
We also used real-valued latent variables z, with deterministic dynamics which were parameterized as follows:
1 K z;
(33)
The absence of noise in comparison to Equation 32 led in both cases to an estimate for the Koop- man matrix K that did not yield stable predictions (each the learned K matrices had at least one eigenvalue which was larger than 1). We speculate that the lack of stochasticity made the model less capable dealing with the information loss. We also note Figures 27 and that the predictions obtained are, with an exception of few time-steps. highly inaccurate.
24

---

Published as a conference paper at ICLR 2021
3.0
Figure 27: Irrection-Diffusion system: Predictions at t 0, 20, 80 and 160 obtained with the deterministic Koopman based model of Equation 33
030
0.21
030
Figure 28: Burgers' system: Predictions at 0, 20 and 80 obtained with the deterministic Koopman-baad based model Equation 33 )
25


Reviews:
[{"review_id": "ny1lA_xqjMF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a new dimensional reduction scheme, based on a latent representation, using a generative model and variational inference.\n\nThe method is successfully applied to 2 simple one-dimensional physical processes, for which the macroscopic equations are known from the physics literature.\n\nChoosing only a couple of simple physical variables X (as far as I understand, here it's the density in binned areas of the domain), and an arbitrary number (5) of latent variables (here we see that 5 is enough because it captures all the slow degrees of freedom), the model is able to learn the slow-variables values (eigenvalues controlling the evolution of the macroscopic variables) and then correctly reproduce the long-time behavior of the macroscopic observables (the X), and additionally produce realistic noise (x) / realistic microscopic fluctuations (this last part is not very clear, I am guessing here).\nIn addition, the latent representation of previously unseen initial conditions can be inferred (with decent error bars), allowing to predict the future of unseen initial trajectories.\n\n\nOverall, the idea is interesting and supported by correct mathematical derivations and experimental proofs of concept.\n\nThus I lean on accepting the paper.\n\n\nHowever, I have some questions and remarks that I would like to be addressed.\n\n\nMy main criticism would be the following: the paper does not use any pre-existing benchmarks, which makes accuracy comparisons essentially impossible.\nOf course one may object that the novelty of the approach makes comparison impossible, however I think that it should be possible to re-implement State of the art methods (by the way I appreciated the literature review section), and show how much the new method deals well with smallish data sets (which I believe it does ?).\n\nMy second main criticism is that although the method is general and in principle applicable to complex problems, here it is applied to textbook cases, for which the exact macroscopic equations are known, and for which a mechanically stable state is reached in finite time, in other words, two very simple problems. (although if I recall well the equations (26), (27) do not allow a simple diagonal form in the style of (21)).\nThe simplicity of the problems attacked is seen also in the important (but hidden) remark, that equation (25) is sufficient to produce x, and that \"no parameters need to be learned for p(x_t|X_t)\".\nWhat would happen for more complicated problems ? (e.g. protein folding, even in the case of small molecules like butane (C2H6 if I recall is a simple standard)? )\nIn particular, if the true number of \"independent\" components (number of lambda's) needed was very large ?\n\n\nAlso I have a trivial question, that however I think needs to be discussed in the paper. What is the error on the prediction of the microscopic positions of the {x} variables (the elementary particles) ? My guess is that by construction, the generative model has a 100% error (as good as random, I mean), because it does not track the future of individual particles, but rather predicts the future of the large-scale, slowly-evolving coarse-grained variables (X, governed by the latent z's).\nAm I correct ? I think it may be worth mentioning that quickly, for the readers that are unfamiliar with the topic.\nIf I am completely wrong, it is then even more worth mentioning the accuracy of x(t+P) predictions.\n\nI did not understand why section H of appendix was not part of the main text. It seems quite central to be able to forecast unseen initial conditions, and not just the future of already seen trajectories.\n\n\n\n\nLess important remarks about how the paper presents the research:\n\nsec 2.2 : additional intuitive explanation about how the learning is performed would be welcomed.\nIt is not clear to the unfamiliar reader as to how learning proceeds.\nIt seems to me that you directly learned the parameters of the joint probability distribution, which is quite factored, as you showed in Eq. (8), but needs Variational Inference methods to be solved for. What I don't see clearly is the shape of q_\\phi, or rather, what it means for B to be bi-diagonal. Maybe you could explain which variables are connected with which, given your assumptions on B_\\phi.\nAlso, after reading Appendix , Figure 9 and 11 could be compressed and included in the main text, or at least referred to explicitly.\n\nA number of graphs are un-readable, in particular the x- and y-axis labels are so tiny that one cannot get what is plotted against what.\nPlease correct that, increasing both figure size and x/y-labels sizes.\n\n\n-------\na couple of detailed remarks:\n\nIn Fig 13-16, the central plot does not have a colorbar. Is it because it's exactly the same as for the left plot ? If so, mention it or center the colorbar in a visually suggestive way\n\ntypo: \"is is\" (search it)\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nDAEBjmLAQI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a generative state-space model using two layers of latent variables. The latent variables in the first layer aim to capture long-term dynamics and ensures the stability. The variables in the second layer are physical variables. The authors have shown some promising results in modeling particle dynamics.  \n\nSince the authors claim that the use of X can reduce the complexity/the search space of the learning model, it would be great if the authors can show how the performance change given different number of training samples. \n\nAnother issue is about the effectiveness of this model in simulating real-world physical phenomena. This method can work well on simulated dataset because simulated data always follow the dependencies between X and x (as the simulator is built based on these rules). Real-world physical systems can be complex and usually we do not know exactly the governing physical variables (X). It would be great to discuss whether the proposed method allows some flexibility to automatically discover these unknown physical variables.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iQRRZKI1aW6", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents a generative approach to modeling physical systems with high-dimensional, nonlinear dynamical systems such as those found in fluid mechanics. The authors provide a physics-motivated hierarchical model for high-dimensional time series and a variational inference method for inferring latent variables and dynamical system parameters. They demonstrate its performance on simulated fluid mechanics prediction tasks.\n\nStrong points:\n\nThe paper presents an interesting and motivating case for Bayesian inference in probabilistic generative models: a problem that has inherent uncertainty along with the ability to incorporate domain knowledge that can reduce the inference complexity.\n\nThe results on the simulated tasks are encouraging, especially the results in Appendix H which demonstrate generalization to out-of-distribution data. The paper does an excellent job of presenting and analyzing the results, along with compelling visualizations.\n\nIn general, the paper is well written (apart from some higher-level structural issues discussed below) and the notation is clear and unambiguous. \n\nWeak points:\n\nStructurally, the paper could do a better job of motivating the particular choices made when designing the model. The choices that could be made more clearly motivated are:\n\n1) using a complex-valued latent dynamical system. Although there is a sentence in the “Related Work” section (“the prior proposed and the use of complex variables enable the discovery of slow features”), this important choice could use additional discussion and motivation.\n\n2) On first read, it was unclear what the likelihood model was, especially when it was stated that the observed data were time-series of (e.g.) particle velocities and positions. In Appendix E, it is stated that a Multinomial observation model is used, which could not be used for particle velocities/positions since it models counts of discrete variables. Thus, the observations $x_t$ in this case are statistics of velocities/positions (i.e. counts of particles in discrete buckets). This choice should be discussed in the main text, as it both changes the dataset size and dimensionality, along with the goal of the model itself. Rather than modeling particle positions and velocities, it seems like the model is generating statistics of the system. While this isn’t inherently problematic, this choice is not clearly indicated in the text.\n\n3) Finally, the paper includes $z_t$ in the model to help make predictions converge to equilibrium. While intuitively, having $z_t$ decay according to the model corresponds to some notion of “stability”, the paper should also explain how the exclusion of $z_t$ can lead to unstable or diverging predictions. Although the paper cites related work to justify the choice, it is such an important choice that I think it merits more discussion in the main text.\n\nThese structural issues with the paper are worsened by the fact that there are no ablations or comparisons in the paper. I think the minimal set of additional experiments would include 1) a model with real-valued (not complex) latent dynamics and 2) a model without $z$ in it, to demonstrate the necessity of a stabilizing element in the model. Even better would be comparisons to non-probabilistic models, demonstrating the necessity of uncertainty in the face of “information loss”. Finally, comparisons to related methods (e.g. Koopman-operator baselines) would help understand how the proposed model improves on the surrounding literature.\n\nRecommendation:\n\nWhile I find the problem setting and proposed approach very interesting, the writing of the paper and the results still need work. The paper needs to better justify the modeling choices both in writing and with additional experiments. The provided results are encouraging, but I think a proper evaluation of the approach with the appropriate ablations and baselines is necessary for publication. I recommend a reject on these grounds.\n\nClarifying questions:\n\nAre predictions obtained using MAP estimates of the parameters $\\theta$ or by sampling the posterior predictive distribution (i.e. marginalizing out $\\theta$)?\nIn what way is your inference approach “hybrid” as opposed to fully Bayesian?\n\nAdditional feedback:\n\nDefining terms like “fine-grained” and “multiscale” would help make the paper more accessible to readers without physics backgrounds\nNit: “parametrized” -> “parameterized”\nPage 4: “algrithm” -> “algorithm”\n\n\n===========================================\nUpdates:\n\nAfter considering the author's response and updates to the paper, I have bumped the score to a 6. The addition of Appendix H, in my opinion, considerably strengthens the paper's story and case for acceptance. I still have minor concerns about the writing surrounding the use of the Multinomial likelihood - for example, the paper still claims to be dealing with high-dimensional data, but bucketing into 25 buckets immediately reduces the complexity of the observation space to 25-dimensional counts. However, the authors have addressed most of my major concerns.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gZ11_l6TwB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a generative model for learning a low-dimensional representation of a dynamical system from high-dimensional observations. The novelty of the approach is to introduce two latent spaces, one representing the standard physics-agnostic latent space learned from the data and one representing physics-motivated variables. The goal is to learn the dynamics of the first layer, denoted by z_t, which is a coarse-grained representation of the dynamical system and mostly captures the slow processes that drive the system. \n\nQuality, clarity, originality and significance:\nPro: The paper is very well written, clear, and the generative approach is novel. Adding domain knowledge is relevant and significant when dealing with real world applications. Cons: Below are a few comments and questions to clarify some of the aspects and choices made. \n\nHow are the physics-motivated variables chosen? Is there a guarantee that the chosen representation/variables are sufficient to allow a one-on-one mapping between the physics-based latent space $X_t$ and the observations/data $x_t$? (As a small comment, I found it slightly confusing to use the same letter for the latent space and data space, even if one is lower case and the other upper case.) \n\nWhat does equilibrium represent here? Does it necessarily need to be a stationary process as in the experiments, or can it be a trend, or a periodic signal? Are the latter two cases handled well by the method?\n\nFig. 1: Would it be possible to add the maps F and G to the figure? It is not very clear to me how the latent variables X are obtained, are they computed from the data? If X is the generative process for x, why do we need the latent space z? Would it be possible that the latent space X is sufficient, and if not, would the authors have examples when this is not the case?\n\nThe framework is applied to physics simulations. How easy/difficult would it be to apply the method to real world data, and what would be the main challenges?\n\nSect. 2: Should it be $x_t \\in \\mathcal{M}$, instead of $x \\in \\mathcal{M}$? The paper mentions that the $z_t$’s correspond to a nonlinear coordinate transformation, but do not specify of what?\n\nWhat is the motivation for using the Ornstein-Uhlenbeck process for z, and what would be other alternatives? Why exactly does z need to be complex, is it in order to be able to capture slow and fast processes? In Figs. 2 and 5, the slow processes seem to have imaginary part close to zero, and fast processes real part close to zero. What is the explanation for this (this might be a known fact in dynamical systems theory, but not so much in ML)? Does it always have to be the case that the \\lambda’s have either the real or imaginary parts close to zero? What would happen if the system is still multi-scale but on a much more continuous scale than in the experiments presented here and where there wouldn’t be such a clear distinction between slow/intermediate/fast scales (this would probably be the case in complex real world systems)? \n\nAccording to eq (4) to ensure long-term stability, the real part of $\\lambda_j$ should be\nnegative. This only applies to the slow processes, is that right? What is an explanation for this? \n\nIn the definition for $\\sigma_j$ there is a 2 factor which is not in eq (7). Same line: the latent dynamics are stationary refers to which layer, $z_t$ or $X_t$? Using clearly defined names for the two latent spaces would make the paper easier to read. Maybe even add this to Fig. 1.\n\nIn ML we don’t see often complex distribution, and it would be good to specify that $\\mathcal{CN}$ is the complex distribution.\n\nEq (1): as $\\lambda$ and $\\sigma$ do not depend on time, it means that all hidden processes $z_t$ follow the same dynamical model. What are the implications of this, and would this be a reasonable choice in a real world scenario?\n\nHow robust are the results to the choice of the number of latent variables h, and the choice of the physics-based variables?\n\n--------------------------\nRebuttal: Thank you to the authors for their detailed response. I am happy with the response and will keep the original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-gbBGE2uvXi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a good paper, in my opinion. I have some suggestion to improve the quality of the work.\nSee below.\n\n - I suggest to improve Section 3, open a bit the range of references,  considering general and relevant works such as\n \nC. Grigo et al. A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime\narXiv:1902.03968, 2019.\n\nG. Camps-Valls, et al. \"Physics-Aware Gaussian Processes in Remote Sensing\", Applied Soft Computing, Volume 28, Pages: 69-82, 2018.\n\nSungyong Seo et al. \"Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics\",  ICLR 2020,\n\n- Explain better figure 2, improving its caption.\n\n- Figure 3 is completely unclear, remove it or improve.\n\n- Please explain better your system in Eqs (1)-(2)-(3). What are the measurements/observations? what is your inference goal? please clearer state these points.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The consensus of the reviews is to accept the paper. I agree.

Reviewers highlighted many strengths, including a compelling main idea:
* R5: "The paper presents an interesting and motivating case for Bayesian inference in probabilistic generative models: a problem that has inherent uncertainty along with the ability to incorporate domain knowledge that can reduce the inference complexity."
* R3: "Overall, the idea is interesting and supported by correct mathematical derivations and experimental proofs of concept."
* R4: "the generative approach is novel. Adding domain knowledge is relevant and significant when dealing with real world applications"

As well as compelling experiments, substantially improved in the discussion period:
* R1: "The authors have shown some promising results in modeling particle dynamics."
* R5: "The addition of Appendix H, in my opinion, considerably strengthens the paper's story and case for acceptance. [... T]he authors have addressed most of my major concerns."

And clear writing:
* R5: "In general, the paper is well written (apart from some higher-level structural issues discussed below) and the notation is clear and unambiguous."
* R4: "The paper is very well written, clear"

The main weaknesses highlighted were in experiments (lacking good baselines, as well as ablations), and in discussing some choices in the model's construction. These were effectively addressed in the discussion (though R5 still points to some places that could be improved).
Decision: Accept (Poster)