Physics-aware, probabilistic model order reduction with guaranteed stability

Given ( small amounts of ) time-series ’ data from a high-dimensional , fine-grained , multiscale dynamical system , we propose a generative framework for learning an effective , lower-dimensional , coarse-grained dynamical model that is predictive of the fine-grained system ’ s long-term evolution but also of its behavior under different initial conditions . We target fine-grained models as they arise in physical applications ( e.g . molecular dynamics , agent-based models ) , the dynamics of which are strongly non-stationary but their transition to equilibrium is governed by unknown slow processes which are largely inaccessible by brute-force simulations . Approaches based on domain knowledge heavily rely on physical insight in identifying temporally slow features and fail to enforce the long-term stability of the learned dynamics . On the other hand , purely statistical frameworks lack interpretability and rely on large amounts of expensive simulation data ( long and multiple trajectories ) as they can not infuse domain knowledge . The generative framework proposed achieves the aforementioned desiderata by employing a flexible prior on the complex plane for the latent , slow processes , and an intermediate layer of physics-motivated latent variables that reduces reliance on data and imbues inductive bias . In contrast to existing schemes , it does not require the a priori definition of projection operators or encoders and addresses simultaneously the tasks of dimensionality reduction and model estimation . We demonstrate its efficacy and accuracy in multiscale physical systems of particle dynamics where probabilistic , long-term predictions of phenomena not contained in the training data are produced . 1 INTRODUCTION . High-dimensional , nonlinear systems are ubiquitous in engineering and computational physics . Their nature is in general multi-scale1 . E.g . in materials , defects and cracks occur on scales of millimeters to centimeters whereas the atomic processes responsible for such defects take place at much finer scales ( Belytschko & Song , 2010 ) . Local oscillations due to bonded interactions of atoms ( Smit , 1996 ) take place at time scales of femtoseconds ( 10−15s ) , whereas protein folding processes which can be relevant for e.g . drug discovery happen at time scales larger than milliseconds ( 10−3s ) . In Fluid Mechanics , turbulence phenomena are characterized by fine-scale spatiotemporal fluctuations which affect the coarse-scale response ( Laizet & Vassilicos , 2009 ) . In all of these cases , macroscopic observables are the result of microscopic phenomena and a better understanding of the interactions between the different scales would be highly beneficial for predicting the system ’ s evolution ( Givon et al. , 2004 ) . The identification of the different scales , their dynamics and connections however is a non-trivial task and is challenging from the perspective of statistical as well as physical modeling . 1With the term multiscale we refer to systems whose behavior arises from the synergy of two or more processes occurring at different ( spatio ) temporal scales . Very often these processes involve different physical descriptions and models ( i.e . they are also multi-physics ) . We refer to the description/model at the finer scale as fine-grained and to the description/model at the coarser scale as coarse-grained . In this paper we propose a novel physics-aware , probabilistic model order reduction framework with guaranteed stability that combines recent advances in statistical learning with a hierarchical architecture that promotes the discovery of interpretable , low-dimensional representations . We employ a generative state-space model with two layers of latent variables . The first describes the latent dynamics using a novel prior on the complex plane that guarantees stability and yields a clear distinction between fast and slow processes , the latter being responsible for the system ’ s long-term evolution . The second layer involves physically-motivated latent variables which infuse inductive bias , enable connections with the very high-dimensional observables and reduce the data requirements for training . The probabilistic formulation adopted enables the quantification of a crucial , and often neglected , component in any model compression process , i.e . the predictive uncertainty due to information loss . We finally want to emphasize that the problems of interest are Small Data ones due to the computational expense of the physical simulators . Hence the number of time-steps as well as the number of time-series used for training is small as compared to the dimension of the system and to the time-horizon over which predictions are sought . 2 PHYSICS-AWARE , PROBABILISTIC MODEL ORDER REDUCTION . Our data consists of N times-series { x ( i ) 0 : T } Ni=1 over T time-steps generated by a computational physics simulator . This can represent positions and velocities of each particle in a fluid or those of atoms in molecular dynamics . Their dimension is generally very high i.e . xt ∈M ⊂ Rf ( f > > 1 ) . In the context of state-space models , the goal is to find a lower-dimensional set of collective variables or latent generators zt and their associated dynamics . Given the difficulties associated with these tasks and the solutions that have been proposed in statistics and computational physics literature , we advocate the use of an intermediate layer of physically-motivated , lower-dimensional variables Xt ( e.g . density or velocity fields ) , the meaning of which will become precise in the next sections . These variables provide a coarse-grained description of the high-dimensional observables and imbue interpretability in the learned dynamics . Using Xt alone ( without zt ) would make it extremely difficult to enforce long-term stability ( see Appendix H.2 ) while ensuring sufficient complexity in the learned dynamics ( Felsberger & Koutsourelakis , 2019 ; Champion et al. , 2019 ) . Furthermore and even if the dynamics of xt are first-order Markovian , this is not necessarily the case forXt ( Chorin & Stinis , 2007 ) . The latent variables zt therefore effectively correspond to a nonlinear coordinate transformation that yields not only Markovian but also stable dynamics ( Gin et al. , 2019 ) . The general framework is summarized in Figure 1 and we provide details in the next section . 2.1 MODEL STRUCTURE . Our model consists of three levels . At the first level , we have the latent variables zt which are connected with Xt in the second layer through a probabilistic map G. The physical variables Xt are finally connected to the high-dimensional observables through another probabilistic map F . We parametrize F , G with deep neural networks and denote by θ1 and θ2 the corresponding parameters ( see Appendix D ) . In particular , we postulate the following relations : zt , j = zt−1 , j exp ( λj ) + σj t , j λj ∈ C , t , j ∼ CN ( 0 , 1 ) , j = 1 , 2 , . . . , h ( 1 ) Xt = G ( zt , θ1 ) ( 2 ) xt = F ( Xt , θ2 ) ( 3 ) We assume that the latent variables zt are complex-valued and a priori independent . Complex variables were chosen as their evolution includes a harmonic components which are observed in many physical systems . In Appendix H.1 we present results with a real-valued latent variables zt , j and illustrate their limitations . We model their dynamics with a discretized Ornstein-Uhlenbeck process on the complex plane with initial conditions z0 , j ∼ CN ( 0 , σ20 , j ) 2 . The parameters associated with this level are denoted summarily by θ0 = { σ20 , j , σ2j , λj } hj=1 . These , along with θ1 , θ2 mentioned earlier , and the state variables Xt and zt have to be inferred from the data xt . We explain each of the aforementioned components in the sequel . 2.1.1 STABLE LOW-DIMENSIONAL DYNAMICS . While the physical systems ( e.g . particle dynamics ) of interest are highly non-stationary , they generally converge to equilibrium in the long-term . We enforce long-term stability here by ensuring that the real-part of the λj ’ s in Equation ( 1 ) is negative , i.e . : λj = < ( λj ) + i = ( λj ) with < ( λj ) < 0 ( 4 ) which guarantees first and second-order stability i.e . the mean as well as the variance are bounded at all time steps . The transition density each process zt , j is given by : p ( zt , j | zt−1 , j ) = N ( [ < ( zt , j ) = ( zt , j ) ] | sj Rj [ < ( zt−1 , j ) = ( zt−1 , j ) ] , I σ2j 2 ) ( 5 ) where the orthogonal matrixRj depends on the imaginary part of λj : Rj = [ cos ( = ( λj ) ) − sin ( = ( λj ) ) sin ( = ( λj ) ) cos ( = ( λj ) ) ] ( 6 ) and the decay rate sj depends on the real part of λj : sj = exp ( < ( λj ) ) ( 7 ) i.e . the closer to zero the latter is , the ” slower ” the evolution of the corresponding process is . As in probabilistic Slow Feature Analysis ( SFA ) ( Turner & Sahani , 2007 ; Zafeiriou et al. , 2015 ) , we set σ2j = 1− exp ( 2 < ( λj ) ) = 1− s2j and σ20 , j = 1 . As a consequence , a priori , the latent dynamics are stationary3 and an ordering of the processes zt , j is possible on the basis of < ( λj ) . Hence the only independent parameters are the λj , the imaginary part of which can account for periodic effects in the latent dynamics ( see Appendix B ) . The joint density of zt can finally be expressed as : p ( z0 : T ) = h∏ j=1 ( T∏ t=1 p ( zt , j | zt−1 , j , θ0 ) p ( z0 , j |θ0 ) ) ( 8 ) The transition density between states at non-neighbouring time-instants is also available analytically and is useful for training on longer trajectories or in cases of missing data . Details can be found in Appendix B . 2A short review of complex normal distributions , denoted by CN , can be found in Appendix A . 3More details can be found in Appendix B . 2.1.2 PROBABILISTIC GENERATIVE MAPPING . We employ fully probabilistic maps between the different layers which involve two conditional densities based on Equations ( 2 ) and ( 3 ) , i.e . : p ( xt |Xt , θ2 ) and p ( Xt | zt , θ1 ) ( 9 ) In contrast to the majority of physics-motivated papers ( Chorin & Stinis , 2007 ; Champion et al. , 2019 ) as well as those based on transfer-operators Klus et al . ( 2018 ) , we note that the generative structure adopted does not require the prescription of a restriction operator ( or encoder ) and the reduced variables need not be selected a priori but rather are adapted to best reconstruct the observables . The splitting of the generative mapping into two parts through the introduction of the intermediate variables Xt has several advantages . Firstly , known physical dependencies between the data x and the physical variables X can be taken into account , which reduces the complexity of the associated maps and the total number of parameters . For instance , in the case of particle simulations where X represents a density or velocity field , i.e . it provides a coarsened or averaged description of the fine-scale observables , it can be used to ( probabilistically ) reconstruct the positions or velocities of the particles . This physical information can be used to compensate for the lack of data when only few training sequences are available ( Small data ) and can seen as a strong prior to the model order reduction framework . Due to the lower dimension of associated variables , the generative map between zt and Xt can be more easily learned even with few training samples . Lastly , the inferred physical variablesX can provide insight and interpretability to the analysis of the physical system .

Summary: The paper proposes a generative model for learning a low-dimensional representation of a dynamical system from high-dimensional observations. The novelty of the approach is to introduce two latent spaces, one representing the standard physics-agnostic latent space learned from the data and one representing physics-motivated variables. The goal is to learn the dynamics of the first layer, denoted by z_t, which is a coarse-grained representation of the dynamical system and mostly captures the slow processes that drive the system.