On the Stochastic Stability of Deep Markov Models

1 Introduction . Modeling , analysis , and control of dynamical systems are of utmost importance for various physical and engineered systems such as fluid dynamics , oscillators , power grids , transportation networks , and autonomous driving , to name just a few . The systems are generally subjected to uncertainties arising from a plethora of factors such as exogenous noise , plant-model mismatch , and unmodeled system dynamics , which have led researchers to model the dynamics in stochastic frameworks . One of the most commonly used probabilistic frameworks to model dynamical system is the Hidden Markov Models ( HMMs ) [ Rabiner and Juang , 1986 , Eddy , 1996 ] which in their vanilla form have been extensively investigated for representation , learning , and inference problems [ Ghahramani and Jordan , 1997 , Cappé et al. , 2006 , Beal et al. , 2002 ] . One of its variants , the Gaussian state space models , have been used in the systems and control community for decades [ Beckers and Hirche , 2016 , Eleftheriadis et al. , 2017 ] . It has been shown that expressivity of Markov models to emulate complex dynamics and sequential behaviors is greatly improved by parametrizing such models using deep neural networks , giving rise to Deep Markov Models ( DMMs ) [ Krishnan et al. , 2017 ] . Main research activities have been focused on the inference of these models . In particular , works such as Awiszus and Rosenhahn [ 2018 ] , Liu et al . [ 2019 ] , Mustafa et al . [ 2019 ] , Qu et al . [ 2019 ] proposed parametrizing the probability distributions using deep neural networks for modeling various complex dynamical systems . Despite the rising popularity of DMMs , many of their theoretical properties such as robustness to perturbations , and stability of the generated trajectories remain open research questions . Many natural systems exhibit complex yet stable dynamical behavior that is described by converging trajectories towards an attractor set [ Brayton and Tong , 1979 ] . Additionally , safety-critical systems such as an autonomous 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . driving call for formal verification methods to ensure safe operation . Thus the ability to guarantee stability in data-driven models could lead to improved generalization or act as a safety certificate in real-world applications . In this paper , we propose a new analytical method to assess stochastic stability of DMMs . More specifically , we utilize spectral analysis of deep neural networks ( DNNs ) modeling DMM ’ s distributions . This allows us to make connections between the stability of deterministic DNNs and the stochastic stability of deep Markov models . As a main theoretical contribution we provide sufficient conditions for stochastic stability of DMMs . Based on the proposed theory we introduce several practical methods for design of constrained DMM with stability guarantees . In summary , the main contributions of this paper include : 1 . Stability analysis method for deep Markov models : we base the analysis on the operator norms of deep neural networks modeling mean and variance of the DMM ’ s distributions . 2 . Sufficient conditions for stochastic stability of deep Markov models : we show the sufficiency of the operator norm-based contraction conditions for DMM ’ s deep neural networks by leveraging Banach fixed point theorem . 3 . Stability constrained deep Markov models : we introduce a set of methods for the design of provably stable deep Markov models . 4 . Numerical case studies : we analyze connections between the design parameters of neural networks , stochastic stability , and operator norms of deep Markov models . 2 Related Work . Deep Markov models ( DMMs ) have been used as a scalable and expressive generalization of Hidden Markov Models ( HMM ) for learning probabilistic generative models of complex high-dimensional dynamical systems from sequential data [ Rezende et al. , 2014 , Krishnan et al. , 2017 , Fraccaro et al. , 2016 ] . DMMs have been successfully applied to speech recognition problems [ Li et al. , 2013 , Prasetio et al . ] , control problems [ Shashua and Mannor , 2017 ] , human pose forecasting [ Toyer et al. , 2017 ] , fault detection [ Wang et al. , 2018 ] , climate data forecasting [ Che et al. , 2018 ] , molecular dynamics [ Wu et al. , 2018 ] , or as internal models in model-based deep RL applied to automatic trading [ Ferreira , 2020 ] . Several modifications of DMMs have been proposed to handle incomplete data [ Tan et al. , 2019 ] , or multi-rate time series [ Che et al. , 2018 ] multivariate time series [ Montanez et al. , 2015 ] , training DMMs in unsupervised settings [ Tran et al. , 2016 ] , or architectures inspired by Kalman Filters [ Krishnan et al. , 2015 , Shashua and Mannor , 2017 , Becker et al. , 2019 ] . However , works focusing on formal analysis to ensure stability guarantees for DMMs are missing . Stability notions and analysis for stochastic dynamic systems have been studied in the automatic control literature in various forms , depending on the representation of the system dynamics . Some classical results on stochastic stability for analysis and control can be found in McLane [ 1971 ] , Willems and Willems [ 1976 ] . These results are presented mainly for stochastic differential equation ( SDE ) models . Khasminskii [ 2012 ] discusses different notions of stochastic stability , among them mean-square based stability notions have gained interest in works such as Lu and Skelton [ 2002 ] , Farmer et al . [ 2009 ] , Elia et al . [ 2013 ] , Nandanoori et al . [ 2018 ] , Wu et al . [ 2019 ] . We resort to such mean-square based stability notions when analyzing probabilistic state transition models parametrized by deep neural networks . We show that when the stochastic transitions are modeled by DNNs , the probabilistic stability requirements can be translated to deterministic stability notions of nonlinear discrete-time dynamics [ Khalil , 2002 ] . In recent years , deep neural networks have been extensively studied from the viewpoint of dynamical systems [ Chen et al. , 2018 , Raissi et al. , 2017 , Ciccone et al. , 2018 ] , allowing for the application of stability analysis methods to DNNs . For instance , Manek and Kolter [ 2019 ] proposed neural Lyapunov function to stabilize learned neural dynamics of autonomous system . Haber and Ruthotto [ 2017 ] interpret residual connections in neural networks as Euler discretization of ODEs and provide stability guarantees of ResNet . Goel and Klivans [ 2017 ] makes connections between eigenvalue decay and learnability of neural networks . Engelken et al . [ 2020 ] , Vogt et al . [ 2020 ] studies the Lyapunov spectrum to of the input-output Jacobian of recurrent neural networks ( RNNs ) to assess RNN ’ s stability . In this work we leverage advances in the spectral analysis of deep neural networks and apply them to derive stochastic stability guarantees for DMMs . Besides analytical methods , many authors introduced provably stable neural architectures [ Haber et al. , 2019 , Greydanus et al. , 2019 , Cranmer et al. , 2020 ] or stability constraints [ John et al. , 2017 ] . Another popular strategy is to employ stabilizing regularizations . This can be achieved by minimizing eigenvalues of squared weights [ Ludwig et al. , 2014 ] , using symplectic weights [ Haber and Ruthotto , 2017 ] , orthogonal parametrizations [ Mhammedi et al. , 2017 ] , Perron-Frobenius theorem [ Tuor et al. , 2020 ] , Gershgorin dics theorem [ Lechner et al. , 2020 ] , or via singular value decomposition ( SVD ) [ Zhang et al. , 2018 ] . In this paper we leverage different weight factorization to empirically validate the proposed theoretical guarantees on stochastic stability of DMMs . 3 Methodology . This section presents stochastic stability analysis method for deep Markov models ( DMM ) . First , we demonstrate the equivalence of DNNs with pointwise affine ( PWA ) functions . Next , we recall the definition of DMM with transition probabilities modeled by deep neural networks ( DNNs ) . We introduce definitions of stochastic stability and show how can we leverage deterministic stability analysis in the probabilistic context . Finally , we will leverage the equivalence of DNNs with PWA maps to pose sufficient stability stochastic conditions for DMMs based on contraction of PWA maps . 3.1 Equivalence of Deep Neural Networks with Pointwise Affine Maps . Let us consider deep neural network ( DNN ) ψ : Rn → Rm parametrized by θψ = { Aψ0 , . . .A ψ L , b0 , . . . , bL } with hidden layers 1 ≤ l ≤ L with bias given as follows : ψθψ ( x ) = A ψ Lh ψ L + bL ( 1a ) hψl = v ( A ψ l−1h ψ l−1 + bl−1 ) ( 1b ) with hψ0 = x , and v : Rnz → Rnz representing element-wise application of an activation function to vector elements such that v ( z ) : = [ v ( z1 ) . . .v ( znz ) ] ᵀ. Lemma 1 . For a multi-layer feedforward neural networkψθψ ( 3.2 ) with arbitrary activation function v , there exists an equivalent pointwise affine map ( PWA ) parametrized by x which satisfies : ψθψ ( x ) = Aψ ( x ) x + bψ ( x ) . ( 2 ) Where Aψ ( x ) is a parameter varying matrix given as : Aψ ( x ) x = A ψ LΛ ψ zL−1A ψ L−1 . . .Λ ψ z0A ψ 0 x ( 3 ) And bψ ( x ) is a parameter varying vector , both parametrized by input vector x given by following recurrent formula : bψ ( x ) = b ψ L ( 4 ) bψl : = A ψ i Λ ψ zl−1 bψl−1 + A ψ i σl−1 ( 0 ) + bl , l ∈ N L 1 ( 5 ) with bψ0 = b0 , and i representing index of the network layer . Here Λ ψ zl represents parameter varying diagonal matrix of activation patterns defined as : σ ( z ) = σ ( z1 ) −σ ( 0 ) z1 . . . σ ( zn ) −σ ( 0 ) zn z + σ ( 0 ) ... σ ( 0 ) = Λψz z + σ ( 0 ) ( 6 ) Proof . First lets observe the following : σ ( z ) = σ ( z1 ) ... σ ( zn ) = z1 ( σ ( z1 ) −σ ( 0 ) +σ ( 0 ) ) z1 ... zn ( σ ( zn ) −σ ( 0 ) +σ ( 0 ) ) zn = σ ( z1 ) −σ ( 0 ) z1 . . . σ ( zn ) −σ ( 0 ) zn z + σ ( 0 ) ... σ ( 0 ) ( 7 ) Remember σ ( 0 ) − σ ( 0 ) = 0 , and zizi = 1 are identity elements of addition and multiplication , respectively . Thus ( 7 ) demonstrates the equivalence σ ( z ) = Λψz z +σ ( 0 ) as given in ( 6 ) . Then if we let zl = A ψ l xl + bl , we can represent a neural network layer in a parameter varying affine form : σl ( A ψ l xl + bl ) = Λ ψ zl ( Aψl xl + bl ) + σ ( 0 ) = Λ ψ zl Aψl xl + Λ ψ zl bl + σl ( 0 ) ( 8 ) Now for simplicity of exposition lets assume only activations with trivial null space , i.e . σ ( 0 ) = 0 . Thus σ ( z ) = Λψz z . By composition , a DNN ψθψ ( x ) can now be formulated as a parameter-varying affine map Aψ ( x ) x + bψ ( x ) , parametrized by input x ψθψ ( x ) : = Aψ ( x ) x + bψ ( x ) = AψLΛ ψ zL−1 ( A ψ L−1 ( . . .Λ ψ z1 ( A ψ 1 Λ ψ z0 ( A ψ 0 x + b0 ) + b1 ) . . . ) + bL−1 ) x + bL Aψ ( x ) x = A ψ LΛ ψ zL−1A ψ L−1 . . .Λ ψ z0A ψ 0 x bψ ( x ) = A ψ L . . .A ψ 2 Λ ψ z1A ψ 1 Λ ψ z0b0 + A ψ L . . .A ψ 2 Λ ψ z1A ψ 1 σ0 ( 0 ) + AψL . . .A ψ 2 Λ ψ z1b1 + A ψ L . . .A ψ 2 σ1 ( 0 ) + . . .+ A ψ LΛ ψ zL−1bL−1 + A ψ LσL−1 ( 0 ) + bL ( 9 ) Hence , each input feature vector x generates a unique affine map Aψ ( x ) x + bψ ( x ) of the DNN ψθψ ( x ) . Thus proving the equivalence of DNN map ( 3.2 ) with the form ( 2 ) . The case with σ ( 0 ) 6= 0 can be derived following the same algebraic operations as as above .

Summary: The paper discusses stable deep Markov models (DMM), which can be used to model stochastic dynamical systems, restricting to the case where state transition probabilities are given by normal distributions with mean and covariance matrices parameterized by two deep neural nets. One of the main contributions of this paper is to study sufficient conditions under which such DMMs are stable (in the mean-square sense), the second is to provide a list of practical (re)parameterization of the DNNs that satisfy the sufficient conditions.  The paper concludes with a series of small scale numerical experiments aimed at showing the possible behaviours of the class of proposed models.