On the Stochastic Stability of Deep Markov Models

Abstract: Deep Markov models (DMM) are generative models which are scalable and expressive generalization of Markov models for representation, learning, and inference problems. However, the fundamental stochastic stability guarantees of such models have not been thoroughly investigated. In this paper, we present a novel stability analysis method and provide sufficient conditions of DMM's stochastic stability.  The proposed stability analysis is based on the contraction of probabilistic maps modeled by deep neural networks. We make connections between the spectral properties of neural network's weights and different types of used activation function on the stability and overall dynamic behavior of DMMs with Gaussian distributions. Based on the theory, we propose a few practical methods for designing constrained DMMs with guaranteed stability. We empirically substantiate our theoretical results via intuitive numerical experiments using the proposed stability constraints.


Reviews and author responses:

The paper discusses stable deep Markov models (DMM), which can be used to model stochastic dynamical systems, restricting to the case where state transition probabilities are given by normal distributions with mean and covariance matrices parameterized by two deep neural nets.
One of the main contributions of this paper is to study sufficient conditions under which such DMMs are stable (in the mean-square sense), the second is to provide a list of practical (re)parameterization of the DNNs that satisfy the sufficient conditions. 
The paper concludes with a series of small scale numerical experiments aimed at showing the possible behaviours of the class of proposed models.
  Clarity:
I found the paper very well written (found only one typo in Eq (7), missing t=0) and generally easy to follow.  
The work is well motivated; I appreciated that the proofs of the main results are in the paper, and  I found section 3.4., particularly informative and well organized, linking theoretical findings to practical design principles.
A few comments regarding clarity:
- In the proof of lemma 1, in eq (4), it is not clear whether the bias terms are included or not, especially since in line 95 there is written $z_{l+1} = A_l h_l$ whereas I would have expected to see also the bias in $z$.
- In theorems 4 and 5, since the statements are about mean-square stability, perhaps it would be better to state in the hypotheses that K<1 and $c^A c^{\Lambda}<1$ and then in the proof (or in a remark after those) relax these conditions referring to eq (11).
- (Question) It is not clear to me how the parametric stability constraints (also used in sec 4.3) relate to the hypotheses of the theorems. 
- The experimental section is the only section that I found rather difficult to follow, this is because: (i) there is no description, however short, of the experimental settings, and (ii) I think the discussion could be a bit more focused, better accompanying the reader on what the should look at. For (i), I would recommend making a bit of space for short descriptions of the setting in the main paper.

Originality: 
I find the contributions of this work overall novel, although not surprising, as they are in a way adaptations of known facts that have been studied in other circumstances (as per related section and also see added references below, which could be added to the related work). I think however the results are not obvious, and hence they deserve publication. In section 3.4, the paragraph which I found most original (and interesting) is the last one, which probably deserves some more discussion. 

Significance:
I believe the theoretical results and especially the design principles will be useful in the ML community and beyond; this paper may become a handy reference for DMM designers.

Quality:
The mathematical treatment is clear and precise, and the results look correct to me. 
The experimental section could be improved, by describing the experimental settings (as in the clarity section) and by including some quantitative comparative results about optimizing with various parameterizations (ideally this should be addressed in the rebuttal).
Related literature is discussed well enough, but perhaps the authors could consider including also works with deterministic models e.g. [1] and [2] (in particular, in [2] there is a discussion about equilibrium models and stability thorough projections which could complement the discussion of section 4.3.)


References:
[1] Miller, J. and Hardt, M.  Stable recurrent models. ICLR, 2019.
[2] Grazzi, Riccardo, et al. "On the iteration complexity of hypergradient computation." International Conference on Machine Learning. PMLR, 2020.

____
Post-rebuttal

Thanks for your reply. I am keeping my score and recommend acceptance. Addressed 

This paper analyzes deep Markov models from a theoretical perspective. More specifically, it derives conditions on the construction/parameters of models which allow for a specific notion of stochastic stability. This is accomplished by first proving that a multilayer perceptron can be equivalently represented as a pointwise affine map (PWA). It is then proven that achieving specific properties on the norms of these PWAs leads to global stability, and that bounding the norms of the layers in neural networks leads to achieving these desired properties within the corresponding PWAs. There is a discussion on how one can achieve these properties when training a neural network, followed by empirical analysis demonstrating the connections between the properties of DMMs and their stability.
  ### Originality: 
I am unaware of prior work which analyzes DMMs in this manner. Connections to prior work are made clear.

### Quality: 
As far as I could tell, the analysis done in this paper is sound. The conclusions regarding the analysis, and suggestions are made/analyzed regarding how to construct/train networks that achieve the required properties. The required assumptions are made very clear, and the authors are open and honest about the limitations of the current analysis.

### Clarity: 
The paper is clearly written and straightforward to follow. All important properties are defined, and the proofs are organized in a logical order. The empirical analysis is well-visualized, and the points being made are conveyed well by the plots.

### Significance: 
Assessing the properties of deep neural networks and their use in a variety of modeling frameworks is an open field, and understanding the theoretical properties of networks and how to achieve desired configurations can help lead to better performing models. The analysis presented here is another step in this direction, and thus I imagine many will find it interesting.

### Miscellaneous Comments
Equation 7 needs to be rewritten slightly - for example, $P(\mathbf{y}_0|\mathbf{x}_0)$ appears twice on the right side the way it is currently written.

# POST-REBUTTAL UPDATE
Thanks for the responses. After reading other reviews and the responses to each of them, I am maintaining the original score I gave. I think this paper makes a valuable contribution and should be accepted.
 The limitations are presented clearly and thoroughly. I agree with the authors that this work is basic science and does not have any direct connections to societal impact.

 We want to thank the reviewer for their comments and overall positive assessment which is encouraging for continuing further studies of the presented topic. 

The Eq.~(7) will be updated accordingly. 

Furthermore, in the discussion, we will make conceptual connections with and assess the suitability of the proposed design methods for different classes of real-world dynamical systems. For instance, we will discuss the importance of guarantees of DMM convergence to single-point attractors for stochastic closed-loop control systems such as those considered in reinforcement learning settings. Additional conceptual applications include surrogate modeling with stability guarantees of systems such as the Wiener process, or Brownian motion, or stochastic differential equations, and stochastic time series modeling. 

 We thank the reviewer for their valuable suggestions and comments. The authors went thoroughly through this feedback which will be incorporated in the revised version of the manuscript.

In particular, the revised will include the “Threats to Validity” section analyzing the applicability to large-scale dynamical systems via scalability analysis of the proposed matrix factorizations and parametric constraints. We will evaluate the computational time of the forward pass of DMMs with different constraints as a function of an increasing number of hidden dimensions and network depth.
We will consider higher-dimensional systems with up to 100 dimensions. Additionally, we will include the study of the empirical spectrum of the DMM models as a function of the neural network's depth. 

Furthermore, we will elaborate on expressivity limitations associated with the proposed stability constraints. We will focus on single-point attractors of DMMs via contraction constraints, and attractor sets via parametric constraints. As an additional contribution, we will expand the theory and propose a design method for multi-point attractor DMMs by introducing parametric constraints on the equilibria by leveraging Corollary 12 in the appendix.

Finally, we will motivate the proposed constraints and suggest suitable applications, such as single-point attractors in closed-loop control for reference tracking and stabilization, or multi-point attractors of path-dependent trajectories present in economic or social dynamical systems, or phenomenon of multistable perception in visual systems.   

The literature review will be expanded as per the suggestion of the reviewer to include references studying the spectral properties of DMM models.

The revised version will be spell-checked and proofread before submission. 

 We thank the reviewer for the comments and suggestions. The authors went thoroughly through this valuable feedback which will be incorporated in the updated version of the manuscript. 

Answer to the comments:

1, Lemma 1 will be updated in the revised version to explicitly contain the bias terms for both the affine maps as well as the activations. We will leverage the fact that  activation functions can be expressed as parameter varying affine maps: $ \sigma(z) =  \frac{\sigma(z) -\sigma(0)}{z}z + \sigma(0) = A(z)z + b$, where $A(z) =\frac{\sigma(z) -\sigma(0)}{z} $, and $b = \sigma(0)$. 
Thus the aggregate dynamics of the deep neural network at a point can be expressed as a product of affine maps explicitly including bias terms. 

2, We will refactor the formal statements in theorems 4 and 5 according to the reviewer's suggestions. 

3, The motivation behind the parametric stability constraints is to allow the user to enforce the stability in selected regions (violations of conditions of Theorem 4 will be penalized) and relax these conditions in other regions of the domain (no enforcement of the conditions of Theorem 4). For instance, as shown in Figure 3, one could enforce the contraction in the outer regions of the domain, thus effectively preventing the trajectories to run away, while relaxing the inner regions to allow the neural networks to have the full expressive capacity and learn more complex behaviors than the single point attractor.  To make more theoretically sound connections of Theorem 4  with parametric constraints (in Eq.~(23)) we will include a new corollary defining conditions for attractor regions that are more generic than single point attractor. 

4, In the revised version we will include a compact description of the experimental setup with a full description in the appendix. 
We will also highlight the implication of the theoretical results. We will also provide a quantitative comparison of the computational cost associated with the proposed stability constraints as a function of the complexity of neural networks encoding mean and variance of the DMMs.

Originality: As per the reviewer's suggestion, in the revised version we will expand the parametric stability constraints discussion and supporting theory by adding a new corollary with conditions for stable attractors based on Eq.~(23). 

Quality: We thank the reviewer for the extra references, we will go through the suggested literature and expand our literature review and related works on deterministic models. 



This paper analyses stochastic stability of Deep Markov Models (DMMs) based on the contraction of probabilistic maps modeled by Deep Neural Networks (DNNs). To this end, DMMs are modeled using latent space representations of a dynamical system, and Gaussian distribution is used in modeling the transition and emission dynamics through DNN representation. With this modeling assumption, the authors perform stochastic stability analysis of DMM focused on bounding the norm of the model parameters of DNN. They state sufficient conditions for achieving stability and outline techniques for doing so. Finally, their claim is substantiated through small-scale experimentation of DMMs.  The paper analyses the stability of Deep Markov Models (DMMs) and provides sufficient conditions for stochastic stability of DMM. The authors perform mean-squared based stability analysis by using Deep Neural Networks (DNNs) to parameterize the probabilistic state transitions. In doing so, deterministic stability analysis is used in a probabilistic setting.

Their proof is based on modeling DMMs as a dynamical system with latent space variables and using the property that a dynamical system is said to be mean-square stable if its first and second moment converge with time. This leads to equivalent stability conditions outlined in Theorem 4 and Corollary 5 relying on contractive mapping of the DNN (modeling the DMM). These conditions are sufficient for the convergence of a DMM to a stable equilibrium. The proof is simple and naturally flows from existing literature dealing with probabilistic transitions in a dynamical system. 

In my opinion, the main crux of the paper lies in using the sufficient conditions (in Theorem 4 and Corollary 5) to outline the methods for designing stable DMMs by enforcing the contractivity of the weight matrices through different matrix factorizations. More importantly, the discussions on how some design considerations can have a negative impact on the expressivity of DMMs and can be ameliorated using parametric stability constraints have value in real-life applications. Going along with this, the numerical case studies in Section 4 shows a nice intuitive illustration of the ideas that have been theoretically proven earlier.

While the paper has potential in motivating design of (mean)-stable DMMs for real-life applications, there is little discussion connecting how much validity mean-stability analysis has in real-life applications. Also, there could have been more literature review and basics of spectral analysis of DNNs. This is not very clear at the moment. 

In spite of the strong points, my main concern about the paper remains about the applicability of the design choices (Section 3.4) with respect to large-scale dynamical systems. It would have been better to discuss the limitations with more reference to the literature and state the possible use cases where this kind of design might not be feasible.

Minor comments: The paper requires proofreading as there are typos, repeated abbreviations, and grammatical errors.
 While the authors have done a good job in outlining a good theoretical framework for their work, the practical implications are somewhat lacking. From a practical point of view, it would have been good if they would have shown how their findings could be applied to a DMM in real-life. My main concern is that in large-scale systems where it might be computationally challenging to apply the design principles that have been stated in the Section “Design of Stable Deep Markov Models”. It would have been better if there was a section titled “Threats to Validity” for discussing the above issue.

Meta-review: This paper analyzes the stability properties of _deep Markov models_ (DMMs), which are nonlinear dynamical systems that are parameterized by deep neural networks (basically, replacing the transition/emission functions of an HMM with neural nets). The form of stability used in this paper is _stochastic stability_, which essentially means that the first and second moments of a dynamical process asymptotically converge. (Note that this is different from the notion of _algorithmic stability_ used in learning theory.) The paper provides sufficient conditions under which a DMM is stochastically stable, and then analyzes how various architectural considerations (activation, weights, depth, etc.) affect stability. A numerical study rounds out the picture.

The reviews were unanimously positive, with 7s (Accept) across the board. The theory is sound; the work feels complete; and the paper is well written and easy to follow, with minor typos. None of the reviewers were all that confident in their assessments (perhaps this is a niche subject for the NeurIPS audience), but all agreed that it's definitely a solid paper.

In my opinion, the main weakness (echoed by Reviewer fyK8) is that not much discussion is devoted to motivating stability. Why is stochastic stability important in applications of dynamical systems? The introduction ignores this question and starts from the assumption that stability is important and, therefore, worth studying. As far as I can tell, the only discussion of why stability is important comes up at the end, in the Broader Impact statement, which only says, "Stability is the major concern of many safety-critical systems. ... [T]he methods presented in this paper have the potential for practical impact in many real-world applications such as unmanned autonomous vehicles, robotics, or process control applications." OK, but why? What would happen if an _unstable_ system were deployed in an autonomous vehicle? How specifically does stochastic stability ensure safety in an autonomous vehicle? I can kind of fill in the blanks myself, but I'd like the paper to be more explicit about how this highly theoretical study has practical impact.

One other thing I'll note is that the term "stability," in the context of ML Theory, typically implies something different from stochastic stability (which comes from Control Theory). Readers familiar with _algorithmic stability_ (see Bousquet & Elisseeff, JMLR 2002) may be confused or misled by the title/abstract. I recommend adding something in the abstract and intro that (a) briefly describes stochastic stability and (b) makes it clear that it is not the same as algorithmic stability.

Decision: accept