Variational Inference for Continuous-Time Switching Dynamical Systems

Abstract: Switching dynamical systems provide a powerful, interpretable modeling framework for inference in time-series data in, e.g., the natural sciences or engineering applications. Since many areas, such as biology or discrete-event systems, are naturally described in continuous time, we present a model based on a Markov jump process modulating a subordinated diffusion process. We provide the exact evolution equations for the prior and posterior marginal densities, the direct solutions of which are however computationally intractable. Therefore, we develop a new continuous-time variational inference algorithm, combining a Gaussian process approximation on the diffusion level with posterior inference for Markov jump processes. By minimizing the path-wise Kullback-Leibler divergence we obtain (i) Bayesian latent state estimates for arbitrary points on the real axis and (ii) point estimates of unknown system parameters, utilizing variational expectation maximization. We extensively evaluate our algorithm under the model assumption and for real-world examples.


Reviews and author responses:

The paper introduces a novel variational inference method for hybrid models in which latent variables evolve in continuous time with switching dynamics, and are sampled at irregularly-spaced discrete time points to lend the observations. The inference of this model is detailed, and the parameters are then inferred for several simulated cases, as well as switching ion channel data.  This is a very interesting model class for different applications in biology and real-world physical systems. However, the authors need to address the following questions before the paper can be accepted.

a)	How does the model scale in # of dimensions y and # of states z?
b)	How does this model compare to existing methods, including for example a discrete-time SLDS? 
c)	Please show how this model scales with N, the number of samples of X available, and the sampling frequency. Yes

 I thank the authors for the detailed replies.

Hierarchical models (additional priors on parameters) and sampling based inference are out of the scope here.

Regarding the 'girsanov' term, since you are working with Markovian Gaussian processes, the expectation under Gaussian marginal distributions could be approximated via monte carlo method or quadrature. This is commonly done to deal with non conjugate variational inference for GP models (see the Gpflow toolbox). Using linear drift is good if you don't want additional approximations.

It would be great to have the 'revert to the prior' behavior in a figure - which you agreed to do, thanks. For me this is a key feature that make me doubt a method when I don't see it. Typically, this should happen but can be lost depending on the approximation or optimization problems (in practice).

In light of this response and the responses to the other reviewers' concerns, I maintain my favorable appreciation of the manuscript and recommend it to be accepted.

The paper proposes a framework to learn continuous dynamical systems that depend on a switching discrete variable, regulating the mode of the process. The procedure is optimised using variational inference algorithm. Experiments with respect to ground truth show the method to correctly recover the underlying system.  The paper reads well, and even though I'm not familiar with the literature I can understand this is an interesting problem and a relevant contribution in the field. I am a bit confused by the sentences "In many real-world scenarios, discrete modes that drive continuous dynamics do not exist a priori, but continuous dynamics often exhibit regimes that are qualitatively different from one another", which seem to be opposite to the "Many natural and engineered dynamical systems can be understood in terms of continuous-discrete hybrid models, in which a given system switches between discrete modes exhibiting continuous dynamics" in the introduction. In general I guess the authors wanted to say that the discrete modes would provide a useful modelling of continuous dynamics that is switching modes, but do not generally exists. As such, I wonder what would be the benefit of modelling such behaviour in the proposed way? Especially when there is no ground truth on the modes (contrarily for example to the movement primitives of sec 4.4). I think one interesting application where it would be beneficial it is for example for clustering or unsupervised learning (https://web.stanford.edu/~boyd/papers/pdf/tvgl.pdf), where the method can be used to discretise a continuous process and then employ methods that work with discrete time. It would be interesting to see an application of the proposed approach beyond just the recovery of the ground truth (which in real cases we don't have). 

My main comments are on the experiment section. First, there is no comparison with the state of the art. Even if the method is novel and there is no other methods that combined a switching variable and a continuous dynamics, I cannot appreciate what is the benefit of the proposed method over the state of the art. In particular, I wonder: in the showed experiments, what would be the limitation of the state of the art? How accurately, even without considering a switching variable or a continuous process, it would be another method? For example, I wonder how a simple GP would perform on the showed experiments as a baseline.

Second, there is no clear quantitative results shown. From what I can see visually, it seems that the method is able to recover the ground truth. However, how is the performance of the method decreasing based on dimensionality, outliers, noise and so on? What is the computational time? How scalable is the method? 
 none

 I thank the authors for replying and addressing my concerns.

I understand the difficulty in comparing with the state of the art given that no other similar hybrid framework exists. While a better quantitative analysis would be needed to understand the benefit of the method from someone not familiar with the literature, I can appreciate the novelty of the method and willing to reassess my score.

 Thank you for addressing the questions posed in the review, and for the detailed response to other reviewers as well. 

In (b), I meant a comparison with any existing method, which the authors claim is not possible (in response to other reviewers). However, a comparison with either a purely discrete-time model by artificially discretizing the samples (as suggested in my review), or a purely continuous-time model without the discrete states, or a two-step inference process, would have been helpful to drive home the limitations that are present with existing approaches. Moreover, it would be good to see how the discrete states (perhaps) naturally fall out of the data, again driving home the need for the proposed approach. 

Again, thank you for the responses; they are taken into consideration. 

 Thank you for your helpful thoughts and comments.
First, the assertion is indeed correct that “the authors wanted to say that the discrete modes would provide a useful modeling of continuous dynamics that is switching modes”; we will clarify this in the camera-ready version. 

To demonstrate the usefulness of our method in cases where a ground truth on the modes does not exist (i.e., in absence of a “real’’ discrete switching process), we included experiments 4.2 and 4.3, where we obtain sensible results. Part of what the method achieves is precisely a clustering over time. It however goes beyond that in that it yields posterior distributions for both the discrete process (which corresponds to the cluster assignment over (continuous) time), and also the continuous process. The general benefit of this approach is two-fold: first, it is readily interpretable. Interpretability is a key interest in modeling for basic research, e.g., in the context of molecular dynamics (see [47]). A large body of literature exists for instance around the topic of Markov state modeling (MSM), where one tries to construct simple, easily interpretable models of molecules switching between different conformations (see, e.g., [48-51]). Second, the modeling results can also be readily utilized for engineering applications (e.g., imposing different continuous control signals depending on the discrete mode assignment).

This is also precisely the limitation of the state of the art; there is no hybrid process inference framework. Any established method could hence only infer either a discrete or a continuous latent process, but our goal is to infer both jointly. The proposed simple GP fit would not give us any information about the latent discrete state, i.e., we would not obtain a cluster assignment and lose the interpretability in terms of distinct modes.

Referring to your proposal to discretize the continuous process, we note that, while conversion to a discrete-time model formulation or even a two-step process (clustering in continuous time with subsequent discretization, as proposed) is possible, one would need to introduce pseudo-observations at time points where no observation is present, which is a modeling problem on its own; this situation necessarily occurs whenever the observations are not equally spaced. We do believe that natural processes that evolve in real time are most adequately described in a unified continuous-time framework rather than utilizing multi-step analysis procedures relying on ‘artificial’ time-discretization; this also has the substantial practical advantage that one can employ sophisticated differential equation solvers, which control numerical instabilities and errors via, e.g., adaptive step-size methods.

Regarding the problem of higher dimensions and computational effort, we agree that a quantification of the performance is desirable to gain a deeper understanding of the method and its limitations. However, since none of these are trivial questions (for instance, straightforward scaling of the framework to higher dimensions requires additional approximations, see Vrettas, Opper, Cornford; Phys Rev E 2015), a thorough analysis would be beyond the scope of this work, which aims to introduce the framework as such. See also the discussion with reviewer 4Rfw.


 Thank you for your thoughtful and concise feedback.
Potential biases are two-fold in our method: first, as is common for variational inference approaches (see, e.g., [19]), the posterior uncertainty is expected to be underestimated (also known as ‘mode-seeking property’). Second, the VEM optimization scheme may cause an inherent bias; note, however, that we utilize VEM merely for simplicity and we are aware that more refined methods may be used, which would be beyond the scope of the present work. For further discussion, see reviewer x8VX and the corresponding response. We will add this discussion to the final version of the paper.


 Thank you for the concise and thoughtful questions, which we answer one by one: 

a) The forward equations (15) for the Gaussian parameters $\mu, \Sigma$ scale as $\mathcal{O}\left(n^2 \lvert\mathcal{Z}\rvert\right)$ (as there have to be carried out $\lvert\mathcal{Z}\rvert$ matrix-vector multiplications of size $n$) and $\mathcal{O}\left(\frac{n(n+1)}{2}\lvert\mathcal{Z}\rvert\right)$ (as due to the symmetry of $\Sigma$, only the upper diagonal has to be propagated). Note that the backward equations (18) scale similarly. Furthermore, when using a conventional ODE solver, the master equation eq. (13) scales quadratically in $\lvert\mathcal{Z}\rvert$. We will make this explicit in the camera-ready version.

b) We think that a comparison to advanced discrete-time SLDS methods would not be fair; our intention is not to outperform established discrete methods, but to introduce a principled framework for continuous time hybrid systems. Additionally, our assumed general setting with non-equally spaced observations does not readily transfer to the discrete-time case.

c) Our method scales linearly with the number of observations, since the ODEs resulting from the variational approach are solved in between observations; we will also make this explicit in the final version.



 Thank you for the constructive feedback and insightful comments. 
We will include the mentioned citation about potential problems of VEM and extend our discussion in this respect, as we agree it is worth mentioning. This presumably relates to the empirically observed fact that learning via gradient ascent on the ELBO is indeed a hard problem. Also, gradient ascent only guarantees reaching a local optimum (not necessarily a global one). To alleviate this issue, one can employ multi-start methods utilizing multiple initial values; we will add this to Sect. 3.3. in the camera-ready version. We emphasize, however, that the use of VEM is a decision we make for simplicity and that should be refined in the future; as mentioned in the paper, one can for instance establish Bayesian estimates for the model parameters by placing priors on them and extending the VI framework. Closed-form, tractable updates for the parameter distributions are available if one chooses conjugate prior distributions. An extensive analysis of this and related issues is out of scope for the present work, but should be analyzed in future work to better understand and generalize the model class.

We deliberately did not include non-linear prior models to keep things as simple and concise as possible, since our main goal was to present the hybrid variational framework as such. Non-linear prior models would (i) entail an additional approximation error, if the variational process is still chosen linear and (ii) even for such linear variational models, one would encounter analytically intractable expectations (e.g., $\mathbb{E}[\Vert g – f\Vert^2]$, cf. eq. (11)). Such expressions could be evaluated using sampling techniques, resulting in a set of effective SDEs instead of ODEs for the Lagrange multipliers, eq. (15).  While being outside the scope of the present work, we think this is a very interesting topic that is worth being studied in isolation as a form of stochastic variational inference for continuous-time systems.

As described above, gradient optimization of the objective is indeed hard. Unfortunately, sampling (for a direct comparison) from the exact posterior is a highly non-trivial task; a sampling scheme does not exist to the best of our knowledge. To obtain exact samples, either the PDE (7) has to be solved, which is infeasible for general problems, or some form of importance sampling has to be employed. This is neither straightforward, because one has to carefully engineer a sensible proposal distribution that is (i) simple enough, so that sampling from it is easy, but (ii) as ‘close’ as possible to the exact posterior to obtain high acceptance rates. Prior work exists, for instance, for diffusions utilizing sample ‘bridges’ between observations (Mider, Schauer, van der Meulen; “Continuous-discrete smoothing of diffusions”). One would need to transfer this to the more general hybrid systems, which is outside the scope of the present study, and hence we did not include such a comparison here. 

As to the question of uncertainty propagation in the individual GPs, we note that, since the presented framework is a Bayesian approach, the uncertainty of the posterior processes relaxes to the prior values in regions where no evidence for a given mode z is present. A corresponding figure showing the individual modes will be included in the supplement for completeness.

Confusion regarding the initial condition might result from our notation, where we use the subscript $0$ to indicate the initial time point. Line 74 should be understood as a definition, where the initial condition is composed of two separate functions (in z and z,y, respectively) at $t=0$.


The authors extend the variational treatment of path inference and learning in generative models with SDE prior, by Archambeau and Opper, to the ‘switching’ setting, whereby a ‘sticky’ discrete continuous time process governs the index of the driving SDE within a discrete set.

The proposed approach is a natural extension. It requires additional assumptions on the variational distribution : specific factorization of the variational posterior over path y and identity variable z.  The particular posterior coupling allows for closed form update for the variational parameters via the tools of lagrangian duality and boils down to classic continuous time filtering.

The approach is evaluated on a series of problems where it demonstrates its inference and learning abilities.
  The manuscript is very well written (really!). The background material is introduced in the right amount of details and relevant related work is cited on the way.

The method makes a lot of intuitive sense and is well described. 
This continuous time variational treatment of swtiching SDE models is new and well conducted.
As such it is significant and useful to share with the community as is.

There is not much discussion of the effect of the mean field approximation on the learning algorithm.
Richard turner has a 2011 paper discussing the problems of VEM for learning in time series models that would be worth mentioning.

The empirical evaluation is thorough and focused on providing intuition and interpretability rather that quantitative evaluation / comparison to alternative - which is great.

I would have like to see a learning example with non-linear SDEs - all evaluations assume linear models, but I am happy enough with the linear examples shown here.

Regarding the results, the parameters recovered in sec 4.1 have the right sign but are quite far from the ground truth. Is it because learning (climbing the objective) is hard ? or because the VEM objective is biased? or because there is not enough date ( it would be reassuring to know that the values get better as you add more observations). Or maybe you could compare with an 'exact' sampling based inference scheme.

For the results, it would be useful to also see the posterior process for each z separately. Most importantly, do the posterior process for say z' has  growing uncertainty on domains of time where index z is driving the process?


Overall, the scope is narrow (in a positive way) and the authors acknowledge that, yet the presented work is well conducted and significant. Very clear. Hard to find a typo. I recommend for acceptance as is.


Comments
- typo on the initial conditions ? p(y,z,0) 
- I personally find the notation having time t as an argument in the pdf confusing, I prefer p(x(t)) over p(x,t). But that is really a matter of taste I guess.
- I m not sure I'd introduce the pdf as the gradient of the cdf for every distribution as is done here. It does not provide that much in the manuscript.


 None

This paper presents a variational inference algorithm for approximating the smoothing posterior of a so-called hybrid stochastic process, wherein a SDE jumps between operational “modes” following an independent Markov jump process (MJP). This is a paper on methods, and the primary contribution is the derivation of the corresponding constrained ELBO objective that is to be optimized to compute the approximating VI posterior.  Overall, I found the paper to be interesting, if somewhat narrow. VI for diffusions/SDEs and MJPs has been introduced in the literature, and the under the stated assumptions of this paper (i.e., the independence of the diffusion and MJP) the extension of these methods is not hard to see. My understanding is the primary import of the paper, and one that should be of interest more broadly, is the derivation of the constrained ELBO and the modeling of the variational posterior family as a mixture of GPs. This method should be useful in other problems as well.

Originality: The approach proposed in the paper is not new, and it largely extends existing methods to the hybrid process setting under consideration here. 

Quality and clarity: The derivations and numerical results are somewhat inadequate, in my opinion. In particular, it would be useful to have a sense of the inherent bias in the method. Overall, the derivation of the VI posterior and the “lagrangified” ELBO is adequate. Overall quality of the writing is acceptable.

Significance: The method proposed should be useful to a broad range of application domains, and therefore of significance to users in those areas.These are noted in the introduction of the paper. The authors do not address the potential negative social impact of their work. On the other hand, this is a paper on a specific method, and I would not expect extensive societal impact.

Meta-review: Reviewers agree that the proposed inference framework for continuous-time switching dynamical systems is technically innovative and very clearly described.  In future revisions, please be sure to include more details about computational costs, and also expand the discussion of connections to related work (as highlighted by reviewers).

Decision: accept