Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This manuscript concerns a continuous-time variant of switching state-space models, aka switching stochastic differential equations (SSDEs). Such hybrid models have a Markov jump process $z_t$ that modulates the diffusion process $y_t$. The contribution of this work is a new and unifying mean-field variational framework that leads to scalable approximation. In particular, time evolutions of the approximate posteriors over the switch variables $q(z,t)$ and the conditional $q(y,t|z)$ are computed individually. The final optimization objective is a Lagrangian consisting of the evidence lower bound (ELBO) and implicit-form ordinary differential equation (ODE) constraints describing the above-mentioned time evolutions. As usual, Lagrange multipliers are given by the optimality conditions defined by the Euler-Lagrange equations. The method is demonstrated on simple toy and real-world datasets with irregular observations. It is shown to consistently learn the continuous dynamics and infer accurate approximate posteriors over the switch $z_t$.
main_review: With the neural ODE breakthrough and succeeding impressive NODE-based methodologies, continuous-time models have gained huge popularity. So, the manuscript studies a theoretically interesting and timely topic. Unlike existing approaches, the real-world applications of this work might be a little limited (perhaps for the time being) but the rigorous theoretical investigation easily outweighs it. 

The proposed variational inference framework leads to computationally convenient expressions as shown by the authors, which is often rare in such hierarchical models. Therefore, although the inference framework overlaps with [20] in large parts, I find the contribution significant.

The authors motivate and describe the problem very well. Despite the heavy use of mathematics, I was able to follow the main arguments of the manuscript. So, I could say that it is written as clear as possible. Also, please note that I was not able to follow all the equations in the Appendix line by line but there should be no issue in the ones in the main paper.

Overall, I recommend an accept as the submission is above average in all acceptance criteria. That being said, I am slightly concerned that the entire framework might rely on a wrong assumption. More specifically, is the factorization in eq.(12) correct? Should not one of the t's be on the rhs (to be conditioned to)? I wonder what the authors (and other reviewers) think about this.

Below are more detailed (minor) comments:
- The approximations typically hold when the posterior concentrates on a single value of $z$. What if it does not? When does it tend to hold?
- I wish to see the examples in which the framework starts failing, which would help us understand the limits of the method. Also, comparing the method with baselines would even further highlight its merits.
- It would be much better if the authors can state which ideas are inspired by which papers. This would allow the reader to check reference papers more easily.
- How important is the linear drift choice? How does the model behave with less/more data?
- The derivation in Section3.2 is deeply connected to Pontryagin's maximum principle, which goes uncited. I suggest the authors check out the connections, which would allow the model to draw more attention.
- Why is back-tracking line search needed? In general, how long does the optimization take? How sensitive is the optimization?
- In line 242, do the left hand side expressions resemble Viterbi path? If so, the relationship should be shown.

**Post-rebuttal comment:** Thanks for the rebuttal. All in all, I believe the paper should be accepted. It would be better if the following are addressed in the final version (or any arxiv submission or similar):
- Lack of quantitative comparisons
- References should be improved (I don't very closely follow the "Switching Dynamical Systems" literature but at least the VI framework is heavily inspired from [20], which is not explicitly mentioned in the text).
- I still believe the factorization in eq.(12) is invalid. I'm starting to think this is just a notation issue.
limitations_and_societal_impact: All the limitations and potential negative societal impacts are addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The manuscript extends the literature of statistical inference in switching linear systems to the continuous domain. Starting from the specifics of the generative model (a discrete valued latent stochastic Markov Jump process modeling the "mode" of the hybrid process and a continuous valued latent diffusion process modeling the "state" of the process and finally an observation model connecting the latent space to discretely sampled observations) and then provides the expressions for equations governing the joint, posterior (or smoothing) and marginal distributions. The authors then show why exact inference is intractable and provide the methodology needed for approximate (structured mean-field) variational inference. The variational family considered is a mixture of GPs (which turns out to be the case because of the structural PDE equations enforced on the variational family to mimic the true posterior distribution) where the time-specific mixture weights follow a master equation and variational factor q(y,t|z) follows a a Fokker-Planck equation. The ELBO cost function is then derived and optimized using forward-backward sweeping algorithm. To optimize the model parameters (transition rate function, parameters of the drift function, and dispersion) a variational EM approach is taken where the parameters are updated iteratively after satisfying structural constraints and updating variational parameters. Experiments are performed on toy examples as well as multi-well potentials and ion channel data.
main_review: Strengths
=======
The question under investigation is timely, significant, and relevant to NeurIPS community in my opinion. 

The background section includes a nice summary of the prerequisites needed for following the ideas throughout the paper. 

The introduction provides a nice methodological literature as well as the application of the methods in various fields. 

Although the content of the manuscript is fairly complex I found the flow of the writing very sensible and easy to follow. 

The amount of mathematical details included in the main text seems appropriate and every details that is mentioned in the supplementary is properly referenced in the main text. 

Finally the experiments are diverse and the visualizations are very helpful for digesting the outputs of the model and its inner workings.

Suggestions
==========
Adding some analytical intuitions (not derivations) would help the reader to have a better sense about the equations.

It would be nice to supplement the visualizations with the observations x(t) to show what type of data is being considered.

Weaknesses
==========
The experiments are performed mostly on the cases where the analytical form of the drift function is known. However in real world scenarios it's rarely the case and in many cases the function is approximated using another parametric family. For discrete cases to the best of my knowledge neural networks are used to model the drift function (e.g. https://arxiv.org/abs/1605.08454). It would be useful to investigate if the model can recover dynamical regimes even in the presence of mismatch in the drift function.

Almost all the model validation that is done in the experiments section is qualitative. Even for a low-D system the parameter error (beta) seems relatively large. A more thorough investigation of the error as a function of dimension, number of modes, and separability between different dynamical regimes would help practitioners to apply the method with some prior caution.

Since the field of statistical inference in discrete switching dynamical systems is fairly stablished it would be nice to perform comparisons between the fit given by the continuous model (proposed) and discrete models.
limitations_and_societal_impact: For limitations, see above.

As the authors mentioned, the work is theoretical and hence there are not trivial societal and ethical impacts subject to discussion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a variational inference for continuous-time switching dynamical systems, whose inference is believed to be lacking in need of biology and engineering applications. The form of the variational distributions is constrained by a master equation for the jump process and a Fokker-Planck equation for the diffusion process. These constraints are encoded in the Lagrangian. Both model parameters and variational distribution are inferred by a variational EM algorithm. The validity of the model is demonstrated with low-dimensional experiments.
main_review: The derivation is very fundamental, and the statement is mathematically rigorous. It starts from the Markov process formulation and arrives at the inference algorithm for switching dynamical systems. My main concern is that whether such lower-level formulation is required for applications. The following are specific concerning points and please correct me if my understanding is wrong.
- By the formulation, since continuous variables in both variational processes and the prior processes have linear drift, can we have explicit mean and variance representations for the diffusion process, e.g., Ornstein-Uhlenbeck process has an explicit time-dependent mean and variance? If this is possible, then we can work with a parametric Gaussian as the diffusion process and are free from the constraints.
- The experiments are conducted in low-dimensional space, does it work in a high-dimensional setup? There are some works using diffusion processes for deep models or high-dimensional data. 
[Kurle, Richard, et al. "Continual learning with bayesian neural networks for non-stationary data." International Conference on Learning Representations. 2019.]
[Bamler, Robert, and Stephan Mandt. "Dynamic word embeddings." International conference on Machine learning. PMLR, 2017.]
 
limitations_and_societal_impact: I didn't find the limitation statements, although the checklist indicates there is one in a quite large page range. It would be better to describe the computation aspects and whether it fits modern high-dimensional applications.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors propose a variational inference framework for continuous time hybrid process models. The particular assumption on the true posterior allows for a mixture of GPs as an approximation to it. The proposed framework is tested on various benchmark tasks including real-world biological data and the results demonstrate its ability to faithfully reconstruct complex latent dynamics and to learn the unknown system parameters, in particular in applications to meta-stable systems. It extends the toolbox for inference will be of great utility for the analysis and control of such hybrid systems.
main_review: Originality: The proposed method is new for continuous-time SLDS. The assumed variational posterior enables inference via optimization. The relevant previous contributions were clearly discussed and cited.

Quality: The submission is technically sound. The methodology is rigorously derived and empirically tested with various synthetic and real data. The work in this manuscript is a complete piece of work. The weaknesses of the work was not substantially evaluated, especially for non-meta-stable systems like continuous attractors. The experiments shown in the manuscript do not include a case that the number of state $Z$ is mismatched. Since the optimization of ELBO is generally a non-convex problem, the authors better talk about the convergence, e.g. empirically convergence reached and converging to the same or different optima with different initial values, and if the optimas are equivalent.

Clarity: Overall, the manuscript was well organized and written clearly. See itemized comments.
- Line 202: Where are $A_p$ and $b_p$ defined? What does the subscript $p$ represent, prior? Are they assumed constant as $A$ and $b$ latter on.
- Fig2 B: I suppose the curve attached to the right is the wells, but it is not described in the caption.

Significance: This work extends SLDS to continuous-time regime and proposes easy approximate inference via optimization. SLDS is well known in e.g. the field neuroscience for modeling meta-stable dynamics. This work would have high impact in such fields and be of great utility for the analysis and control of such hybrid systems.
limitations_and_societal_impact: The authors do not talk much about limitations or societal impact.
review_title: null

Meta-review: This paper presents a model for continuous-time temporal datasets. The model takes the form of a Markov jump process that modulates a diffusion process. The work addresses posterior inference by applying a continuous-time variational inference algorithm that combines Gaussian processes for the diffusion process and posterior inference for the Markov jump process.

The reviewers agreed the paper is timely, significant, and relevant for the community, and that it deserves acceptance.

I encourage the authors to address the concerns raised by the reviewers. In particular:
+ Include a more thorough investigation of the parameter error in different regimes (i.e., dimensions, modes, separability of the modes). A discussion of which of these might be the limiting factor from a practical perspective would be useful for the readers.
+ Improve the references [R1-R9], including the ones in the discussion with reviewer d46C and specially the references about the variational inference method, e.g., mention [20] in the text.
+ Clarify the issue with the factorization in Eq. (12). The authors argued that it's just an attempt to make the notation uncluttered, but I agree with reviewer czKb that it creates confusion and it makes it look like the factorization is incorrect. Consider changing the notation accordingly.
+ Address the other changes that were discussed during the review period (e.g., extend Section 3.3, include an example of ELBO vs iterations, clarify the points about the drift function and about Figure 2, etc.).
Decision: Accept (Spotlight)