Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose Recurrent Explicit Duration Switching Dynamical System model, that is used to detect state and time switching dynamics. The method falls to the category of generalized switching state space model (Ghahramani & Hinton 2000). In particular, they use recurrent state-to-state switching with explicit duration models and provide approximate inference algorithm that is using evidence lower bound. Authors verify they model on 3 segmentation task from time series with ground truth and forecasting task on 5 datasets. 
main_review: Authors make a nice study, however I am not sure about the following:
1. Related work on jump models
The authors are missing big part of related work related to Jump models. Jump models have been intensively used in financial machine learning and they also generalize hidden markov models by having abrubt and presistent change of state dynamics. 
See Bemporad et al. (2018), that has proved that the HMM is a special case of a jump model. 

Bemporad, A., V. Breschi, D. Piga, and S. Boyd. “Fitting jump models.” Automatica,
vol. 96 (2018), pp. 11–21.

2. How does their model tolerate more noisy datasets
Authors should try to see how they model behaves on very noisy datasets e.g. financial.

3. Based on the results in table 2 for the forecasting, the method is performing the best only on 2 datasets. Do authors have intuition why?
limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors introduce RED-SDS an extension to switching dynamical systems (SDS) that combines state-dependent and duration-dependent transitions. 
main_review: # Strengths
I think the combination of state-dependent and duration-dependent transitions is very cool, as it leads to a much more flexible model. For learning RED-SDS, the authors build upon previous work on variational inference for SDS [1]. Crucially, the authors demonstrate the gradient for$ \underset{q(x_{1:T} \vert y_{1:T})}{\mathbb{E}}  [\log p(y_{1:T}, x_{1:T})]$ can be computed more succinctly as opposed to [1]; moreover, this also leads to better training, removing the need of the regularizer that was used in [1]. The experiments section was also very well done and very thorough.

# Weaknesses
I noticed that details are missing for the experiments section. In neither the main text nor the appendix, are there details regarding the architectures being used i.e. number of hidden units and activation functions. Moreover, I don't see any details regarding the annealing schedule for the tempered softmax function.

# Questions/Comments 
1) In line 556, shouldn't it be $ P \in \mathbb{R}^{K \times (d_{max} - d_{min})}$, not $ P \in \mathbb{R}^{K \times d_{max}}$?
2) In line 602, since the latent dimension is 2, I don't think it qualifies as a univariate time series.
3) In lines 619-629, the dancing bees dataset was described where it was stated that the data is comprised of 6 bees. Were the methods only trained on data from one bee or where each bee considered iid?
4) In line 671, it was stated the dimensionality of the state variables was set to 4 for the bouncing ball and 3 mode datasets. I find this weird (and slightly concerning), as 1) the true latent dimension is known and 2) the data generating process is a subset of RED-SDS, thus setting the latent dimension to be the true dimension should suffice.
5) In lines 769-773, the authors noted that they took the scaling of the data into effect by using the change of variables formula. This is interesting because the standard is just to use scale the data and treat it as if it wasn't scaled. I'm curious as to why the authors did this (Note, that the change of variables formula is only for one-to-one function; the sample average and standard deviation are not one-to-one).
6) In line 776, the authors stated they considered a range of $K \in [2, 5]$. I think it would be more appropriate to write it as $K \in$ {$2, 3, 4, 5$}, as $K$ can only be discrete.

# Conclusion
I think this was a good paper overall! Only minor modifications are needed.

# References
[1] Collapsed amortized variational inference for switching nonlinear dynamical systems, https://arxiv.org/abs/1910.09588
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors proposed a new state-space model which combines the previous two models: recurrent switching LDS and explicit duration together. The authors proposed an inference method to fit the model and applied the proposed method to several real datasets.
main_review: Originality: The proposed method combines recurrent state-to-switch with explicit duration. The model is new, but the idea seems straightforward and not revolutionary.

Quality: The paper did extensive experiments on multiple datasets qualitatively and quantitatively. The proposed model has a more complicated structure but doesn't seem to improve previous methods too much. The forecasting results are just comparable with the current methods. The segmentation results will be discussed below. 
1. Fig. 3, what are the inputs u for each dataset? Why SNLDS, ED-SDS, RED-SDS got very similar reconstruction, yet different segmentation results?
2. For dancing bee dataset, check paper https://people.eecs.berkeley.edu/~jordan/papers/fox-etal-nips08.pdf. They also fit ARHMM & SLDS with enhanced self-persistence on the dancing bee datasets. The probability distribution of staying at the previous state still follows a geometric distribution. They don't have recurrent state-to-switch or explicit duration in their model, but the results are better than the one shown in the paper.
3. Have you tried a more complicated segmentation task (e.g. number of switching states more than 3, like the one used in https://arxiv.org/pdf/1610.08466.pdf)? Current tasks in the paper are simple and seem not enough to show the superiority of having recurrent state-to-switch and explicit duration. Bouncing ball & 3 mode system are simulated data. Dancing bees data already has good (even better) SOTA results. 

Clarity: The paper is clearly written.

Significance: As discussed above, the paper did comparisons with previous methods qualitatively and quantitatively, but doesn't seem to improve current methods too much. The tasks in real data seem not complicated enough. It would be good if the code can be open-source so that others can apply the proposed model to their own datasets.
limitations_and_societal_impact: See the main review section above for suggestions for improvement. I don't foresee any negative societal impact of their work.
review_title: null

Meta-review: This paper takes a nice idea (combining recurrent state-to-switch dependencies with semi-Markov duration distributions), pairs it with an amortized inference algorithm, describes the approach very well, and does a thorough set of experiments with reasonable baseline comparisons to recent methods. While it is a relatively simple idea, it is well executed and I think it will be a very nice contribution to the field. 

In private discussion, Reviewers Naxb and GyB2 agreed that they would increase their scores to a 6, which would yield two 6s and a 7, for an average of 6.33.  I strongly encourage the authors to make the promised changes in the final publication.
Decision: Accept (Poster)