Title: Recurrent Hidden Semi-Markov Model
Authors: Hanjun Dai
Bo Dai
Yan-Ming Zhang
Shuang Li
Le Song
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Structured prediction

Abstract: Segmentation and labeling of high dimensional time series data has wide applications in behavior understanding and medical diagnosis. Due to the difficulty in obtaining the label information for high dimensional data, realizing this objective in an unsupervised way is highly desirable. Hidden Semi-Markov Model (HSMM) is a classical tool for this problem. However, existing HSMM and its variants has simple conditional assumptions of observations, thus the ability to capture the nonlinear and complex dynamics within segments is limited. To tackle this limitation, we propose to incorporate the Recurrent Neural Network (RNN) to model the generative process in HSMM, resulting the Recurrent HSMM (R-HSMM). To accelerate the inference while preserving accuracy, we designed a structure encoding function to mimic the exact inference. By generalizing the penalty method to distribution space, we are able to train the model and the encoding function simultaneously. Empirical results show that the proposed R-HSMM achieves the state-of-the-art performances on both synthetic and real-world datasets. 

TL;DR: We propose to incorporate the RNN to model the generative process in Hidden Semi-Markov Model for unsupervised segmentation and labeling.

Review discussion:
How would this method compare to conditional random field-based approaches (e.g. Ammar, Dyer, and Smith NIPS 2014)?
Thanks for your important referenced paper. The differences between the proposed model and algorithm and the CRF autoencoder are listed below:
i), Model:
The generative model (reconstruction) used in Ammar et.al., is HMM with conditional iid emission, which doesn’t have duration concept. Their generative model needs to be simple, in order to get a tractable form to optimize in their \lambda step.
Thus in their paper the generative HMM can be factorized and get a nice form in Eq (1).
Our model incorporates explicitly modeling the duration.
Moreover, we use the mixture of  recurrent neural networks as the emission model to capture the switch between different generative processes in different state, which are more flexible in modeling and much harder to be learned (Chiappa 2014, Murphy 2012).
ii), Inference:
In Ammar et.al., the inference is done by solving MAP in CRF, which requires a dynamic programming. This is efficient for HMM, but not for our recurrent HSMM.
Thus we propose to use variational encoder. Our variational bi-rnn decoder is much more efficient while getting the similar performance, as shown in our experiments.
iii), Learning:
In general, we extended the the penalty method into distributional space to learn the proposed recurrent-HSMM and the sequential VAE.
Specifically,
**for encoder:
In Ammar et.al., the \lambda-step learns encoder via gradient based algorithm. Optimizing their Eq (1) will marginalize over all possible latent variables.
Therefore, a restricted generative model, as well as a tractable CRF is necessary. In our model, the latent variables are sampled from auxiliary distribution \tilde{Q}.
Thus the variational bi-rnn only needs to focus on the sampled latent variables, which doesn’t put too much limitation to the generative model.
**for decoder:
Ammar et.al., use EM to learn the generative HMM. We argued that full EM is computationally too expensive for recurrent HSMM.
So we also use samples of latent variables to learn the parameters of HSMM and generative RNNs.
We will include this reference into our main text.
This is a very good paper. My only question so far: did you observe a minimal sequence length where the bidirectional RNN decoder would break down (i.e., not be able to learn properly, as compared to the other HSMM approaches)?
Thanks for your positive feedback! The length of the sequence will affect the learning of both the generative RNNs and the bi-rnn encoder. We’ve got several observations empirically when we test the method on synthetic datasets. The proposed algorithm can get almost-perfect recovery of underlying latent model if the duration of each state  >= 20 and the entire sequence length roughly >= 80. The first requirement, makes sure the sequence contains a segment covering an entire period of observations, and allows the generative RNNs to learn the dynamics in each state. The second condition makes sure the entire sequence at least cover 2~3 states, in order to let bi-rnn encoder learn the transitions.
For experiments on real datasets, we didn't observe this break down, since typically the real world sequences are very long.
This paper presents a novel model for unsupervised segmentation and classification of time series data.  A recurrent hidden semi-markov model is proposed.  This extends regular hidden semi-markov models to include a recurrent neural network (RNN) for observations.  Each latent class has its own RNN for modeling observations for that category.
Further, an efficient training procedure based on a variational approximation.  Experiments demonstrate the effectiveness of the approach for modeling synthetic and real time series data.
This is an interesting and novel paper.  The proposed method is a well-motivated combination of duration modeling HMMs with state of the art observation models based on RNNs.  The combination alleviates shortcomings of standard HSMM variants in terms of the simplicity of the emission probability.  The method is technically sound and demonstrated to be effective.
It would be interesting to see how this method compares quantitatively against CRF-based methods (e.g. Ammar, Dyer, and Smith NIPS 2014).  CRFs can model more complex data likelihoods, though as noted in the response phase there are still limitations.
Regardless, I think the merits of using RNNs for the class-specific generative models are clear.
Thank you very much for your review and suggestions!
We have included the quantitative comparison against CRF-autoencoder in our paper on all the datasets.
Regarding the CRF autoencoder, although we’ve obtained the code from the first author, it is designed for discretized observations, and is not trivial to extend it to continuous observations (as mentioned by the first author in personal communication, Dr. Ammar).
Thus we re-implemented with the extension to continuous case. We also extended with more powerful emission model for P(x_i | y_i), like mixture of Gaussian (where the number of mixtures is tuned).
Overall the crf-autoencoder is not performing well in our segmentation task. It achieves similar or worse performance than classical HSMM. We conjecture the major issue lies in the modeling part. In CRF-autoencoder, the reconstruction model is conditional i.i.d, and the CRF encoder is markovian. Comparing to HSMM, it ignores the segmentation structures in modeling and more similar to HMM, which is weaker for segmentation task.
We believe the CRF-autoencoder can be further improved by introducing more useful features (e.g., in the original paper they used information like word-cluster, etc.), but it will be beyond the focus in our paper. For the fair comparison, all the methods use only the provided observations to do segmentation.
Putting the score for now, will post the full review tomorrow.
This paper proposes a novel and interesting way to tackle the difficulties of performing inference atop HSMM. The idea of using an embedded bi-RNN to approximate the posterior is a reasonable and clever idea.
That being said, I think two aspects may need further improvement:
(1) An explanation as to why a bi-RNN can provide more accurate approximations than other modeling choices (e.g. structured mean field that uses a sequential model to formulate the variational distribution) is needed. I think it would make the paper stronger if the authors can explain in an intuitive way why this modeling choice is better than some other natural choices (in addition to empirical verification).
(2) The real world datasets seem to be quite small (e.g. less than 100 sequences). Experimental results reported on larger datasets may also strengthen the paper.
Thank you very much for your review and suggestions! We will emphasize the benefits of utilizing bi-RNN as encoder in the refined version.
To answer your questions:
1) For intuition: We’ve briefly discussed the intuition of choosing bi-RNN in Section 3. Recall the forward-backward algorithm is doing the exact inference, we expect the bi-RNN to mimics this forward-backward procedure, and thus, achieving a good approximation. From this perspective, the bi-RNN already takes the structure into account.
Specifically, the forward messages are computed in a recursive markovian and linear way;
while in forward RNN, the hidden layers are also recursively computed in a nonlinear way. For backward process, similar analogy is applied. Computing the marginal in bi-RNN is also analogous.
The structured mean field could be another option.
However, it requires the lower bound implied by the structured distribution must be available in closed form (Hoffman & Blei, 2014) in order to be efficient.
Also, it’s inference still requires the calculation of the normalization term, which won’t have advantage of speed over bi-RNN.
For implementation: bi-RNN is quite simple to implement and easy to train (by sgd);
For performance: We’ve also explored one directional RNN in our early stage, and found bi-RNN works better.
2) In our earlier experiments, the sequences in each datasets are very long (e.g., ~40k in Physionet).
So each dataset has roughly 0.1~0.3*10^6 high dimensional observations. Actually, the main computational challenge comes from the length of sequence (complexity is analyzed in Section 4.1) rather than the number of sequences (since we can always use stochastic training to sample i.i.d. sequences, as in line 5 of Algorithm 1).
We’ve addressed your suggestion by including an experiment on a new dataset constructed based on Physionet. The dataset consists of 2750 sequences, where each sequence has ~1.5k 4-D observations, resulting ~4.5 *10^6 observations in total. We did 5-fold cross validation in this case.
For more information about the dataset and experimental details, please check our updated paper.
Dear reviewers, we have revised our paper according to your insightful suggestions and comments.
Specifically, we added a baseline CRF-autoencoder, and did the quantitative comparison across all datasets. We’ve also added a large dataset, which contains 2750 sequences, where each sequence has ~1.5k 4-D observations.
For more details about the dataset (called PN-FULL in the paper) and comparison with new baseline (called CRF-AE in the paper),  please check our revised pdf.

Meta-review:
role: PC
data: comment: This paper is a by-the-numbers extension of the hidden semi-Markov model to include nonlinear observations, and neural network-based inference. The paper is fairly clear, although the English isn't great. The experiments are thorough.
 
 Where this paper really falls down is on originality. In particular, in the last two years there have been related works that aren't cited (and unfortunately weren't mentioned by the reviewers) that produce similar models. In particular, Johnson et al's 2016 NIPS paper develops almost the same inference strategy in almost the same model class. 
 
 http://stat.columbia.edu/~cunningham/pdf/GaoNIPS2016.pdf
 https://arxiv.org/abs/1511.05121
 https://arxiv.org/abs/1603.06277
 
 This paper is borderline, but I think makes the cut by virtue of having experiments on real datasets, and by addressing a timely problem (how to have interpretable structure in neural network latent variable models).

Decision: Accept (Poster)