Following the recent progress in deep learning, researchers and practitioners of machine learning are recognizing the importance of understanding and interpreting what goes on inside these black box models. Recurrent neural networks have recently revolutionized speech recognition and translation, and these powerful models could be very useful in other applications involving sequential data. However, adoption has been slow in applications such as health care, where practitioners are reluctant to let an opaque expert system make crucial decisions. If we can make the inner workings of RNNs more interpretable, more applications can benefit from their power.There are several aspects of what makes a model or algorithm understandable to humans. One aspect is model complexity or parsimony. Another aspect is the ability to trace back from a prediction or model component to particularly influential features in the data BIBREF0 BIBREF1 . This could be useful for understanding mistakes made by neural networks, which have human-level performance most of the time, but can perform very poorly on seemingly easy cases. For instance, convolutional networks can misclassify adversarial examples with very high confidence BIBREF2 , and made headlines in 2015 when the image tagging algorithm in Google Photos mislabeled African Americans as gorillas. It's reasonable to expect recurrent networks to fail in similar ways as well. It would thus be useful to have more visibility into where these sorts of errors come from, i.e. which groups of features contribute to such flawed predictions.Several promising approaches to interpreting RNNs have been developed recently. BIBREF3 have approached this by using gradient boosting trees to predict LSTM output probabilities and explain which features played a part in the prediction. They do not model the internal structure of the LSTM, but instead approximate the entire architecture as a black box. BIBREF4 showed that in LSTM language models, around 10% of the memory state dimensions can be interpreted with the naked eye by color-coding the text data with the state values; some of them track quotes, brackets and other clearly identifiable aspects of the text. Building on these results, we take a somewhat more systematic approach to looking for interpretable hidden state dimensions, by using decision trees to predict individual hidden state dimensions (Figure 2 ). We visualize the overall dynamics of the hidden states by coloring the training data with the k-means clusters on the state vectors (Figures 3 , 3 ).We explore several methods for building interpretable models by combining LSTMs and HMMs. The existing body of literature mostly focuses on methods that specifically train the RNN to predict HMM states BIBREF5 or posteriors BIBREF6 , referred to as hybrid or tandem methods respectively. We first investigate an approach that does not require the RNN to be modified in order to make it understandable, as the interpretation happens after the fact. Here, we model the big picture of the state changes in the LSTM, by extracting the hidden states and approximating them with a continuous emission hidden Markov model (HMM). We then take the reverse approach where the HMM state probabilities are added to the output layer of the LSTM (see Figure 1 ). The LSTM model can then make use of the information from the HMM, and fill in the gaps when the HMM is not performing well, resulting in an LSTM with a smaller number of hidden state dimensions that could be interpreted individually (Figures 3 , 3 ).We compare a hybrid HMM-LSTM approach with a continuous emission HMM (trained on the hidden states of a 2-layer LSTM), and a discrete emission HMM (trained directly on data).We use a character-level LSTM with 1 layer and no dropout, based on the Element-Research library. We train the LSTM for 10 epochs, starting with a learning rate of 1, where the learning rate is halved whenever $\exp (-l_t) > \exp (-l_{t-1}) + 1$ , where $l_t$ is the log likelihood score at epoch $t$ . The $L_2$ -norm of the parameter gradient vector is clipped at a threshold of 5.The HMM training procedure is as follows:Initialization of HMM hidden states:(Discrete HMM) Random multinomial draw for each time step (i.i.d. across time steps).(Continuous HMM) K-means clusters fit on LSTM states, to speed up convergence relative to random initialization.At each iteration:Sample states using Forward Filtering Backwards Sampling algorithm (FFBS, BIBREF7 ).Sample transition parameters from a Multinomial-Dirichlet posterior. Let $n_{ij}$ be the number of transitions from state $i$ to state $j$ . Then the posterior distribution of the $i$ -th row of transition matrix $T$ (corresponding to transitions from state $i$ ) is: $T_i \sim \text{Mult}(n_{ij} | T_i) \text{Dir}(T_i | \alpha )$ where $\alpha $ is the Dirichlet hyperparameter.(Continuous HMM) Sample multivariate normal emission parameters from Normal-Inverse-Wishart posterior for state $i$ : $ \mu _i, \Sigma _i \sim N(y|\mu _i, \Sigma _i) N(\mu _i |0, \Sigma _i) \text{IW}(\Sigma _i) $ (Discrete HMM) Sample the emission parameters from a Multinomial-Dirichlet posterior.Evaluation:We evaluate the methods on how well they predict the next observation in the validation set. For the HMM models, we do a forward pass on the validation set (no backward pass unlike the full FFBS), and compute the HMM state distribution vector $p_t$ for each time step $t$ . Then we compute the predictive likelihood for the next observation as follows: $ P(y_{t+1} | p_t) =\sum _{x_t=1}^n \sum _{x_{t+1}=1}^n p_{tx_t} \cdot T_{x_t, x_{t+1}} \cdot P(y_{t+1} | x_{t+1})$ where $n$ is the number of hidden states in the HMM.Our main hybrid model is put together sequentially, as shown in Figure 1 . We first run the discrete HMM on the data, outputting the hidden state distributions obtained by the HMM's forward pass, and then add this information to the architecture in parallel with a 1-layer LSTM. The linear layer between the LSTM and the prediction layer is augmented with an extra column for each HMM state. The LSTM component of this architecture can be smaller than a standalone LSTM, since it only needs to fill in the gaps in the HMM's predictions. The HMM is written in Python, and the rest of the architecture is in Torch.We also build a joint hybrid model, where the LSTM and HMM are simultaneously trained in Torch. We implemented an HMM Torch module, optimized using stochastic gradient descent rather than FFBS. Similarly to the sequential hybrid model, we concatenate the LSTM outputs with the HMM state probabilities.We test the models on several text data sets on the character level: the Penn Tree Bank (5M characters), and two data sets used by BIBREF4 , Tiny Shakespeare (1M characters) and Linux Kernel (5M characters). We chose $k=20$ for the continuous HMM based on a PCA analysis of the LSTM states, as the first 20 components captured almost all the variance.Table 1 shows the predictive log likelihood of the next text character for each method. On all text data sets, the hybrid algorithm performs a bit better than the standalone LSTM with the same LSTM state dimension. This effect gets smaller as we increase the LSTM size and the HMM makes less difference to the prediction (though it can still make a difference in terms of interpretability). The hybrid algorithm with 20 HMM states does better than the one with 10 HMM states. The joint hybrid algorithm outperforms the sequential hybrid on Shakespeare data, but does worse on PTB and Linux data, which suggests that the joint hybrid is more helpful for smaller data sets. The joint hybrid is an order of magnitude slower than the sequential hybrid, as the SGD-based HMM is slower to train than the FFBS-based HMM.We interpret the HMM and LSTM states in the hybrid algorithm with 10 LSTM state dimensions and 10 HMM states in Figures 3 and 3 , showing which features are identified by the HMM and LSTM components. In Figures 3 and 3 , we color-code the training data with the 10 HMM states. In Figures 3 and 3 , we apply k-means clustering to the LSTM state vectors, and color-code the training data with the clusters. The HMM and LSTM states pick up on spaces, indentation, and special characters in the data (such as comment symbols in Linux data). We see some examples where the HMM and LSTM complement each other, such as learning different things about spaces and comments on Linux data, or punctuation on the Shakespeare data. In Figure 2 , we see that some individual LSTM hidden state dimensions identify similar features, such as comment symbols in the Linux data.Hybrid HMM-RNN approaches combine the interpretability of HMMs with the predictive power of RNNs. Sometimes, a small hybrid model can perform better than a standalone LSTM of the same size. We use visualizations to show how the LSTM and HMM components of the hybrid algorithm complement each other in terms of features learned in the data.

Q: What kind of features are used by the HMM models, and how interpretable are those?
A: A continuous emission HMM uses the hidden states of a 2-layer LSTM as features and a discrete emission HMM uses data as features. 
The interpretability of the model is shown in Figure 2.

Q: What kind of information do the HMMs learn that the LSTMs don't?
A: The HMM can identify punctuation or pick up on vowels.

Q: Which methods do the authors use to reach the conclusion that LSTMs and HMMs learn complementary information?

Q: How large is the gap in performance between the HMMs and the LSTMs?
A: With similar number of parameters, the log likelihood is about 0.1 lower for LSTMs across datasets. When the number of parameters in LSTMs is increased, their log likelihood is up to 0.7 lower.