Title: Automatic Rule Extraction from Long Short Term Memory Networks
Authors: W. James Murdoch
Arthur Szlam
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Applications

Abstract: Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear.   As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns.  In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.

TL;DR: We introduce a word importance score for LSTMs, and show that we can use it to replicate an LSTM's performance using a simple, rules-based classifier.

Review discussion:
It's known that many types of QA problems can be solved by matching simple patterns. See, for example, "A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task" by Chen et al. At a first glance, WikiMovies seems to be in the same category as these in terms of complexity.
Do you think you can apply or extend your rule extraction approach to more complex tasks? If so, which tasks and what kinds of extensions would your method require?
Apologies in advance if you already discussed this or if I misunderstood something.
Thank you for the question, and apologies for the delayed response.
Although our results on WikiMovies are interesting in their own right, their primary purpose was to provide empirical validation of the information contained in our proposed importance scores.
Given a trained LSTM, we currently have very limited means for explaining it's predictions. Empirically, we found that examining our importance scores provides sensible explanations, but we wanted to provide stronger verification of the information contained in these scores. We feel that the rule extraction fulfills this role.
To contrast with Chen et. al, they specified and trained a model completely independently of any other model, as is generally done in ML. In this work, we extracted the parameters of our pattern matching model solely from our importance scores, which are derived from a trained LSTM's parameters.
Thus, the training data is only observed indirectly through the importance scores. We interpret the fact that we're able to achieve competitive accuracy by only looking at the importance scores, and not the training data, as a verification of our empirical observations that they are a highly informative metric.
As for more complex tasks, we are currently finalizing results on sentiment analysis, and will be adding them soon. Modifications to the algorithm are relatively minimal, and could be adapted to any text-based classification task.
Thanks for clarifying this.
This paper proposes a novel method for extracting rule-based classifiers from trained LSTM models. The proposed method is applied to a factoid question-answering task, where it is demonstrated that the extracted rules perform comparatively to the original LSTM. The analysis of the extracted rules illustrate the features the LSTM model picks up on.
Analyzing and visualizing the computations carried out by RNNs in order to understand the functions they compute is an important direction of research. This sort of analysis will help us understand the pitfalls of RNNs, and how we can improve them.
Although the approach taken is relatively inflexible - each rule is defined as an ordered sequence of words - the authors experiment with three different scores for picking salient words (state-difference, cell-difference and gradient) and their approach yields comparable performance, which suggests that the extracted rules mimic the RNN closely. The results are also somewhat surprising, since most of the rules consist only of two or three words.
It would have been interesting to try extend the approach on other natural language processing tasks, such as machine translation. Presumably the rules learned here will be quite different.
Other comments:
- Eq. (12) is over-parametrized with two vectors $P$ and $Q$. The same function can be computed with a single vector. This becomes clear when you divide both the numerator and denominator by $e^{P h_t}$.
- Section 4.1. Is it correct that this section is focused on the forward LSTM? If so, please clarify it in the text.
- In Eq. (13), define $c_0 = 0$.
- Eq. (13) is exactly the same as Eq. (15). Is there a mistake?
- In Table 1, third column should have word "film" highlighted.
- "are shown in 2" -> "are shown in Table 2".
- Since there are some problems representing numbers, it may help to replace each digit with the hashtag symbol #.
Thank you for your helpful question and review! I've replied to your points below, and hope that you enjoy our new findings.
Q: It would have been interesting to try extend the approach on other natural language processing tasks, such as machine translation. Presumably the rules learned here will be quite different.
A: In response to your question, we added results on sentiment analysis, and reformulated our framework into a general document classification setting.
Although I agree machine translation would be interesting, it's complexity would necessitate more space than a conference paper affords.
However, with our new results, we feel that we have covered a sufficiently broad class of problems.
Q: - Eq. (12) is over-parametrized with two vectors $P$ and $Q$. The same function can be computed with a single vector. This becomes clear when you divide both the numerator and denominator by $e^{P h_t}$.
A: A valid point.
However, that over-parametrized form of the softmax is a reasonably standard layer used in most classification neural nets, so we feel we are justified in sticking to standard practices.
Q: - Section 4.1. Is it correct that this section is focused on the forward LSTM? If so, please clarify it in the text.
A: It was, but for simplicity we have now removed bidirectional LSTMs from the paper.
Q: - In Eq. (13), define $c_0 = 0$.
A: Good point, we added this in 3.1 in the revision
Q: - Eq. (13) is exactly the same as Eq. (15). Is there a mistake?
A: The indices are a little hard to keep track of, but the two equations are actually different, as reflected in the former equation (16), now equation (11). The indices have been simplified in the revision, and are hopefully more reader-friendly.
Q: - In Table 1, third column should have word "film" highlighted.
- "are shown in 2" -> "are shown in Table 2".
A: Both good points. Table 1 no longer exists, but we have fixed the references.
Q: - Since there are some problems representing numbers, it may help to replace each digit with the hashtag symbol #.
A: This would have been a good point for our prior model, but our new approach seems to do a much better job at dealing with numbers, with an increased accuracy in the movie to year category of 84% from 64%.
This work proposes a pattern extraction method to both understand what a trained LSTM has learnt and to allow implementation of a hand-coded algorithm that performs similarly to the LSTM. Good results are shown on one dataset for one model architecture so it is unclear how well this approach will generalize, however, it seems it will be a useful way to understand and debug models.
The questions in WikiMovies seem to be generated from templates and so this pattern matching approach will likely work well.
However, from the experiments it's not clear if this will extend to other types of Q&A tasks where the answer may be free form text and not be a substring in the document. Is the model required to produce a continuous span over the original document?
The approach also seems to have some deficiencies in how it handles word types such as numbers or entity names. This can be encoded in the embedding for the word but from the description of the algorithm, it seems that the approach requires an entity detector. Does this mean that the approach is unable to determine when it has reached an entity from the decomposition of the output of the LSTM? The results where 'manual pattern matching' where explicit year annotations are used, seem to show that the automatic method is unable to deal with word types.
It would also be good to see an attention model as a baseline in addition to the gradient-based baseline.
Minor comments:
- P and Q seem to be undefined.
- Some references seem to be bad, e.g. in section 5.1: 'in 1' instead of 'in table 1'.
Similarly above section 7: 'as shown in 3' and in section 7.1.
- In the paragraph above section 6.3: 'adam' -> 'Adam'.
Thanks for the review! I've replied to your concerns below
Q: Good results are shown on one dataset for one model architecture so it is unclear how well this approach will generalize
A: An excellent point, which our other reviewers also pointed out. In our update, we give a general framework for document classification, and show additional results on a standard LSTM for 2 different sentiment analysis datasets. We feel these are more convincing as to the generality of our approach.
Q: The questions in WikiMovies seem to be generated from templates and so this pattern matching approach will likely work well.
However, from the experiments it's not clear if this will extend to other types of Q&A tasks where the answer may be free form text and not be a substring in the document. Is the model required to produce a continuous span over the original document?
A: I agree that it would be an interesting direction for future work to extend this to QA datasets with free form questions and/or answers, such as the Stanford QA Dataset, but it probably lies outside the scope of this paper.
However, the problem of answering non-freeform questions is still an open, actively studied problem in it's own right.
In addition to WikiMovies from EMNLP 2016, another recent example of this is the WikiReading paper in ACL 2016. Combined with our new results on sentiment, we feel that our results cover a sufficiently broad scope of problems.
Q: The approach also seems to have some deficiencies in how it handles word types such as numbers or entity names. This can be encoded in the embedding for the word but from the description of the algorithm, it seems that the approach requires an entity detector. Does this mean that the approach is unable to determine when it has reached an entity from the decomposition of the output of the LSTM? The results where 'manual pattern matching' where explicit year annotations are used, seem to show that the automatic method is unable to deal with word types.
A: As we made clearer in our revision, WikiMovies provides a list of entities, allowing us to only check whether a subset of the words are answers.
However, for many entities, our approach does demonstrate an ability to identify them as important phrases. Generally, in the movie to X questions, many instances of X (actors, directors, languages, etc) emerge as top patterns, along with the general entity character.
For instance, in table 4, starring Ben Afleck emerges as a pattern for identifying actors, and many language names emerge when searching for languages.
The point regarding the explicit year annotations is valid, but seems to have been resolved by our updated approach, as reflected by the automatic movie to year scores at 84%, very close to the 87% accuracy that the manual pattern matching achieved, and a substantial boost from the 65% the prior automatic approach yielded.
Q: It would also be good to see an attention model as a baseline in addition to the gradient-based baseline.
A:The primary goal of this paper is to provide an approach for interpreting existing models, not devise new ones. For the problems we consider, attention is not generally used, and we don't feel it is reasonable to add additional complexity onto a model in order to make it more interpretable.
Q: P and Q seem to be undefined.
A: We have fixed this - P and Q have been replaced by W_1 and W_2, denoting the rows of the cell to output matrix
Q:
- Some references seem to be bad, e.g. in section 5.1: 'in 1' instead of 'in table 1'.
Similarly above section 7: 'as shown in 3' and in section 7.1.
- In the paragraph above section 6.3: 'adam' -> 'Adam'.
A: Thank you for pointing these out, they have been fixed in the revision
Thanks for performing the additional experiments and deriving rules for classification.
However, the sentiment rules discovered seem to be significantly worse than the LSTM and CNN. In the SST dataset, a unigram Naive Bayes model (82.6%) performs significantly better than the automatic rules based classifiers (77.4%) [1], so at least for this dataset it seems that using the unigrams would yield a more interpretable and better performing rules based model vs the LSTM cell decomposition.
I will keep my previous rating, however I would like the authors to note this in the paper.
[1] Socher et al. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Thanks for your reply. We'd like to emphasize that our goal in this paper was not to achieve high predictive performance, but rather to introduce and validate a general approach for analysing the behaviour of a trained LSTM. Convincing researchers and practitioners that a particular visualization is sufficiently informative and trustworthy to actually use is not an easy or well-defined task, so we provided multiple qualitative and quantitative verifications through predictive performance, showing extracted phrases, and heat maps.
To this end, our pattern matching model is deliberately simple in order to provide better insight into the trained LSTM.
Thus, we wouldn't expect it to outperform existing, performance-optimized algorithms.
However, we do agree that naive Bayes is a pertinent baseline, so we have uploaded a revision including the result, and adding a paragraph reflecting this comment.
You seem to imply that the update was disappointing, inasmuch as our predictive performance was inferior to NB. In our update, we broadened the class of problems that can be analysed from a particular QA problem to general document classification, which is a substantial widening of the potential use cases of this work. Taking both the quantitative and qualitative evidence presented into account, we assert that this work, and the update in particular, makes a compelling case that our introduced visualizations and pattern extraction are worth using on a broad range of problems.
EDIT: the revisions made to this paper are very thorough and address many of my concerns, and the paper is also easier to understand. i recommend the latest version of this paper for acceptance and have increased my score.
This paper presents a way of interpreting LSTM models, which are notable for their opaqueness.
In particular, the authors propose decomposing the LSTM's predictions for a QA task into importance scores for words, which are then used to generate patterns that are used to find answers with a simple matching algorithm. On the WikiMovies dataset, the extracted pattern matching method achieves accuracies competitive with a normal LSTM, which shows the power of the proposed approach.
I really like the motivation of the paper, as interpreting LSTMs is definitely still a work-in-progress, and the high performance of the pattern matching was surprising.
However, several details of the pattern extraction process are not very clear, and  the evaluation is conducted on a very specific task, where predictions are made at every word.
As such, I recommend the paper in its current form as a weak accept but hope that the authors clarify their approach, as I believe the proposed method is potentially useful for NLP researchers.
Comments:
- Please introduce in more detail the specific QA tasks you are applying your models on before section 3.3, as it's not clear at that point that the answer is an entity within the document.
- 3.3: is the softmax predicting a 0/1 value (e.g., is this word the answer or not?)
- 3.3: what are the P and Q vectors? do you just mean that you are transforming the hidden state into a 2-dimensional vector for binary prediction?
- how does performance of the pattern matching change with different cutoff constant values?
- 5.2: are there questions whose answers are not entities?
- how could the proposed approach be used when predictions aren't made at every word? is there any extension for, say, sentence-level sentiment classification?
Thanks for the helpful comments! Many of your concerns were incorporated into our update, I'll address them below.
Q:
However, several details of the pattern extraction process are not very clear, and  the evaluation is conducted on a very specific task, where predictions are made at every word.
how could the proposed approach be used when predictions aren't made at every word? is there any extension for, say, sentence-level sentiment classification?
A:
In response to your concerns, we included extensions for sentence-level, and multi-sentence (Yelp) sentiment classification, and phrased the pattern extraction in terms of general document classification, a very general task. Doing so required forced us to come up with a cleaner pattern extraction process, which we hope is better communicated in the new writeup. We'd welcome comments on the presentation.
Q:
- Please introduce in more detail the specific QA tasks you are applying your models on before section 3.3, as it's not clear at that point that the answer is an entity within the document.
A:
Although the sections got shuffled around in the update, we now introduce the dataset, followed by the LSTM-model, followed by the pattern extraction modifications, so that the flow of information should be clear.
Q:
- 3.3: what are the P and Q vectors? do you just mean that you are transforming the hidden state into a 2-dimensional vector for binary prediction?
A:
Yes, that is exactly what we mean. To make this clearer, we denoted the transform matrix as W and replaced P and Q with W_1 and W_2.
Q:
- how does performance of the pattern matching change with different cutoff constant values?
A:
This is a valid concern for our prior model, where the cutoff played a relatively critical role. Under the new framework, although we do still have a cutoff, it plays a different, lesser role, as it serves to limit the amount of computation required by our search process. If you think this is still relevant in our new model, I could to run some experiments to figure out, although I strongly suspect the changes would be minimal.
Q:
- 5.2: are there questions whose answers are not entities?
A:
No, there are no such questions. Part of the WikiMovies dataset is a list of ~43k "entities" which contains all possible answers. This was not properly communicated in the original work - we have fixed this in the update.
In response to helpful comments from reviewers, we have just uploaded a revision. The main changes are as follows
- In response to requests for extensions to other datasets, we now have results on 2 different binary sentiment analysis datasets - Stanford Sentiment Treebank and Yelp reviews
- We introduced a simpler, more general approach for extracting rules from LSTMs trained on document classification, and demonstrate that with some easy modifications it can replace our prior, more complex, rule extraction mechanism on WikiMovies.
- Our rules now take the form of simple phrases, rather than allowing for variable-sized gaps between words as before
- Our LSTM baseline on WikiMovies is now SOTA by nearly 4%, and the automatically extracted patterns outperform the manual patterns in the earlier version.
- We added a discussion on instances correctly classified by the LSTM, but incorrectly classified by our rules-based algorithm
- For simplicity, we changed our WikiMovies baseline to a unidirectional LSTM, and removed bidirectional LSTMs from the paper. Extension of our new approach to bidirectional LSTMs would be straightforward, but we feel would add unneeded complexity to the presentation
All told, we feel that these algorithms and results are simpler, more powerful and more general than our prior work, and we look forward to discussing them.

Meta-review:
role: PC
data: comment: Timely topic (interpretability of neural models for NLP), interesting approach, surprising results.

Decision: Accept (Poster)