Venue: NAACL 2016

# Learning Natural Language Inference with LSTM

Shuohang Wang

School of Information Systems

Singapore Management University

shwang.2014@phdis.smu.edu.sg &Jing Jiang

School of Information Systems

Singapore Management University

jingjiang@smu.edu.sg

###### Abstract

Natural language inference (NLI) is a fundamentally important task in natural language processing that has many applications. The recently released Stanford Natural Language Inference (SNLI) corpus has made it possible to develop and evaluate learning-centered methods such as deep neural networks for natural language inference (NLI). In this paper, we propose a special long short-term memory (LSTM) architecture for NLI. Our model builds on top of a recently proposed neural attention model for NLI but is based on a significantly different idea. Instead of deriving sentence embeddings for the premise and the hypothesis to be used for classification, our solution uses a match-LSTM to perform word-by-word matching of the hypothesis with the premise. This LSTM is able to place more emphasis on important word-level matching results. In particular, we observe that this LSTM remembers important mismatches that are critical for predicting the contradiction or the neutral relationship label. On the SNLI corpus, our model achieves an accuracy of 86.1%, outperforming the state of the art.

## 1 Introduction

Natural language inference (NLI) is the problem of determining whether from a premise sentence \(P\) one can infer another hypothesis sentence \(H\)MacCartney (2009). NLI is a fundamentally important problem that has applications in many tasks including question answering, semantic search and automatic text summarization. There has been much interest in NLI in the past decade, especially surrounding the PASCAL Recognizing Textual Entailment (RTE) Challenge Dagan et al. (2005). Existing solutions to NLI range from shallow approaches based on lexical similarities Glickman et al. (2005) to advanced methods that consider syntax Mehdad et al. (2009), perform explicit sentence alignment MacCartney et al. (2008) or use formal logic Clark and Harrison (2009).

Recently, Bowman et al. (2015) released the Stanford Natural Language Inference (SNLI) corpus for the purpose of encouraging more learning-centered approaches to NLI. This corpus contains around 570K sentence pairs with three labels: _entailment_, _contradiction_ and _neutral_. The size of the corpus makes it now feasible to train deep neural network models, which typically require a large amount of training data. Bowman et al. (2015) tested a straightforward architecture of deep neural networks for NLI. In their architecture, the premise and the hypothesis are each represented by a sentence embedding vector. The two vectors are then fed into a multi-layer neural network to train a classifier. Bowman et al. (2015) achieved an accuracy of 77.6% when long short-term memory (LSTM) networks were used to obtain the sentence embeddings.

A more recent work by Rocktaschel et al. (2016) improved the performance by applying a neural attention model. While their basic architecture is still based on sentence embeddings for the premise and the hypothesis, a key difference is that the embedding of the premise takes into consideration the alignment between the premise and the hypothesis. This so-called _attention-weighted_ representation of the premise was shown to help push the accuracy to

83.5% on the SNLI corpus.

A limitation of the aforementioned two models is that they reduce both the premise and the hypothesis to a single embedding vector before matching them; i.e., in the end, they use two embedding vectors to perform sentence-level matching. However, not all word or phrase-level matching results are equally important. For example, the matching between stop words in the two sentences is not likely to contribute much to the final prediction. Also, for a hypothesis to _contradict_ a premise, a single word or phrase-level mismatch (e.g., a mismatch of the subjects of the two sentences) may be sufficient and other matching results are less important, but this intuition is hard to be captured if we directly match two sentence embeddings.

In this paper, we propose a new LSTM-based architecture for learning natural language inference. Different from previous models, our prediction is not based on whole sentence embeddings of the premise and the hypothesis. Instead, we use an LSTM to perform _word-by-word_ matching of the hypothesis with the premise. Our LSTM sequentially processes the hypothesis, and at each position, it tries to match the current word in the hypothesis with an attention-weighted representation of the premise. Matching results that are critical for the final prediction will be "remembered" by the LSTM while less important matching results will be "forgotten." We refer to this architecture a match-LSTM, or \(m\)LSTM for short.

Experiments show that our \(m\)LSTM model achieves an accuracy of 86.1% on the SNLI corpus, outperforming the state of the art. Furthermore, through further analyses of the learned parameters, we show that the \(m\)LSTM architecture can indeed pick up the more important word-level matching results that need to be remembered for the final prediction. In particular, we observe that good word-level matching results are generally "forgotten" but important mismatches, which often indicate a _contradiction_ or a _neutral_ relationship, tend to be "remembered."

## 2 Model

In this section, we first review LSTM. We then review the word-by-word attention model by Rocktaschel et al. (2016), which is their best performing model. Finally we present our \(m\)LSTM architecture for natural language inference.

### Background

**LSTM:** Let us first briefly review LSTM (Hochreiter and Schmidhuber, 1997). LSTM is a special form of recurrent neural networks (RNNs), which process sequence data. LSTM uses a few gate vectors at each position to control the passing of information along the sequence and thus improves the modeling of long-range dependencies. While there are different variations of LSTMs, here we present the one adopted by Rocktaschel et al. (2016). Specifically, let us use \({\bf X}=({\bf x}_{1},{\bf x}_{2},\ldots,{\bf x}_{N})\) to denote an input sequence, where \({\bf x}_{k}\in\mathbb{R}^{l}\) (\(1\leq k\leq N\)). At each position \(k\), there is a set of internal vectors, including an input gate \({\bf i}_{k}\), a forget gate \({\bf f}_{k}\), an output gate \({\bf o}_{k}\) and a memory cell \({\bf c}_{k}\). All these vectors are used together to generate a \(d\)-dimensional hidden state \({\bf h}_{k}\) as follows:

\[{\bf i}_{k} = \sigma({\bf W}^{i}{\bf x}_{k}+{\bf V}^{i}{\bf h}_{k-1}+{\bf b}^{ j}),\] \[{\bf f}_{k} = \sigma({\bf W}^{f}{\bf x}_{k}+{\bf V}^{f}{\bf h}_{k-1}+{\bf b}^{ f}),\] \[{\bf o}_{k} = \sigma({\bf W}^{o}{\bf x}_{k}+{\bf V}^{o}{\bf h}_{k-1}+{\bf b}^{ o}),\] \[{\bf c}_{k} = {\bf f}_{k}\odot{\bf c}_{k-1}+{\bf i}_{k}\odot\tanh({\bf W}^{c}{ \bf x}_{k}+{\bf V}^{c}{\bf h}_{k-1}+{\bf b}^{c}),\] \[{\bf h}_{k} = {\bf o}_{k}\odot\tanh({\bf c}_{k}), \tag{1}\]

where \(\sigma\) is the sigmoid function, \(\odot\) is the elementwise multiplication of two vectors, and all \({\bf W}^{*}\in\mathbb{R}^{d\times l}\),\({\bf V}^{*}\in\mathbb{R}^{d\times d}\) and \({\bf b}^{*}\in\mathbb{R}^{d}\) are weight matrices and vectors to be learned.

**Neural Attention Model:** For the natural language inference task, we have two sentences \({\bf X}^{\rm s}=({\bf x}^{\rm s}_{1},{\bf x}^{\rm s}_{2},\ldots,{\bf x}^{\rm s }_{M})\) and \({\bf X}^{\rm t}=({\bf x}^{\rm t}_{1},{\bf x}^{\rm t}_{2},\ldots,{\bf x}^{\rm t }_{N})\), where \({\bf X}^{\rm s}\) is the premise and \({\bf X}^{\rm t}\) is the hypothesis. Here each \({\bf x}\) is an embedding vector of the corresponding word. The goal is to predict a label \(y\) that indicates the relationship between \({\bf X}^{\rm s}\) and \({\bf X}^{\rm t}\). In this paper, we assume \(y\) is one of _entailment_, _contradiction_ and _neutral_.

Rocktaschel et al. (2016) first used two LSTMs to process the premise and the hypothesis, respectively, but initialized the second LSTM (for the hypothesis) with the last cell state of the first LSTM (for the premise). Let us use \({\bf h}^{s}_{j}\) and \({\bf h}^{t}_{k}\) to denote the resulting hidden states corresponding to 

\({\bf x}_{k}^{t}\), respectively. The main idea of the word-by-word attention model by Rocktaschel et al. (2016) is to introduce a series of attention-weighted combinations of the hidden states of the premise, where each combination is for a particular word in the hypothesis. Let us use \({\bf a}_{k}\) to denote such an _attention vector_ for word \({\bf x}_{k}^{t}\) in the hypothesis. Specifically, \({\bf a}_{k}\) is defined as follows1:

Footnote 1: We present the word-by-word attention model by Rocktaschel et al. (2016) in a different way but the underlying model is the same. Our \({\bf h}_{k}^{s}\) is their \({\bf r}_{t}\), our \({\bf H}^{s}\) (all of \({\bf h}_{j}^{s}\)) is their \({\bf Y}\), our \({\bf h}_{k}^{t}\) is their \({\bf h}_{i}\), and our \(\alpha_{k}\) is their \(\alpha_{t}\). Our presentation is close to the one by Bahdanau et al. (2015), with our attention vectors a corresponding to the context vectors \({\bf c}\) in their paper.

\[{\bf a}_{k} = \sum_{j=1}^{M}\alpha_{kj}{\bf h}_{j}^{s}, \tag{2}\]

where \(\alpha_{kj}\) is an attention weight that encodes the degree to which \({\bf x}_{k}^{t}\) in the hypothesis is aligned with \({\bf x}_{j}^{s}\) in the premise. The attention weight \(\alpha_{kj}\) is generated in the following way:

\[\alpha_{kj} = \frac{\exp(e_{kj})}{\sum_{j^{\prime}}\exp(e_{kj^{\prime}})}, \tag{3}\]

where

\[e_{kj}={\bf w}^{\sf e}\cdot\tanh({\bf W}^{s}{\bf h}_{j}^{s}+{\bf W}^{t}{\bf h }_{k}^{t}+{\bf W}^{a}{\bf h}_{k-1}^{a}). \tag{4}\]

Here \(\cdot\) is the dot-product between two vectors, the vector \({\bf w}^{\sf e}\in\mathbb{R}^{d}\) and all matrices \({\bf W}^{*}\in\mathbb{R}^{d\times d}\) contain weights to be learned, and \({\bf h}_{k-1}^{a}\) is another hidden state which we will explain below.

The attention-weighted premise \({\bf a}_{k}\) essentially tries to model the relevant parts in the premise with respect to \({\bf x}_{k}^{t}\), i.e., the \(k^{\rm th}\) word in the hypothesis. Rocktaschel et al. (2016) further built an RNN model over \(\{{\bf a}_{k}\}_{k=1}^{N}\) by defining the following hidden states:

\[{\bf h}_{k}^{a} = {\bf a}_{k}+\tanh({\bf V}^{a}{\bf h}_{k-1}^{a}), \tag{5}\]

where \({\bf V}^{a}\in\mathbb{R}^{d\times d}\) is a weight matrix to be learned. We can see that the last \({\bf h}_{N}^{a}\) aggregates all the previous \({\bf a}_{k}\) and can be seen as an attention-weighted representation of the whole premise. Rocktaschel et al. (2016) then used this \({\bf h}_{N}^{a}\), which represents the whole premise, together with \({\bf h}_{N}^{t}\), which can be approximately regarded as an aggregated representation of the hypothesis2, to predict the label \(y\).

Footnote 2: Strictly speaking, in the model by Rocktaschel et al. (2016), \({\bf h}_{N}^{t}\) encodes both the premise and the hypothesis because the two sentences are chained. But \({\bf h}_{N}^{t}\) places a higher emphasis on the hypothesis given the nature of RNNs.

### Our Model

Although the neural attention model by Rocktaschel et al. (2016) achieved better results than Bowman et al. (2015), we see two limitations. First, the model still uses a single vector representation of the premise, namely \({\bf h}_{N}^{a}\), to match the entire hypothesis. We speculate that if we instead use each of the attention-weighted representations of the premise for matching, i.e., use \({\bf a}_{k}\) at position \(k\) to match the hidden state \({\bf h}_{k}^{t}\) of the hypothesis while we go through the hypothesis, we could achieve better matching results. This can be done using an RNN which at each position takes in both \({\bf a}_{k}\) and \({\bf h}_{k}^{t}\) as its input and determines how well the overall matching of the two sentences is up to the current position. In the end the RNN will produce a single vector representing the matching of the two entire sentences.

The second limitation is that the model by Rocktaschel et al. (2016) does not explicitly allow us to place more emphasis on the more important matching results between the premise and the hypothesis and down-weight the less critical ones. For example, matching of stop words is presumably less important than matching of content words. Also, some matching results may be particularly critical for making the final prediction and thus should be remembered. For example, consider the premise "_A dog jumping for a Frisbee in the snow._" and the hypothesis "_A cat washes his face and whiskers with his front paw._" When we sequentially process the hypothesis, once we see that the subject of the hypothesis _cat_ does not match the subject of the premise _dog_, we have a high probability to believe that there is a contradiction. So this mismatch should be remembered.

Based on the two observations above, we propose to use an LSTM to sequentially match the two sentences. At each position the LSTM takes in both \({\bf a}_{k}\) and \({\bf h}_{k}^{t}\) as its input. Figure 1 gives an overview of our model in contrast to the model by Rocktaschel 

et al. (2016).

Specifically, our model works as follows. First, similar to Rocktaschel et al. (2016), we process the premise and the hypothesis using two LSTMs, but we do not feed the last cell state of the premise to the LSTM of the hypothesis. This is because we do not need the LSTM for the hypothesis to encode any knowledge about the premise but we will match the premise with the hypothesis using the hidden states of the two LSTMs. Again, we use \({\bf h}_{j}^{s}\) and \({\bf h}_{k}^{i}\) to represent these hidden states.

Next, we generate the attention vectors \({\bf a}_{k}\) similarly to Eqn (2). However, Eqn (4) will be replaced by the following equation:

\[e_{kj}={\bf w}^{e}\cdot\tanh({\bf W}^{s}{\bf h}_{j}^{s}+{\bf W}^{l}{\bf h}_{k} ^{i}+{\bf W}^{m}{\bf h}_{k-1}^{m}). \tag{6}\]

The only difference here is that we use a hidden state \({\bf h}^{\text{m}}\) instead of \({\bf h}^{\text{a}}\), and the way we define \({\bf h}^{\text{m}}\) is very different from the definition of \({\bf h}^{\text{a}}\).

Our \({\bf h}_{k}^{\text{m}}\) is the hidden state at position \(k\) generated from our _m_LSTM. This LSTM models the _matching_ between the premise and the hypothesis. Important matching results will be "remembered" by the LSTM while non-essential ones will be "forgotten." We use the concatenation of \({\bf a}_{k}\), which is the attention-weighted version of the premise for the \(k^{\rm th}\) word in the hypothesis, and \({\bf h}_{k}^{\text{t}}\), the hidden state for the \(k^{\rm th}\) word itself, as input to the _m_LSTM.

Specifically, let us define

\[{\bf m}_{k} = \begin{bmatrix}{\bf a}_{k}\\ {\bf b}_{k}^{i}\end{bmatrix}. \tag{7}\]

We then build the _m_LSTM as follows:

\[{\bf i}_{k}^{\text{m}} = \sigma({\bf W}^{\text{mi}}{\bf m}_{k}+{\bf V}^{\text{mi}}{\bf h} _{k-1}^{\text{m}}+{\bf b}^{\text{mi}}),\] \[{\bf f}_{k}^{\text{m}} = \sigma({\bf W}^{\text{mf}}{\bf m}_{k}+{\bf V}^{\text{mf}}{\bf h} _{k-1}^{\text{m}}+{\bf b}^{\text{mf}}),\] \[{\bf o}_{k}^{\text{m}} = \sigma({\bf W}^{\text{mo}}{\bf m}_{k}+{\bf V}^{\text{mo}}{\bf n} _{k-1}^{\text{m}}+{\bf b}^{\text{mo}}),\] \[{\bf c}_{k}^{\text{m}} = {\bf f}_{k}^{\text{m}}\odot{\bf c}_{k-1}^{\text{m}}+{\bf i}_{k}^{ \text{m}}\odot\tanh({\bf W}^{\text{mc}}{\bf m}_{k}+{\bf V}^{\text{mc}}{\bf h} _{k-1}^{\text{m}}\] \[+{\bf b}^{\text{mc}}),\] \[{\bf h}_{k}^{\text{m}} = {\bf o}_{k}^{\text{m}}\odot\tanh({\bf c}_{k}^{\text{m}}). \tag{8}\]

With this _m_LSTM, finally we use only \({\bf h}_{N}^{\text{m}}\), the last hidden state, to predict the label \(y\).

### Implementation Details

Besides the difference of the LSTM architecture, we also introduce a few other changes from the model by Rocktaschel et al. (2016). First, we insert a special word _NULL_ to the premise, and we allow words in the hypothesis to be aligned with this _NULL_. This is inspired by common practice in machine translation. Specifically, we introduce a vector \({\bf h}_{0}^{\text{s}}\), which is fixed to be a vector of 0s of dimension \(d\). This \({\bf h}_{0}^{\text{s}}\) represents _NULL_ and is used with other \({\bf h}_{j}^{\text{s}}\) to derive the attention vectors \(\{{\bf a}_{k}\}_{k=1}^{N}\).

Second, we use word embeddings trained from GloVe Pennington et al. (2014) instead of word2vec vectors. The main reason is that GloVe covers more words in the SNLI corpus than word2vec3.

Footnote 3: The SNLI corpus contains 37K unique tokens. Around 12.1K of them cannot be found in word2vec but only around 4.1K of them cannot be found in GloVe.

Third, for words which do not have pre-trained word embeddings, we take the average of the embeddings of all the words (in GloVe) surrounding the unseen word within a window size of 9 (4 on the left and 4 on the right) as an approximation of the embedding of this unseen word. Then we do not update any word embedding when learning our model. Although this is a very crude approximation, it reduces

Figure 1: The top figure depicts the model by Rocktäschel et al. (2016) and the bottom figure depicts our model. Here \({\bf H}^{\text{s}}\) represents all the hidden states \({\bf h}_{j}^{\text{s}}\). Note that in the top model each \({\bf h}_{k}^{\text{a}}\) represents a weighted version of the premise only, while in our model, each \({\bf h}_{k}^{\text{m}}\) represents the matching between the premise and the hypothesis up to position \(k\).

 

the number of parameters we need to update, and as it turns out, we can still achieve better performance than Rocktaschel et al. (2016).

## 3 Experiments

### Experiment Settings

* indicates a lack of consensus from the human annotators. We discard the sentence pairs labeled with
- and keep the remaining ones for our experiments. In the end, we have 549,367 pairs for training, 9,842 pairs for development and 9,824 pairs for testing. This follows the same data partition used by Bowman et al. (2015) in their experiments. We perform three-class classification and use accuracy as our evaluation metric.

**Parameters:** We use the Adam method (Kingma and Ba, 2014) with hyperparameters \(\beta_{1}\) set to 0.9 and \(\beta_{2}\) set to 0.999 for optimization. The initial learning rate is set to be 0.001 with a decay ratio of 0.95 for each iteration. The batch size is set to be 30. We experiment with \(d=150\) and \(d=300\) where \(d\) is the dimension of all the hidden states.

**Methods for comparison:** We mainly want to compare our model with the word-by-word attention model by Rocktaschel et al. (2016) because this model achieved the state-of-the-art performance on the SNLI corpus. To ensure fair comparison, besides comparing with the accuracy reported by Rocktaschel et al. (2016), we also re-implemented their model and report the performance of our implementation. We also consider a few variations of our model. Specifically, the following models are implemented and tested in our experiments:

* Word-by-word attention (\(d=150\)): This is our implementation of the word-by-word attention model by Rocktaschel et al. (2016), where we set the dimension of the hidden states to 150. The differences between our implementation and the original implementation by Rocktaschel et al. (2016) are the following: (1) We also add a _NULL_ token to the premise for matching. (2) We do not feed the last cell state of the LSTM for the premise to the LSTM for the hypothesis, to keep it consistent with the implementation of our model. (3) For word representation, we also use the GloVe word embeddings and we do not update the word embeddings. For unseen words, we adopt the same strategy as described in Section 2.3.
* _m_LSTM (\(d=150\)): This is our _m_LSTM model with \(d\) set to 150.
* _m_LSTM with bi-LSTM sentence modeling (\(d=150\)): This is the same as the model above except that when we derive the hidden states \(\mathbf{h}^{s}_{j}\) and \(\mathbf{h}^{t}_{k}\) of the two sentences, we use bi-LSTMs (Graves, 2012) instead of LSTMs. We implement this model to see whether bi-LSTMs allow us to better align the sentences.
* _m_LSTM (\(d=300\)): This is our _m_LSTM model with \(d\) set to 300.
* _m_LSTM with word embedding (\(d=300\)): This is the same as the model above except that we directly use the word embedding vectors \(\mathbf{x}^{s}_{j}\) and \(\mathbf{x}^{t}_{k}\) instead of the hidden states \(\mathbf{h}^{s}_{j}\) and \(\mathbf{h}^{t}_{k}\) in our model. In this case, each attention vector \(\mathbf{a}_{k}\) is a weighted sum of \(\{\mathbf{x}^{s}_{j}\}_{j=1}^{M}\). We experiment with this setting because we hypothesize that the effectiveness of our model is largely related to the _m_LSTM architecture rather than the use of LSTMs to process the original sentences.

### Main Results

Table 1 compares the performance of the various models we tested together with some previously reported results.

We have the following observations: (1) First of all, we can see that when we set \(d\) to 300, our model achieves an accuracy of 86.1% on the test data, which to the best of our knowledge is the highest on this data set. (2) If we compare our _m_LSTM model with our implementation of the word-by-word attention model by Rocktaschel et al. (2016) under the same setting with \(d=150\), we can see that our performance on the test data (85.7%) is higher than that of their model (82.6%). We also tested statistical significance and found the improvement to be statistically significant at the 0.001 level. (3) The performance of _m_LSTM with bi-LSTM sentence modeling compared with the model with standard LSTM sentence modeling when \(d\) is set to 150 shows that using bi-LSTM to process the original sentences helps 

(86.0% vs. 85.7% on the test data), but the difference is small and the complexity of bi-LSTM is much higher than LSTM. Therefore when we increased \(d\) to 300 we did not experiment with bi-LSTM sentence modeling. (4) Interestingly, when we experimented with the _m_LSTM model using the pre-trained word embeddings instead of LSTM-generated hidden states as initial representations of the premise and the hypothesis, we were able to achieve an accuracy of 85.3% on the test data, which is still better than previously reported state of the art. This suggests that the _m_LSTM architecture coupled with the attention model works well, regardless of whether or not we use LSTM to process the original sentences.

Because the NLI task is a three-way classification problem, to better understand the errors, we also show the confusion matrix of the results obtained by our _m_LSTM model with \(d=300\) in Table 2. We can see that there is more confusion between _neutral_ and _entailment_ and between _neutral_ and _contradiction_ than between _entailment_ and _contradiction_. This shows that _neutral_ is relatively hard to capture.

### Further Analyses

To obtain a better understanding of how our proposed model actually performs the matching between a premise and a hypothesis, we further conduct the following analyses. First, we look at the learned word-by-word alignment weights \(\alpha_{kj}\) to check whether the soft alignment makes sense. This is the same as what was done by Rocktaschel et al. (2016). We then look at the values of the various gate vectors of the _m_LSTM. By looking at these values, we aim to check (1) whether the model is able to differentiate between more important and less important word-level matching results, and (2) whether the model forgets certain matching results and remembers certain other ones.

To conduct the analyses, we choose three examples and display the various learned parameter values. These three sentence pairs share the same premise but have different hypotheses and different relationship labels. They are given in Table 3. The values of the alignment weights and the gate vectors are plotted in Figure 2.

Besides using the three examples, we will also give some overall statistics of the parameter values to confirm our observations with the three examples.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Model & \(d\) & \(|\theta|_{\text{W+M}}\) & \(|\theta|_{\text{M}}\) & Train & Dev & Test \\ \hline LSTM [Bowman et al. (2015)] & 100 & 10M & 221K & 84.4 & - & 77.6 \\ Classifier [Bowman et al. (2015)] & - & - & - & 99.7 & - & 78.2 \\ \hline LSTM shared [Rocktäschel et al. (2016)] & 159 & 3.9M & 252K & 84.4 & 83.0 & 81.4 \\ Word-by-word attention [Rocktäschel et al. (2016)] & 100 & 3.9M & 252K & 85.3 & 83.7 & 83.5 \\ \hline Word-by-word attention (our implementation) & 150 & 340K & 340K & 85.5 & 83.3 & 82.6 \\ _m_LSTM & 150 & 544K & 544K & 91.0 & 86.2 & 85.7 \\ _m_LSTM with bi-LSTM sentence modeling & 150 & 1.4M & 1.4M & 91.3 & 86.6 & 86.0 \\ _m_LSTM & 300 & 1.9M & 1.9M & 92.0 & **86.9** & **86.1** \\ _m_LSTM with word embedding & 300 & 1.3M & 1.3M & 88.6 & 85.4 & 85.3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Experiment results in terms of accuracy. \(d\) is the dimension of the hidden states. \(|\theta|_{\text{W+M}}\) is the total number of parameters and \(|\theta|_{\text{M}}\) is the number of parameters excluding the word embeddings. Note that the five models in the last section were implemented by us while the other results were taken directly from previous papers. Note also that for the five models in the last section, we do not update word embeddings so \(|\theta|_{\text{W+M}}\) is the same as \(|\theta|_{\text{M}}\). The three columns on the right are the accuracies of the trained models on the training data, the development data and the test data, respectively.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline  & \multicolumn{3}{c}{ground truth} \\ prediction & \(N\) & \(E\) & \(C\) \\ \hline \(N\) & 2628 & 286 & 255 \\ \(E\) & 340 & 3005 & 159 \\ \(C\) & 250 & 77 & 2823 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The confusion matrix of the results by _m_LSTM with \(d=300\). \(N\), \(E\) and \(C\) correspond to _neutral_, _entailment_ and _contradiction_, respectively.

 



#### Word Alignment

First, let us look at the top-most plots of Figure 2. These plots show the alignment weights \(\alpha_{kj}\) between the hypothesis and the premise, where a darker color corresponds to a larger value of \(\alpha_{kj}\). Recall that \(\alpha_{kj}\) is the degree to which the \(k^{\rm th}\) word

Figure 2: The alignment weights and the gate vectors of the three examples.

\begin{table}
\begin{tabular}{l l l} \hline  & **ID** & **sentence** & **label** \\ \hline Premise & & A dog jumping for a Frisbee in the snow. & _entailment_ \\ \hline \multirow{3}{*}{Hypothesis} & Example 1 & An animal is outside in the cold weather, playing with a plastic toy. & _entailment_ \\  & Example 2 & A cat washed his face and whiskers with his front paw. & _contradiction_ \\  & Example 3 & A pet is enjoying a game of fetch with his owner. & _neutral_ \\ \hline \end{tabular}
\end{table}
Table 3: Three examples of sentence pairs with different relationship labels. The second hypothesis is a contradiction because it mentions a completely different event. The third hypothesis is neutral to the premise because the phrase “with his owner” cannot be inferred from the premise.



in the hypothesis is aligned with the \(j^{\rm th}\) word in the premise. Also recall that the weights \(\alpha_{kj}\) are configured such that for the same \(k\) all the \(\alpha_{kj}\) add up to 1. This means the weights in the same row in these plots add up to 1. From the three plots we can see that the alignment weights generally make sense. For example, in Example 1, "animal" is strongly aligned with "dog" and "toy" aligned with "Frisbee." The phrase "cold weather" is aligned with "snow." In Example 3, we also see that "pet" is strongly aligned with "dog" and "game" aligned with "Frisbee."

In Example 2, "cat" is strongly aligned with "dog" and "washes" is aligned with "jumping." It may appear that these matching results are wrong. However, "dog" is likely the best match for "cat" among all the words in the premise, and as we will show later, this match between "cat" and "dog" is actually a strong indication of a contradiction between the two sentences. The same explanation applies to the match between "washes" and "jumping."

We also observe that some words are aligned with the _NULL_ token we inserted. For example, the word "is" in the hypothesis in Example 1 does not correspond to any word in the premise and is therefore aligned with _NULL_. The words "face" and "whiskers" in Example 2 and "owner" in Example 3 are also aligned with _NULL_. Intuitively, if some important content words in the hypothesis are aligned with _NULL_, it is more likely that the relationship label is either contradiction or neutral.

#### Values of Gate Vectors

Next, let us look at the values of the learned gate vectors of our _m_LSTM for the three examples. We show these values under the setting where \(d\) is set to 150. Each row of these plots corresponds to one of the 150 dimensions. Again, a darker color indicates a higher value.

An input gate controls whether the input at the current position should be used in deriving the final hidden state of the current position. From the three plots of the input gates, we can observe that generally for stop words such as prepositions and articles the input gates have lower values, suggesting that the matching of these words is less important. On the other hand, content words such as nouns and verbs tend to have higher values of the input gates, which also makes sense because these words are generally more important for determining the final relationship label.

To further verify the observation above, we compute the average input gate values for stop words and the other content words. We find that the former has an average value of 0.287 with a standard deviation of 0.084 while the latter has an average value of 0.347 with a standard deviation of 0.116. This shows that indeed generally stop words have lower input gate values. Interestingly, we also find that some stop words may have higher input gate values if they are critical for the classification task. For example, the negation word "not" has an average input gate value of 0.444 with a standard deviation of 0.104.

Overall, the values of the input gates confirm that the _m_LSTM helps differentiate the more important word-level matching results from the less important ones.

Next, let us look at the forget gates. Recall that a forget gate controls the importance of the _previous_ cell state in deriving the final hidden state of the current position. Higher values of a forget gate indicate that we need to remember the previous cell state and pass it on whereas lower values indicate that we should probably forget the previous cell. From the three plots of the forget gates, we can see that overall the colors are the lightest for Example 1, which is an _entailment_. This suggests that when the hypothesis is an entailment of the premise, the _m_LSTM tends to forget the previous matching results. On the other hand, for Example 2 and Example 3, which are _contradiction_ and _neutral_, we see generally darker colors. In particular, in Example 2, we can see that the colors are consistently dark starting from the word "his" in the hypothesis until the end. We believe the explanation is that after the _m_LSTM processes the first three words of the hypothesis, "A cat washes," it sees that the matching between "cat" and "dog" and between "washes" and "jumping" is a strong indication of a contradiction, and therefore these matching results need to be remembered until the end of the _m_LSTM for the final prediction.

We have also checked the forget gates of the other sentence pairs in the test data by computing the average forget gate values and the standard deviations for _entailment_, _neutral_ and _contradiction_, respectively. We find that the values are 0.446\(\pm\)0.123, 

0.507\(\pm\)0.148 and 0.536\(\pm\)0.170, respectively. For _contradiction_ and _neutral_, the forget gates start to have higher values from certain positions of the hypotheses.

Based on the observations above, we hypothesize that the way the _m_LSTM works is as follows. It remembers important mismatches, which are useful for predicting the _contradiction_ or the _neutral_ relationship, and forgets good matching results. At the end of the _m_LSTM, if no important mismatch is remembered, the final classifier will likely predict _entailment_ by default. Otherwise, depending on the kind of mismatch remembered, the classifier will predict either _contradiction_ or _neutral_.

For the output gates, we are not able to draw any important conclusion except that the output gates seem to be positively correlated with the input gates but they tend to be darker than the input gates.

## 4 Related Work

There has been much work on natural language inference. Shallow methods rely mostly on lexical similarities but are shown to be robust. For example, Bowman et al. (2015) experimented with a lexicalized classifier-based method, which only uses lexical information and achieves an accuracy of 78.2% on the SNLI corpus. More advanced methods use syntactic structures of the sentences to help matching them. For example, Mehdad et al. (2009) applied syntactic-semantic tree kernels for recognizing textual entailment. Because inference is essentially a logic problem, methods based on formal logic Clark and Harrison (2009) or natural logic MacCartney (2009) have also been proposed. A comprehensive review on existing work can be found in the book by Dagan et al. (2013).

The work most relevant to ours is the recently proposed neural attention model-based method by Rocktaschel et al. (2016), which we have detailed in previous sections. Neural attention models have recently been applied to some natural language processing tasks including machine translation Bahdanau et al. (2015), abstractive summarization Rush et al. (2015) and question answering Hermann et al. (2015). Rocktaschel et al. (2016) showed that the neural attention model could help derive a better representation of the premise to be used to match the hypothesis, whereas in our work we also use it to derive representations of the premise that are used to sequentially match the words in the hypothesis.

The SNLI corpus is new and so far it has only been used in a few studies. Besides the work by Bowman et al. (2015) themselves and by Rocktaschel et al. (2016), there are two other studies which used the SNLI corpus. Vendrov et al. (2015) used a _Skip-Thought_ model proposed by Kiros et al. (2015) to the NLI task and reported an accuracy of 81.5% on the test data. Mou et al. (2015) used trees-based CNN encoders to obtain sentence embeddings and achieved an accuracy of 82.1%.

## 5 Conclusions and Future Work

In this paper, we proposed a special LSTM architecture for the task of natural language inference. Based on a recent work by Rocktaschel et al. (2016), we first used neural attention models to derive attention-weighted vector representations of the premise. We then designed a match-LSTM that processes the hypothesis word by word while trying to match the hypothesis with the premise. The last hidden state of this _m_LSTM can be used for predicting the relationship between the premise and the hypothesis. Experiments on the SNLI corpus showed that the _m_LSTM model outperformed the state-of-the-art performance reported so far on this data set. Moreover, closer analyses on the gate vectors revealed that our _m_LSTM indeed remembers and passes on important matching results, which are typically mismatches that indicate a _contradiction_ or a _neutral_ relationship between the premise and the hypothesis.

With the large number of parameters to learn, an inevitable limitation of our model is that a large training data set is needed to learn good model parameters. Indeed some preliminary experiments applying our _m_LSTM to the SICK corpus (Marelli et al., 2014), a smaller textual entailment benchmark data set, did not give very good results. We believe that this is because our model learns everything from scratch except using the pre-trained word embeddings. A future direction would be to incorporate other resources such as the paraphrase database Ganitkevitch et al. (2013) into the learning process so that such prior knowledge can be utilized 



## References

* [Bahdanau et al.2015] Dzmitry Bahdanau, HyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the International Conference on Learning Representations_.
* Bowman et al.2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Clark and Harrison2009] Peter Clark and Phil Harrison. 2009. An inference-based approach to recognizing entailment. In _Proceedings of the Text Analysis Conference_.
* Dagan et al.2005] Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. In _Proceedings of the PASCAL Challenges Workshop on Recognizing Textual Entailment_.
* Dagan et al.2013] Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. _Recognizing Textual Entailment: Models and Applications_. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
* Ganitkevitch et al.2013] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics_.
* Glickman et al.2005] Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. Web based probabilistic textual entailment. In _Proceedings of the PASCAL Challenges Workshop on Recognizing Textual Entailment_.
* Graves2012] Alex Graves. 2012. _Supervised sequence labelling with recurrent neural networks_, volume 385. Springer.
* Hermann et al.2015] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems_.
* Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Kingma and Ba2014] Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _Proceedings of the International Conference on Learning Representations_.
* Kiros et al.2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Advances in Neural Information Processing Systems_.
* MacCartney et al.2008] Bill MacCartney, Michel Galley, and Christopher D Manning. 2008. A phrase-based alignment model for natural language inference. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* MacCartney2009] Bill MacCartney. 2009. _Natural Language Inference_. Ph.D. thesis, Stanford University.
* Marelli et al.2014] Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_.
* Mehdad et al.2009] Yashar Mehdad, Alessandro Moschitti1, and Fabio Massiomo Zanzotto. 2009. SemKer: Syntactic/semantic kernels for recognizing textual entailment. In _Proceedings of the Text Analysis Conference_.
* Mou et al.2015] Lili Mou, Men Roui, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2015. Recognizing entailment and contradiction by tree-based convolution. _arXiv preprint arXiv:1512.08422_.
* Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Rocktaschel et al.2016] Tim Rocktaschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In _Proceedings of the International Conference on Learning Representations_.
* Rush et al.2015] Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Vendrov et al.2015] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-embeddings of images and language. _arXiv preprint arXiv:1511.06361_.