Introduction
Natural Language Inference (NLI) is a crucial subtopic in Natural Language Processing (NLP). Most studies treat NLI as a classification problem, aiming at recognizing the relation types of hypothesis-premise sentence pairs, usually including “Entailment”, “Contradiction” and “Neutral”.
NLI is also called Recognizing Textual Entailment (RTE) BIBREF0 in earlier works and a lot of statistical-based BIBREF1 and rule-based approaches BIBREF2 are proposed to solve the problem. In 2015, Bowman released the SNLI corpus BIBREF3 that provides more than 570K hypothesis-premise sentence pairs. The large-scale data of SNLI allows a Neural Network (NN) based model to perform on the NLI. Since then, a variety of NN based models have been proposed, most of which can be divided into two kinds of frameworks. The first one is based on “Siamense" network BIBREF3 , BIBREF4 . It first applies either Recurrent Neural Network (RNN) or Convolutional Neural Networks (CNN) to generates sentence representations on both premise and hypothesis, and then concatenate them for the final classification. The second one is called “matching-aggregation" network BIBREF5 , BIBREF6 . It matches two sentences at word level, and then aggregates the matching results to generate a fixed vector for prediction. Matching is implemented by several functions based on element-wise operations BIBREF7 , BIBREF8 . Studies on SNLI show that the second one performs better.
Though the second framework has made considerable success on the NLI task, there are still some limitations. First, the inference on the mixed matching feature only adopts one-pass process, which means some detailed information would not be retrieved once missing. While the multi-turn inference can overcome this deficiency and make better use of these matching features. Second, the mixed matching feature only concatenates different matching features as the input for aggregation. It lacks interaction among various matching features. Furthermore, it treats all the matching features equally and cannot assign different importance to different matching features.
In this paper, we propose the MIMN model to tackle these limitations. Our model uses the matching features described in BIBREF5 , BIBREF9 . However, we do not simply concatenate the features but introduce a multi-turn inference mechanism to infer different matching features with a memory component iteratively. The merits of MIMN are as follows:
We conduct experiments on three NLI datasets: SNLI BIBREF3 , SCITAIL BIBREF10 and MPE BIBREF11 . On the SNLI dataset, our single model achieves 88.3% in accuracy and our ensemble model achieves 89.3% in terms of accuracy, which are both comparable with the state-of-the-art results. Furthermore, our MIMN model outperforms all previous works on both SCITAIL and MPE dataset. Especially, the model gains substantial (8.9%) improvement on MPE dataset which contains multiple premises. This result shows our model is expert in aggregating the information of multiple premises.
Related Work
Early work on the NLI task mainly uses conventional statistical methods on small-scale datasets BIBREF0 , BIBREF12 . Recently, the neural models on NLI are based on large-scale datasets and can be categorized into two central frameworks: (i) Siamense-based framework which focuses on building sentence embeddings separately and integrates the two sentence representations to make the final prediction BIBREF4 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 ; (ii) “matching-aggregation” framework which uses various matching methods to get the interactive space of two input sentences and then aggregates the matching results to dig for deep information BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF8 , BIBREF6 , BIBREF23 , BIBREF24 , BIBREF18 , BIBREF25 , BIBREF26 .
Our model is directly motivated by the approaches proposed by BIBREF7 , BIBREF9 . BIBREF7 introduces the “matching-aggregation" framework to compare representations between words and then aggregate their matching results for final decision.
BIBREF9 enhances the comparing approaches by adding element-wise subtraction and element-wise multiplication, which further improve the performance on SNLI. The previous work shows that matching layer is an essential component of this framework and different matching methods can affect the final classification result.
Various attention-based memory neural networks BIBREF27 have been explored to solve the NLI problem BIBREF20 , BIBREF28 , BIBREF14 . BIBREF20 presents a model of deep fusion LSTMs (DF-LSTMs) (Long Short-Term Memory ) which utilizes a strong interaction between text pairs in a recursive matching memory. BIBREF28 uses a memory network to extend the LSTM architecture. BIBREF14 employs a variable sized memory model to enrich the LSTM-based input encoding information. However, all the above models are not specially designed for NLI and they all focus on input sentence encoding.
Inspired by the previous work, we propose the MIMN model. We iteratively update memory by feeding in different sequence matching features. We are the first to apply memory mechanism to matching component for the NLI task. Our experiment results on several datasets show that our MIMN model is significantly better than the previous models.
Model
In this section, we describe our MIMN model, which consists of the following five major components: encoding layer, attention layer, matching layer, multi-turn inference layer and output layer. Fig. FIGREF3 shows the architecture of our MIMN model.
We represent each example of the NLI task as a triple INLINEFORM0 , where INLINEFORM1 is a given premise, INLINEFORM2 is a given hypothesis, INLINEFORM3 and INLINEFORM4 are word embeddings of r-dimension. The true label INLINEFORM5 indicates the logical relationship between the premise INLINEFORM6 and the hypothesis INLINEFORM7 , where INLINEFORM8 . Our model aims to compute the conditional probability INLINEFORM9 and predict the label for examples in testing data set by INLINEFORM10 .
Encoding Layer
In this paper, we utilize a bidirectional LSTM (BiLSTM) BIBREF29 as our encoder to transform the word embeddings of premise and hypothesis to context vectors. The premise and the hypothesis share the same weights of BiLSTM. DISPLAYFORM0
where the context vectors INLINEFORM0 and INLINEFORM1 are the concatenation of the forward and backward hidden outputs of BiLSTM respectively. The outputs of the encoding layer are the context vectors INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 is the number of hidden units of INLINEFORM5 .
Attention Layer
On the NLI task, the relevant contexts between the premise and the hypothesis are important clues for final classification. The relevant contexts can be acquired by a soft-attention mechanism BIBREF30 , BIBREF31 , which has been applied to a bunch of tasks successfully. The alignments between a premise and a hypothesis are based on a score matrix. There are three most commonly used methods to compute the score matrix: linear combination, bilinear combination, and dot product. For simplicity, we choose dot product in the following computation BIBREF8 . First, each element in the score matrix is computed based on the context vectors of INLINEFORM0 and INLINEFORM1 as follows: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are computed in Equations ( EQREF5 ) and (), and INLINEFORM2 is a scalar which indicates how INLINEFORM3 is related to INLINEFORM4 .
Then, we compute the alignment vectors for each word in the premise and the hypothesis as follows: DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 is the weighted summaries of thehypothesis in terms of each word in the premise. The same operation is applied to INLINEFORM1 . The outputs of this layer are INLINEFORM2 and INLINEFORM3 . For the context vectors INLINEFORM4 , the relevant contexts in the hypothesis INLINEFORM5 are represented in INLINEFORM6 . The same is applied to INLINEFORM7 and INLINEFORM8 .
Matching Layer
The goal of the matching layer is to match the context vectors INLINEFORM0 and INLINEFORM1 with the corresponding aligned vectors INLINEFORM2 and INLINEFORM3 from multi-perspective to generate a matching sequence.
In this layer, we match each context vector INLINEFORM0 against each aligned vector INLINEFORM1 to capture richer semantic information. We design three effective matching functions: INLINEFORM2 , INLINEFORM3 and INLINEFORM4 to match two vectors BIBREF32 , BIBREF5 , BIBREF9 . Each matching function takes the context vector INLINEFORM5 ( INLINEFORM6 ) and the aligned vector INLINEFORM7 ( INLINEFORM8 ) as inputs, then matches the inputs by an feed-forward network based on a particular matching operation and finally outputs a matching vector. The formulas of the three matching functions INLINEFORM9 , INLINEFORM10 and INLINEFORM11 are described in formulas ( EQREF11 ) () (). To avoid repetition, we will only describe the application of these functions to INLINEFORM12 and INLINEFORM13 . The readers can infer these equations for INLINEFORM14 and INLINEFORM15 . DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 represent concatenation, subtraction, and multiplication respectively, INLINEFORM3 , INLINEFORM4 and INLINEFORM5 are weight parameters to be learned, and INLINEFORM6 are bias parameters to be learned. The outputs of each matching function are INLINEFORM7 , which represent the matching result from three perspectives respectively. After matching the context vectors INLINEFORM8 and the aligned vectors INLINEFORM9 by INLINEFORM10 , INLINEFORM11 and INLINEFORM12 , we can get three matching features INLINEFORM13 , INLINEFORM14 and INLINEFORM15 .
After matching the context vectors INLINEFORM0 and the aligned vectors INLINEFORM1 by INLINEFORM2 , INLINEFORM3 and INLINEFORM4 , we can get three matching features INLINEFORM5 , INLINEFORM6 and INLINEFORM7 . where INLINEFORM8 , INLINEFORM9 , and INLINEFORM10 .
The INLINEFORM0 can be considered as a joint-feature of combing the context vectors INLINEFORM1 with aligned vectors INLINEFORM2 , which preserves all the information. And the INLINEFORM3 can be seen as a diff-feature of the INLINEFORM4 and INLINEFORM5 , which preserves the different parts and removes the similar parts. And the INLINEFORM6 can be regarded as a sim-feature of INLINEFORM7 and INLINEFORM8 , which emphasizes on the similar parts and neglects the different parts between INLINEFORM9 and INLINEFORM10 . Each feature helps us focus on particular parts between the context vectors and the aligned vectors. These matching features are vector representations with low dimension, but containing high-order semantic information. To make further use of these matching features, we collect them to generate a matching sequence INLINEFORM11 . DISPLAYFORM0
where INLINEFORM0 .
The output of this layer is the matching sequence INLINEFORM0 , which stores three kinds of matching features. The order of the matching features in INLINEFORM1 is inspired by the attention trajectory of human beings making inference on premise and hypothesis. We process the matching sequence in turn in the multi-turn inference layer. Intuitively, given a premise and a hypothesis, we will first read the original sentences to find the relevant information. Next, it's natural for us to combine all the parts of the original information and the relevant information. Then we move the attention to the different parts. Finally, we pay attention to the similar parts.
Multi-turn Inference Layer
In this layer, we aim to acquire inference outputs by aggregating the information in the matching sequence by multi-turn inference mechanism. We regard the inference on the matching sequence as the multi-turn interaction among various matching features. In each turn, we process one matching feature instead of all the matching features BIBREF9 , BIBREF26 . To enhance the information interaction between matching features, a memory component is employed to store the inference information of the previous turns. Then, the inference of each turn is based on the current matching feature and the memory. Here, we utilize another BiLSTM for the inference. DISPLAYFORM0
where INLINEFORM0 is an inference vector in the current turn, INLINEFORM1 is the index current turn, INLINEFORM2 , INLINEFORM3 is a memory vector stores the historical inference information, and INLINEFORM4 is used for dimension reduction.
Then we update the memory by combining the current inference vector INLINEFORM0 with the memory vector of last turn INLINEFORM1 . An update gate is used to control the ratio of current information and history information adaptively BIBREF33 . The initial values of all the memory vectors are all zeros. DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters to be learned, and INLINEFORM2 is a sigmoid function to compress the ratio between 0-1. Finally, we use the latest memory matrix INLINEFORM3 as the inference output of premise INLINEFORM4 . Then we calculate INLINEFORM5 in a similar way. The final outputs of this layer are INLINEFORM6 and INLINEFORM7 . DISPLAYFORM0
where INLINEFORM0 stores the inference results of all matching features. The final outputs of multi-turn inference layer are INLINEFORM1 and INLINEFORM2 . The calculation of INLINEFORM3 is the same as INLINEFORM4 .
Output Layer
The final relationship judgment depends on the sentence embeddings of premise and hypothesis. We convert INLINEFORM0 and INLINEFORM1 to sentence embeddings of premise and hypothesis by max pooling and average pooling. Next, we concatenate the two sentence embeddings to a fixed-length output vector. Then we feed the output vector to a multilayer perceptron (MLP) classifier that includes a hidden layer with INLINEFORM2 activation and a softmax layer to get the final prediction. The model is trained end-to-end. We employ multi-class cross-entropy as the cost function when training the model.
Data
To verify the effectiveness of our model, we conduct experiments on three NLI datasets. The basic information about the three datasets is shown in Table TABREF19 .
The large SNLI BIBREF3 corpus is served as a major benchmark for the NLI task. The MPE corpus BIBREF11 is a newly released textual entailment dataset. Each pair in MPE consists of four premises, one hypothesis, and one label, which is different from the standard NLI datasets. Entailment relationship holds if the hypothesis comes from the same image as the four premises. The SCITAIL BIBREF10 is a dataset about science question answering. The premises are created from relevant web sentences, while hypotheses are created from science questions and the corresponding answer candidates.
 Models for Comparison
We compare our model with “matching-aggregation” related and attention-based memory related models. In addition, to verify the effectiveness of these major components in our model, we design the following model variations for comparison: ESIM is considered as a typical model of “matching-aggregation”, so we choose ESIM as the principal comparison object. We choose the LSTMN model with deep attention fusion as a complement comparison, which is a memory related model. Besides above models, following variants of our model are designed for comparing:
ESIM We choose the ESIM model as our baseline. It mixes all the matching feature together in the matching layer and then infers the matching result in a single-turn with a BiLSTM.
600D MIMN: This is our main model described in section SECREF3 .
600D MIMN-memory: This model removes the memory component. The motivation of this experiment is to verify whether the multiple turns inference can acquire more sufficient information than one-pass inference. In this model, we process one matching feature in one iteration. The three matching features are encoded by INLINEFORM0 in multi-turns iteratively without previous memory information. The output of each iteration is concatenated to be the final output of the multi-turn inference layer: Then the Equation ( EQREF14 ) and ( EQREF16 ) are changed into Equation ( EQREF24 ) and () respectively and the Equation ( EQREF15 ) is removed. DISPLAYFORM0
600D MIMN-gate+ReLU : This model replaces the update gate in the memory component with a ReLU layer. The motivation of this model is to verify the effectiveness of update gate for combining current inference result and previous memory. Then the Equation ( EQREF15 ) is changed into Equation ( EQREF26 ). INLINEFORM0 stays the same as Equations ( EQREF16 ). DISPLAYFORM0
Experimental Settings
We implement our model with Tensorflow BIBREF34 . We initialize the word embeddings by the pre-trained embeddings of 300D GloVe 840B vectors BIBREF35 . The word embeddings of the out-of-vocabulary words are randomly initialized. The hidden units of INLINEFORM0 and INLINEFORM1 are 300 dimensions. All weights are constrained by L2 regularization with the weight decay coefficient of 0.0003. We also apply dropout BIBREF36 to all the layers with a dropout rate of 0.2. Batch size is set to 32. The model is optimized with Adam BIBREF37 with an initial learning rate of 0.0005, the first momentum of 0.9 and the second of 0.999. The word embeddings are fixed during all the training time. We use early-stopping (patience=10) based on the validation set accuracy. We use three turns on all the datasets. The evaluation metric is the classification accuracy. To help duplicate our results, we will release our source code at https://github.com/blcunlp/RTE/tree/master/MIMN.
Experiments on SNLI
Experimental results of the current state-of-the-art models and three variants of our model are listed in Table TABREF29 . The first group of models (1)-(3) are the attention-based memory models on the NLI task. BIBREF20 uses external memory to increase the capacity of LSTMs. BIBREF14 utilizes an encoding memory matrix to maintain the input information. BIBREF28 extends the LSTM architecture with a memory network to enhance the interaction between the current input and all previous inputs.
The next group of models (4)-(12) belong to the “matching-aggregation” framework with bidirectional inter-attention. Decomposable attention BIBREF8 first applies the “matching-aggregation” on SNLI dataset explicitly. BIBREF5 enriches the framework with several comparison functions. BiMPM BIBREF6 employs a multi-perspective matching function to match the two sentences. BiMPM BIBREF6 does not only exploit a multi-perspective matching function but also allows the two sentences to match from multi-granularity. ESIM BIBREF9 further sublimates the framework by enhancing the matching tuples with element-wise subtraction and element-wise multiplication. ESIM achieves 88.0% in accuracy on the SNLI test set, which exceeds the human performance (87.7%) for the first time. BIBREF18 and BIBREF1 both further improve the performance by taking the ESIM model as a baseline model. The studies related to “matching-aggregation” but without bidirectional interaction are not listed BIBREF19 , BIBREF7 .
Motivated by the attention-based memory models and the bidirectional inter-attention models, we propose the MIMN model. The last group of models (13)-(16) are models described in this paper. Our single MIMN model obtains an accuracy of 88.3% on SNLI test set, which is comparable with the current state-of-the-art single models. The single MIMN model improves 0.3% on the test set compared with ESIM, which shows that multi-turn inference based on the matching features and memory achieves better performance. From model (14), we also observe that memory is generally beneficial, and the accuracy drops 0.8% when the memory is removed. This finding proves that the interaction between matching features is significantly important for the final classification. To explore the way of updating memory, we replace the update gate in MIMN with a ReLU layer to update the memory, which drops 0.1%.
To further improve the performance on SNLI dataset, an ensemble model MIMN is built for comparison. We design the ensemble model by simply averaging the probability distributions BIBREF6 of four MIMN models. Each of the models has the same architecture but initialized by different seeds. Our ensemble model achieves the state-of-the-art performance by obtains an accuracy of 89.3% on SNLI test set.
Experiments on MPE
The MPE dataset is a brand-new dataset for NLI with four premises, one hypothesis, and one label. In order to maintain the same data format as other textual entailment datasets (one premise, one hypothesis, and one label), we concatenate the four premises as one premise.
Table TABREF31 shows the results of our models along with the published models on this dataset. LSTM is a conditional LSTM model used in BIBREF19 . WbW-Attention aligns each word in the hypothesis with the premise. The state-of-the-art model on MPE dataset is SE model proposed by BIBREF11 , which makes four independent predictions for each sentence pairs, and the final prediction is the summation of four predictions. Compared with SE, our MIMN model obtains a dramatic improvement (9.7%) on MPE dataset by achieving 66.0% in accuracy.
To compare with the bidirectional inter-attention model, we re-implement the ESIM, which obtains 59.0% in accuracy. We observe that MIMN-memory model achieves 61.6% in accuracy. This finding implies that inferring the matching features by multi-turns works better than single turn. Compared with the ESIM, our MIMN model increases 7.0% in accuracy. We further find that the performance of MIMN achieves 77.9% and 73.1% in accuracy of entailment and contradiction respectively, outperforming all previous models. From the accuracy distributions on N, E, and C in Table TABREF31 , we can see that the MIMN model is good at dealing with entailment and contradiction while achieves only average performance on neural.
Consequently, the experiment results show that our MIMN model achieves a new state-of-the-art performance on MPE test set. Besides, our MIMN-memory model and MIMN-gate+ReLU model both achieve better performance than previous models. All of our models perform well on the entailment label, which reveals that our models can aggregate information from multiple sentences for entailment judgment.
Experiments on SCITAIL
In this section, we study the effectiveness of our model on the SCITAIL dataset. Table TABREF31 presents the results of our models and the previous models on this dataset. Apart from the results reported in the original paper BIBREF10 : Majority class, ngram, decomposable attention, ESIM and DGEM, we compare further with the current state-of-the-art model CAFE BIBREF18 .
We can see that the MIMN model achieves 84.0% in accuracy on SCITAIL test set, which outperforms the CAFE by a margin of 0.5%. Moreover, the MIMN-gate+ReLU model exceeds the CAFE slightly. The MIMN model increases 13.3% in test accuracy compared with the ESIM, which again proves that multi-turn inference is better than one-pass inference.
Conclusion
In this paper, we propose the MIMN model for NLI task. Our model introduces a multi-turns inference mechanism to process multi-perspective matching features. Furthermore, the model employs the memory mechanism to carry proceeding inference information. In each turn, the inference is based on the current matching feature and previous memory. Experimental results on SNLI dataset show that the MIMN model is on par with the state-of-the-art models. Moreover, our model achieves new state-of-the-art results on the MPE and the SCITAL datasets. Experimental results prove that the MIMN model can extract important information from multiple premises for the final judgment. And the model is good at handling the relationships of entailment and contradiction.
Acknowledgements
This work is funded by Beijing Advanced Innovation for Language Resources of BLCU, the Fundamental Research Funds for the Central Universities in BLCU (No.17PT05) and Graduate Innovation Fund of BLCU (No.18YCX010).

Question: Which matching features do they employ?
Answer: Matching features from matching sentences from various perspectives.