CA-RNN: Using Context-Aligned Recurrent Neural Networks for Modeling Sentence Similarity

The recurrent neural networks (RNNs) have shown good performance for sentence similarity modeling in recent years. Most RNNs focus on modeling the hidden states based on the current sentence, while the context information from the other sentence is not well investigated during the hidden state generation. In this paper, we propose a context-aligned RNN (CA-RNN) model, which incorporates the contextual information of the aligned words in a sentence pair for the inner hidden state generation. Specifically, we first perform word alignment detection to identify the aligned words in the two sentences. Then, we present a context alignment gating mechanism and embed it into our model to automatically absorb the aligned words’ context for the hidden state update. Experiments on three benchmark datasets, namely TREC-QA and WikiQA for answer selection and MSRP for paraphrase identification, show the great advantages of our proposed model. In particular, we achieve the new state-of-the-art performance on TREC-QA and WikiQA. Furthermore, our model is comparable to if not better than the recent neural network based approaches on MSRP. Introduction and Motivation Sentence similarity modeling plays an important role in various Natural Language Processing (NLP) tasks, such as answer selection and paraphrase identification. For the answer selection task, all the candidate answers are ranked by the sentence similarity with the given question (Wang, Smith, and Mitamura 2007; Yang, Yih, and Meek 2015). As to paraphrase identification, sentence similarity is used to determine whether two sentences have the same meaning (Yin and Schütze 2015; He, Gimpel, and Lin 2015). Most traditional methods rely on the feature engineering and linguistic tools, which are labour consuming and prone to the errors of NLP tools such as dependency parsing (Yih et al. 2013; Wan et al. 2006). Recently, the recurrent neural network (RNN) based approaches have attracted more attention due to the good performance and less human interventions. Specifically, a sequential hidden states were generated and aggregated for each sentence with RNN, and the similarity score was calculated according to the hidden representations (Mueller and Thyagarajan 2016). To capture the Copyright c © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. salient information for better sentence representations, the attention based RNN models that produce a weight for each hidden state start to arouse more interest. (Santos et al. 2016) proposed the attentive pooling networks, which incorporated the word-by-word interactions for the attentive sentence representations. In (Tan et al. 2015), the representation of the question was utilized for the attentive weight generation for the answer. To the best of our knowledge, most attention based RNNs focus on generating the attentive weights after obtaining all the hidden states, while the contextual information from the other sentence is not well studied during the internal hidden state generation (Santos et al. 2016; Tan et al. 2015; Hermann et al. 2015). Noting that the inner activation units in RNN controls the information flow over a sentence, (Wang, Liu, and Zhao 2016) proposed an IARNN-GATE model, which incorporated the question representation into the active gates to influence the hidden state generation for the answer. However, it utilized all the information of the question sentence, which would bring noises if the current hidden state was not relevant to the question. To alleviate this problem, (Bahdanau, Cho, and Bengio 2014) presented an alignment model, which measured how well the input at each position matched the output for neural machine translation. Whereas, the alignment model in fact implemented the attention mechanism, and also leveraged all the input information to generate the output. Moreover, it is still unknown how to integrate the alignment information into RNN for sentence similarity modeling. In this paper, we propose a context-aligned RNN (CARNN) model, where the context information of the aligned words is incorporated into the hidden state generation. To be specific, we first perform word alignment detection to identify the aligned words that are potentially relevant in a sentence pair. Then, a context alignment gating mechanism is presented and embedded into our model, which consists of two steps, namely relevance measurement and context absorption. The relevance measurement step aims to determine how much context can be absorbed, by measuring the relevance between the other sentence and the current hidden state. In the context absorption step, the context information of the aligned words in the other sentence is absorbed for the current hidden state generation. It is worth noting that the absorbed context will be naturally propagated across The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)