Title: A Compare-Aggregate Model for Matching Text Sequences
Authors: Shuohang Wang
Jing Jiang
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Many NLP tasks including machine comprehension, answer selection and text entailment require the comparison between sequences. Matching the important units between sequences is a key to solve these problems. In this paper, we present a general "compare-aggregate" framework that performs word-level matching followed by aggregation using Convolutional Neural Networks. We particularly focus on the different comparison functions we can use to match two vectors. We use four different datasets to evaluate the model. We find that some simple comparison functions based on element-wise operations can work better than standard neural network and neural tensor network. 

TL;DR: A general "compare-aggregate" framework that performs word-level matching followed by aggregation using Convolutional Neural Networks

Review discussion:
I'm having a hard time understanding Equation 1. You refer to a recurrent neural network, but the equations don't show any time index to recur over.
Are A and Q (bold capital letters, no bar) the raw word vectors, or the output of some earlier recurrent component that does preprocessing?
Do you actually use an RNN, or do you just mean that Equation 1 is inspired by an RNN?
If you don't, do you capture word order in any way?
Thank you for the valuable comments! We have made the revisions in the new pdf version.
A and Q are the raw word vectors. Equation 1 is inspired by LSTM/GRU and it doesn't capture word order.
It would be a better way to use LSTM/GRU to chain up the words such that we can capture some contextual information, while this could be computationally expensive for long sequences. In our experiments, we only used LSTM to pre-process the sequences for the SNLI task.
How do the lengths of the sequences affect the performance of the matching models? Especially when there is a big difference between the lengths of the two sequences to be matched?
Thank you for your valuable comment!
We think making full use of the much longer sequence which includes more information will benefit the performance based on the strategy we build the models as follows： When the lengths of the two sequences have a big difference, we always build CNN on the longer sequence and it can get a much better performance than building CNN on the shorter one. (We didn't show the worse performance in this version.) For example, in the MovieQA task where plot is a very long sequence, CNN built on the plot could detect whether the words in the question and the answer appear in the same sentence of the plot.
However, building CNN on the question will not have this benefit. In the InsuranceQA task where the answers are very long sequences, the longer the sequences are, the more differences between the answers. CNN built on the answers is to detect these differences, such as the continuously matched words in the answers and even the frequency of the matched words.
Overall, we think CNN can detect the local matching information which is more fruitful in the longer sequences and that can benefit the performance.
We have also made a comparison between the performance on sequence pairs with big length differences and sequence pairs with similar lengths on the WikiQA development and test data sets. (As the other data sets are not suitable to make this comparison, we didn’t conduct any further evaluation on them.) The set of sequence pairs with big length differences has a length ratio larger than 4 while the other set of sequence pairs has a length ratio below 4. On the development set, we find that the set with big length differences (56 pairs) has a MAP score of 0.82 while the other set (70 pairs) has a MAP score of 0.69. On the test set, the set with big length differences (118 pairs) has nearly the same score as the other set (125 pairs).  Based on these results, we believe the big length difference between the two sequences in a pair does not affect the performance of our model.
The paper presents a general approach to modeling for natural language understanding problems with two distinct textual inputs (such as a question and a source text) that can be aligned in some way. In the approach, soft attention is first used to derive alignments between the tokens of the two texts, then a comparison function uses the resulting alignments (represented as pairs of attention queries and attention results) to derive a representations that are aggregated by CNN into a single vector from which an output can be computed. The paper both presents this as an overall modeling strategy that can be made to work quite well, and offers a detailed empirical analysis of the comparison component of the model.
This work is timely. Language understanding problems of this kind are a major open issue in NLP, and are just at the threshold of being addressable with representation learning methods. The work presents a general approach which is straightforward and reasonable, and shows that it can yield good results. The work borders on incremental (relative to their earlier work or that of Parikh et al.), but it contributes in enough substantial ways that I'd strongly recommend acceptance.
Detail:
- The model, at least as implemented for the problems with longer sequences (everything but SNLI), is not sensitive to word order. It is empirically competitive, but this insensitivity places a strong upper bound on its performance. The paper does make this clear, but it seems salient enough to warrant a brief mention in the introduction or discussion sections.
- If I understand correctly, your attention strategy is based more closely on the general/bilinear strategy of Luong et al. '15 than it is on the earlier Bahdanau work. You should probably cite the former (or some other more directly relevant reference for that strategy).
- Since the NTN risks overfitting because of its large number of parameters, did you try using a version with input dimension l and a smaller output dimension m (so an l*l*m tensor)?
- You should probably note that SubMultNN looks a lot like the strategy for *sentence*-level matching in the Lili Mou paper you cite.
- Is there a reason you use the same parameters for preprocessing the question and answer in (1)? These could require different things to be weighted highly.
We thank you for your valuable comments and your recommendation! We will update the paper according to your comments!
- About the word order insensitivity, we will include it in the introduction and have a discussion about it in the next version.
- About the attention strategy, yes, our method is close to (Luong et al. '15) 's general strategy and we will cite their work instead.
- About the overfitting of NTN matching method, we will try a smaller output dimension and address this issue in the experiment section.
- About the SubMultNN citing, yes, we will cite the Lili Mou's work instead for the element-wise matching methods.
- About the same parameters for preprocessing, we think the pretrained word embeddings can help the attention weight computation. By sharing the parameters, the same/similar words evaluated by the similarity of their pre-trained word embeddings can still be same/similar after the preprocessing layer and that can help get more accurate attention weights. We will add the explanation about it in the next version.
We thank you for your valuable comments again!
Q: The model, at least as implemented for the problems with longer sequences (everything but SNLI), is not sensitive to word order. It is empirically competitive, but this insensitivity places a strong upper bound on its performance. The paper does make this clear, but it seems salient enough to warrant a brief mention in the introduction or discussion sections.
A: We have added the discussion about the word order insensitivity in the analysis of experiment results in the updated version.
Q: If I understand correctly, your attention strategy is based more closely on the general/bilinear strategy of Luong et al. '15 than it is on the earlier Bahdanau work. You should probably cite the former (or some other more directly relevant reference for that strategy).
A: We have cited Luong et al. '15's work stead in the description of attention layer.
Q: Since the NTN risks overfitting because of its large number of parameters, did you try using a version with input dimension l and a smaller output dimension m (so an l*l*m tensor)?
A: We set the m to be 100 and 50 respectively, and tune the regularization in the same way, but didn't get a better performance.
Q: You should probably note that SubMultNN looks a lot like the strategy for *sentence*-level matching in the Lili Mou paper you cite.
A: We've cited Lili Mou's paper instead when describing the element-wise comparison.
This paper proposes a compare-aggregate framework that performs word-level matching followed by aggregation with convolutional neural networks. It compares six different comparison functions and evaluates them on four datasets. Extensive experimental results have been reported and compared against various published baselines.
The paper is well written overall.
A few detailed comments:
* page 4, line5: including a some -> including some
* What's the benefit of the preprocessing and attention step? Can you provide the results without it?
* Figure 2 is hard to read, esp.
when on printed hard copy. Please enhance the quality.
We thank you for your valuable comments!
Q:page 4, line5: including a some -> including some
A:We have fixed this typo. Thank you!
Q: What's the benefit of the preprocessing and attention step? Can you provide the results without it?
A: We think both the preprocessing and attention steps are to select the more important words.
We have updated the paper and show this ablation results in Table 4.
Overall, after the ablation, the performance get poorer.
For the sequence matching with big difference in length, such as the MovieQA and InsuranceQA tasks, the attention layer plays a more important role.
For the sequence matching with smaller difference in length, such as the WikiQA and SNLI tasks, the pre-processing layer plays a more important role.
Q: Figure 2 is hard to read, esp.
when on printed hard copy. Please enhance the quality.
A: We have updated the Figure, by enlarging the text and removing the dimension with small values in the visualization due to the sparsity of the convolutional layer output.
This paper proposed a compare-aggregate model for the NLP tasks that require semantically comparing the text sequences, such as question answering and textual entailment.
The basic framework of this model is to apply a convolutional neural network (aggregation) after a element-wise operation (comparison) over the attentive outputs of the LSTMs.
The highlighted part is the comparison, where this paper compares several different methods for matching text sequences, and the element-wise subtraction/multiplication operations are demonstrated to achieve generally better performance on four different datasets.
While the weak point is that this is an incremental work and a bit lack of innovation. A qualitative evaluation about how subtraction, multiplication and other comparison functions perform on varied kinds of sentences would be more interesting.
We thank you for your valuable comments!
We believe our work has shown valuable insights in sequence comparison, and it can benefit future research that needs to integrate two or three types of sequences.
About the varied kinds of sentences: We think the four tasks we used to evaluate our model covers varied kinds of sentences by the statistical analyse in Table 2. For the MovieQA task, each instance contains three types of sequences and one of the sequences is at document level. For the others, each instance contains two types of sequences. For the InsuranceQA task, one of the sequences is at document level. For the WikiQA and SNLI tasks, the length differences between the sequences comes smaller and smaller. For all these tasks, the element-wise comparisons show better performance.

Meta-review:
role: PC
data: comment: This paper proposes a framework whereby, to an attention mechanism relating one text segment to another piecewise, an aggregation mechanism is added to yield an architecture matching words of one segment to another. Different vector comparison operations are explored in this framework. The reviewers were satisfied that this work is relevant, timely, clearly presented, and that the empirical validation was sound. 

Decision: Accept (Poster)