Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: How do the lengths of the sequences affect the performance of the matching models? Especially when there is a big difference between the lengths of the two sequences to be matched?
review_title: Lengths of the sequences

final_score: rating: null
review_content: I'm having a hard time understanding Equation 1. You refer to a recurrent neural network, but the equations don't show any time index to recur over.

Are A and Q (bold capital letters, no bar) the raw word vectors, or the output of some earlier recurrent component that does preprocessing?

Do you actually use an RNN, or do you just mean that Equation 1 is inspired by an RNN?

If you don't, do you capture word order in any way?
review_title: Clarification for Sec. 2.2/Eqn. 1

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a compare-aggregate framework that performs word-level matching followed by aggregation with convolutional neural networks. It compares six different comparison functions and evaluates them on four datasets. Extensive experimental results have been reported and compared against various published baselines.

The paper is well written overall.

A few detailed comments:
* page 4, line5: including a some -> including some
* What's the benefit of the preprocessing and attention step? Can you provide the results without it?
* Figure 2 is hard to read, esp. when on printed hard copy. Please enhance the quality.

review_title: A solid empirical study

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents a general approach to modeling for natural language understanding problems with two distinct textual inputs (such as a question and a source text) that can be aligned in some way. In the approach, soft attention is first used to derive alignments between the tokens of the two texts, then a comparison function uses the resulting alignments (represented as pairs of attention queries and attention results) to derive a representations that are aggregated by CNN into a single vector from which an output can be computed. The paper both presents this as an overall modeling strategy that can be made to work quite well, and offers a detailed empirical analysis of the comparison component of the model.

This work is timely. Language understanding problems of this kind are a major open issue in NLP, and are just at the threshold of being addressable with representation learning methods. The work presents a general approach which is straightforward and reasonable, and shows that it can yield good results. The work borders on incremental (relative to their earlier work or that of Parikh et al.), but it contributes in enough substantial ways that I'd strongly recommend acceptance.

Detail: 
- The model, at least as implemented for the problems with longer sequences (everything but SNLI), is not sensitive to word order. It is empirically competitive, but this insensitivity places a strong upper bound on its performance. The paper does make this clear, but it seems salient enough to warrant a brief mention in the introduction or discussion sections.
- If I understand correctly, your attention strategy is based more closely on the general/bilinear strategy of Luong et al. '15 than it is on the earlier Bahdanau work. You should probably cite the former (or some other more directly relevant reference for that strategy).
- Since the NTN risks overfitting because of its large number of parameters, did you try using a version with input dimension l and a smaller output dimension m (so an l*l*m tensor)?
- You should probably note that SubMultNN looks a lot like the strategy for *sentence*-level matching in the Lili Mou paper you cite.
- Is there a reason you use the same parameters for preprocessing the question and answer in (1)? These could require different things to be weighted highly.

review_title: Effective model design, great evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a compare-aggregate model for the NLP tasks that require semantically comparing the text sequences, such as question answering and textual entailment.
The basic framework of this model is to apply a convolutional neural network (aggregation) after a element-wise operation (comparison) over the attentive outputs of the LSTMs. 
The highlighted part is the comparison, where this paper compares several different methods for matching text sequences, and the element-wise subtraction/multiplication operations are demonstrated to achieve generally better performance on four different datasets.
While the weak point is that this is an incremental work and a bit lack of innovation. A qualitative evaluation about how subtraction, multiplication and other comparison functions perform on varied kinds of sentences would be more interesting.   


review_title: Official Review

Meta-review: This paper proposes a framework whereby, to an attention mechanism relating one text segment to another piecewise, an aggregation mechanism is added to yield an architecture matching words of one segment to another. Different vector comparison operations are explored in this framework. The reviewers were satisfied that this work is relevant, timely, clearly presented, and that the empirical validation was sound. 
Decision: Accept (Poster)