Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY.
The paper propose a reading-comprehension question answering system for the recent QA task where answers of a question can be either single tokens or spans in the given text passage.
The model first encodes the passage and the query using a recurrent neural network.
With an attention mechanism the model calculates the importance of each word on the passage with respect to each word in the question.
The encoded words in the passage are concatenated with the attention; the resulting vector is re-encoded with a further RNN.
Three convolutional neural networks with different filter size (1,2,3-gram) are used to further capture local features.
Candidate answers are selected either matching POS patterns of answers in the training set or choosing all possible text span until a certain length.
Each candidate answer has three representations, one for each n-gram representation. The compatibility of these representation with the question representation is then calculated.
The scores are combined linearly and used for calculating the probability of the candidate answer being the right answer for the question.

The method is tested on the SQUAD dataset and outperforms the proposed baselines.

----------

OVERALL JUDGMENT
The method presented in this paper is interesting but not very motivated in some points.
For example, it is not explained why in the attention mechanism it is beneficial to concatenate the original passage encoding with the attention-weighted ones.
The contributions of the paper are moderately novel proposing mainly the attention mechanism and the convolutional re-encoding.
In fact, combining questions and passages and score their compatibility has became a fairly standard procedure in all QA models.

----------

DETAILED COMMENTS

Equation (13) i should be s, not s^l.

I still do not understand the sentence " the best function is to concatenate the hidden stat of the fist word in a chunk in forward RNN and that of the last word in backward RNN". The RNN is over what all the words in the chunk? in the passage? 
The answer the authors gave in the response does not clarify this point.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: SYNOPSIS: The paper proposes a new neural network-based model for reading comprehension (reading a passage of text and answering questions based on the passage). It is similar in spirit to several other recent models, with the main exception that it is able to predict answers of different lengths, as opposed to single words/tokens/entities. The authors compare their model on the Stanford Question Answering Dataset (SQuAD), and show improvements over the baselines, while apparently lagging quite far behind the current state of the art reported on the SQuAD leaderboard.

THOUGHTS: The main novelty of the method is to be able to identify phrases of different lengths as possible answers to the question. However, both approaches considered -- using a POS pattern trie tree to filter out word sequences with POS tags matching those of answers in the training set, and brute-force enumeration of all phrases up to length N -- seem somewhat orthogonal to the idea of "learning end-to-end " an answer chunk extraction model. Furthermore, as other reviews have pointed out, it seems that the linguistic features actually contribute a lot to the final accuracy (Table 3). One could argue that these are easy to obtain using standard taggers, but it takes away even more from the idea of an "end-to-end trained" system.

The paper is generally well written, but there are several crucial sections in parts describing the model where it was really hard for me to follow the descriptions. In particular, the attention mechanism seems fairly standard to me in a seq2seq sense (i.e. there is nothing architecturally novel about it, as is for instance the case with the Gated Attentive Reader). I may be missing something, but even after the clarification round I still don't understand how it is novel compared to standard attention used in for instance seq2seq models.

Finally, although the method is shown to outperform the baseline method reported in the original paper introducing the SQuAD dataset, it currently seems to be 12th (out of 15 systems) on the leaderboard (https://rajpurkar.github.io/SQuAD-explorer/). Of course, it may be that further training and hyperparameter optimizations may improve these results.

Therefore, given the lack of model novelty (based on my understanding), and the lack of strong results (based on the leaderboard), I don't feel the paper is ready in its current form to be accepted to the conference.

Note: The GRU citation should be (Cho et al., 2014), not (Bengio et al., 2015).
review_title: Review

final_score: rating: null
review_content: I don't fully understand how your attention mechanism is different compared to other approaches. Could you please elaborate?

When you enumerate chunks in the chunk representation layer, how is the max-length N chosen?

By the way, you may want to replace \cite{x} with \citep{x} for the more readable "(X et. al, YYYY)" citations, instead of the in-text "X et al, YYYY" style from the former.  I would also encourage the description of the model to be discussed in the main body of the paper, as opposed to in the Appendix.
review_title: More clarifications

final_score: rating: null
review_content: 1.	I found the descriptions of DCR not very clear. Could the authors explain how does the convolution layer work? Also, it is also not very clear how DCR dynamically constructs the chunks. 
2.	From Table 3, it seems that the additional linguistic features are very important for the performance of DCR. However, some of the recent models are able to obtain decent results without using these features. Could the authors provide some possible explanations here?

review_title: Pre-review Questions

final_score: rating: null
review_content: Hi, I have a few questions.
1) In the CNN layer wy did you use the context before the word and not a context window around it?  I am also wondering what are the benefits of the unigram filter.
2) Do you have a further RNN for each l-gram, after the CNN layer?
3) Can you speculate on the peak of performance at lenght 9 in Figure 2(a)
4) Why do not you have test set results of the last two models in Table 3?
review_title: Cnn questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed an end-to-end machine learning model called dynamic reader for the machine reading comprehension task. Compared to earlier systems, the proposed model is able to extract and rank a set of answer candidates from a given document.

There are many recent models focusing on building good question answering systems by extracting phrases from a given article. It seems that there are two different aspects that are unique in this work:

1.	The use of convolution model, and
2.	Dynamic chunking

Convolution network is often only used for modeling character-based word embeddings so I am curious about its effectiveness on representing phrases. Therefore, I wish there could be more analysis on how effective it is, as the authors do not compare the convolution framework to other alternative approaches such as LSTM. The comparisons are important, as the authors uses uni-gram, bi-gram and tri-gram information in the convolution network, and it is not clear to me that if tri-gram information is still needed for LSTM models.

The dynamic chunking is a good idea, and a very similar idea is proposed in some of the recent papers such as [Kenton et al, 16], which also targets at the same dataset. However, I would like to see more analysis on the dynamic chunking. Why this approach is a good approach for representing answer chunks? Given the representation of the chunk is constructed by the first and the end word representations generated by a convolution network, I am not sure about the ability of this representation to capture the long answer phrases.

The authors do not use character base embedding but use some of the previous trained NLP models. It would be interesting if the authors could show what are the advantages and disadvantages of using linguistic features compared to character embeddings.

In short, there are several good ideas proposed in the paper, but the lack of proper analysis make it difficult to judge how important the proposed techniques are.

review_title: Review

Meta-review: The program committee appreciates the authors' response to the clarification questions and one review. Unfortunately, reviews are not leaning sufficiently towards acceptance. Reviewers have concerns about the novelty of the approach, its effectiveness in terms of empirical performance, and lack of analysis that would help determine the main contributions of the proposed approach. Authors are strongly encouraged to incorporate reviewer feedback in future iterations of the work.
Decision: Reject