Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposed a dynamic coattention network for the question answering task with long contextual documents. 
The model is able to encode co-dependent representations of the question and the document, and a dynamic decoder iteratively pointing the potential answer spans to locate the final answer. 

Overall, this is a well-written paper. 
Although the model is a bit complicated (coattention encoder, iterative dynamic pointering decoder and highway maxout network), the intuitions behind and the details of the model are clearly presented. 
Also the performance on the SQuAD dataset is good. 
I would recommend this paper to be accepted.


review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The paper proposes a novel deep neural network architecture for the task of question answering on the SQuAD dataset. The model consists of two main components -- coattention encoder and dynamic pointer decoder. The encoder produces attention over the question as well as over the document in parallel and thus learns co-dependent representations of the question and the document. The decoder predicts the starting and the end token of the answer iteratively, with the motivation that multiple iterations will help the model escape local maxima and thus will reduce the errors made by the model. The proposed model achieved state-of-art result on SQuAD dataset at the time of writing the paper. The paper reports some analyses of the results such as performance across question types, document, question, answer lengths, etc. The paper also performs some ablation studies such as performing only single round of iteration on decoder, etc.

Strengths:

1. The paper is well-motivated with two main motivations -- co-attending to the document and the question, and iteratively producing the answer.

2. The proposed model architecture is novel and the design choices made seem reasonable.

3. The experiments show that the proposed model outperforms the existing model (at the time of writing the paper) on the SQuAD dataset by significant margin.

4. The analyses of the results and the ablation studies performed (as per someone's request) provide insights into the various modelling design choices made.

Weaknesses/Questions/Suggestions:

1. In order to gain insights into how much each additional iteration in the decoder help, I would like to see the following -- for every iteration report the mean F1 for questions that converged in that iteration along with the number of questions that converged in that iteration.

2. Example of Question 3 in figure 5 is an interesting example where the model is unable to decide between multiple local maxima despite several iterations. Could authors please report how often this happens?

3. In order to estimate how much modelling of attention in the encoder helps, it would be good if authors could report the performance of the model when attention is not modeled at all in the encoder (neither over question, nor over document).

4. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.

5. In Wang and Jiang (2016), the attention is predicted over question for each word in the document. But in table 2, when performing ablation study to make the proposed model similar to Wang and Jiang, C^D is set to C^Q. But isn’t C^Q attention over document for each word in the question? So, how is this similar to Wang and Jiang’s attention? I think QA^D will be similar to Wang and Jiang's attention since QA^D is attention over question for each word in the document. Please clarify.

6. In section 2.1, “n” and “m” are swapped when explaining the Document and Question encoding matrix. Please fix it.

Review Summary: The paper presents a novel and interesting model for the task of question answering on SQuAD dataset and shows that the model outperforms existing models. However, to gain more insights into the functioning of the model, I would like see more analyses of the results and one more ablation study (see weaknesses section above).

review_title: Interesting model but need some more analyses

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
Paper Summary: 
The paper introduces a question answering model called Dynamic Coattention Network (DCN). It extracts co-dependent representations of the document and question, and then uses an iterative dynamic pointing decoder to predict an answer span. The proposed model achieves state-of-the-art performance, outperforming all published models.

Paper Strengths: 
-- The proposed model introduces two new concepts to QA models -- 1) using attention in both directions, and 2) a dynamic decoder which iterates over multiple answer spans until convergence or maximum number of iterations.
-- The paper also presents ablation study of the proposed model which shows the importance of their design choices.
-- It is interesting to see the same idea of co-attention performing well in 2 different domains -- Visual Question Answering and machine reading comprehension.
-- The performance breakdown over document and question lengths (Figure 6) strengthens the importance of attention for QA task.
-- The proposed model achieves state-of-the-art result on SQuAD dataset.
-- The model architecture has been clearly described.

Paper Weaknesses / Future Thoughts: 
-- The paper provides model's performance when the maximum number of iterations is 1 and 4. I would like to see how the performance of the model changes with the number of iterations, i.e., the model performance when that number is 2 and 3. Is there a clear trend? What type of questions is the model able to get correct with more iterations?
-- As with many deep learning approaches, the overall architecture seems quite complex, and the design choices seem to be driven by performance numbers. As future work, authors might try to analyze qualitative advantages of different choices in the proposed model. What type of questions are correctly answered because of co-attention mechanism instead of attention in a single direction, when using Maxout Highway Network instead of a simple MLP, etc?

Preliminary Evaluation: 
Novel and state-of-the-art question answering approach. Model is clearly described in detail. In my thoughts, a clear accept.
review_title: Review

final_score: rating: null
review_content: The paper mentioned the dynamic decoder is able to escape the initial local maxima several times. I wonder how serious the issue is? If the encoder learns good enough representations, it is able to predict the answer span in one time. If it really needs several iterations to localize, why it must be done in decoder? 
review_title: escape the local maxima

final_score: rating: null
review_content: 1) What is the reason behind allowing some variation between the question encoding space and the document encoding space? What happens if the non-linear projection layer on top of the question embedding is removed?

2) How does the performance (EM and F1) change with respect to iterations in dynamic decoder?

3) Related work talks about previous Statistical QA and Neural QA works. How does the proposed approach relate to / different from all the mentioned approaches?  
review_title: Clarification questions

final_score: rating: null
review_content: Dear Authors,

Could you please clarify how do you combine the cross entropy loss of the start and end points across all iterations? A mathematical formulation of the final objective would be helpful. 

Thanks.
review_title: Objective Function

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. All reviewers agree that this is a good piece of work that should be accepted to ICLR. Authors are encouraged to incorporate reviewer feedback to further strengthen the paper.
Decision: Accept (Poster)