Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposed an extension of the dynamic coattention network (DCN) with deeper residual layers and self-attention. It also introduced a mixed objective with self-critical policy learning to encourage predictions with high word overlap with the gold answer span. The resulting DCN+ model achieved significant improvement over DCN.

Strengths:
The model and the mixed objective is well-motivated and clearly explained.
Near state-of-the-art performance on SQuAD dataset (according to the SQuAD leaderboard).

Other questions and comments:
The ablation shows 0.7 improvement on EM with mixed objective. It is interesting that the mixed objective (which targets F1) also brings improvement on EM. 

review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors of this paper propose some extensions to the Dynamic Coattention Networks models presented last year at ICLR. First they modify the architecture of the answer selection model by adding an extra coattention layer to improve the capture of dependencies between question and answer descriptions. The other main modification is to train their DCN+ model using both cross entropy loss and F1 score (using RL supervision) in order to  reward the system for making partial matching predictions. Empirical evaluations conducted on the SQuAD dataset indicates that this architecture achieves an improvement of at least 3%, both on F1 and exact match accuracy, over other comparable systems. An ablation study clearly shows the contribution of the deep coattention mechanism and mixed objective training on the model performance. 

The paper is well written, ideas are presented clearly and the experiments section provide interesting insights such as the impact of RL on system training or the capability of the model to handle long questions and/or answers. It seems to me that this paper is a significant contribution to the field of question answering systems. 

review_title: Significant improvement of DCN answer selection models using mixed objectives and 2 stacked levels of coattention

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed an improved version of dynamic coattention networks, which is used for question answering tasks. Specifically, there are 2 aspects to improve DCN: one is to use a mixed objective that combines cross entropy with self-critical policy learning, the other one is to imporve DCN with deep residual coattention encoder. The proposed model achieved STOA performance on Stanford Question Asnwering Dataset and several ablation experiments show the effectiveness of these two improvements. Although DCN+ is an improvement of DCN, I think the improvement is not incremental. 

One question is that since the model is compicated, will the authors release the source code to repeat all the experimental results?
review_title: An improvement of DCN model

Meta-review: This is an interesting paper that provides modeling improvements over several strong baselines and presents SOTA on Squad.  One criticism of the paper is that it evaluates only on Squad, which is somewhat of an artificial task, but we think for publication purposes at ICLR, the paper has a reasonable set of components.
Decision: Accept (Poster)