Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: "machine comprehension" is both an overhyped and underspecified title.
why not just use existing terms like question answering?

The final ensemble numers in the paper and the SQUAD leaderboard are different. What makes the difference?
review_title: questions

final_score: rating: 7: Good paper, accept
review_content: This is a solid paper with good results. However, there aren't many very interesting takeaways (most of the architecture seems a concatenation of standard elements to do well in the leaderboard) and some issues in the writing.

The second paragraph of the introduction is very confusing. It's clear the authors got really deep into their world and made no attempts to actually clearly explain their model, not even to an expert in the field, let a lone somebody who isn't familiar with similar approaches.
The authors keep referring to "previously popular attention paradigms" without any citation and then, I believe, incorrectly describe whatever those are supposed to be by writing that these unknown but popular approaches "summarize each modality into a single vector." That's one of the most incorrect descriptions I've yet seen for attention mechanisms. First, I don't know what model works over several modalities in a single attention pass. Maybe the authors don't know what a "modality" is? More importantly, the whole point of most attention mechanisms is that one does not simply summarize the whole input but instead can access all elements of it. So, this paper's supposedly new way of using attention is pretty much exactly the standard way.
Both modeling and modelling spellings are in the text.
I understand the need to sometimes invent new terminology to describe a model but in this paragraph, the authors 3 times talk about a "modeling layer (RNN)"... It's just an RNN, you don't need to give an RNN another name, especially one that's as nondescript as "modeling layer" all layers are part of a model? 
Typo: "let's the modeling (RNN) layer to learn"
This paragraph is supposed to give an overview of the model but just confuses readers.
I would delete it.

"Phrase embedding layer" -- terrible word choice as you are not embedding phrases here. It's a standard bidirectional LSTM over words, not phrases. In all subsequent parts of the paper you just give examples of words embedded in context. No phrases. Please change this to "contextual word embedding layer" or something less incorrect.
Your phrase layer embeddings only show single words, as expected in Table 2.
Section 2: point 4. Second sentence needs citations for "popular" 
Typo: "from both *of* the context and query word"
Typo: "aveaged"

It seems like your output layer changes quite substantially so your claim in the abstract/intro of using the same model isn't quite accurate. I'd say you're changing one module or part of your model.
Section 4: attention isn't countable (no "a" in front of "huge attention"). Also, academic writing usually doesn't include such adjectives in the first place. 
review_title: Conditional accept

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents an architecture for answering questions about text. The paper proposes a novel architecture which jointly attends over the context and the query.

1.	The paper is clearly written and illustrated.

2.	The architecture is new and incorporates novel and interesting aspects:
2.1.	The attention is not summarized immediately but the features are only weighted with the attention to not loose information.
2.2.	The approach estimates two directions of attention, by maximizing in two directions of the similarity matrix S – towards the context and towards the query.

3.	The paper extensively evaluates the approach on three datasets SQuAD, CNN and Daily Mail. In all cases showing state-of-the-art performance. It is worth noting that the SQuAD and the CNN/Daily Mail are slightly different tasks and it is positive that the model works well in both scenarios. 
3.1.	It is worth noting that the paper even compares mainly favorably to concurrent work (including other ICLR 2017 submissions), recently published/listed on the evaluation server for SQuAD

4.	The paper also includes an ablation study and qualitative results.

5.	I think the paper provides a good discussion of related work and I like that it points out the relations to Visual question answering (VQA). It would be interesting to see how the architecture can be adapted and works on the VQA task. 

6.	The authors revised the paper based on the comments from reviewers and others.

7.	It would be interesting to see more qualitative results, e.g. in an appendix. 
7.1.	Fig. 3 seems to miss the predicted answer. 
7.2.	It would also be interesting to compare the results of different approaches, maybe in a more compact format.

Given the new architecture with novel aspects and the strong experimental evaluation I recommend to accept the paper.


review_title: Review: New deep end-to-end attention architecture for machine comprehension with novel aspects and convincing results

final_score: rating: null
review_content: I am interested in knowing how the authors decided the scalar function (equation 1) that encodes the similarity between the phrase-level embeddings of the context (H) and the query (U). In the paper, they mention that it is a linear function of [h; u; h.u]. Did the authors try any other function e.g., a linear function of [h.u]?

Similarly, how was the vector function (equation 2) determined? What happens if this function is an MLP?
review_title: Modelling choices

final_score: rating: null
review_content: You write "These attentions are derived from the similarity matrix, S".

so the attentions are equal to S, or what is the attention exactly equal to.

Thanks.
review_title: Clarification

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
Paper Summary: 
The paper presents a novel approach for machine comprehension. The proposed model, Bi-Directional Attention Flow (BIDAF) network is a multi-stage hierarchical approach that represents the context and query at different levels of granularity and uses bi-directional attention mechanisms to predict an answer. The proposed approach achieves state-of-the-art results on SQuAD dataset and CNN/DailyMail cloze test. 

Paper Strengths: 
-- The proposed model uses attention in both directions instead of uni-directional attention mechanisms used in previous approaches, prevents early summarization by passing all information to RNNs and uses memory-less attention mechanism, which is a novel combination for machine comprehension task.
-- The proposed model is modular, and can be easily changed for other related tasks, as shown through two types of experiments in the paper.
-- The paper presents thorough ablation study of the proposed model, clearly presenting the importance of all major components in the model. The authors also added further detailed studies as requested by reviewer. 
-- Relation of their model to Visual Question Answering approaches, and computing features from context and query at different levels of granularity to multiple layers in Convolutional Neural Networks, are interesting and help in better understanding of the model. It would be very interesting to see how this approach would work for the task of Visual Question Answering.
-- Further visualizations and error analysis can help in identifying failure modes of the model and help in designing next generation of models.
-- The proposed model achieves state-of-the-art result on SQuAD dataset, and CNN/DailyMail cloze test.
-- The paper is well written and the architecture is described in sufficient detail.

Paper Weaknesses / Future Thoughts: 
-- As with many deep learning approaches, the overall architecture seems quite complex, and the design choices seem to be driven by performance numbers. As future work, authors might try to analyze qualitative advantages of different modules in the proposed model. What type of questions are correctly answered after adding Context-to-Query attention? What about Query-to-Context attention? 
-- Some qualitative examples of success and failure cases should be added to the paper.

Preliminary Evaluation: 
Novel, state-of-the-art, and well-studied machine comprehension approach. Paper is well written. In my thoughts, a clear accept.
review_title: Review

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. All reviewers agree that this is a good piece of work that should be accepted to ICLR. Authors are encouraged to incorporate reviewer feedback to further strengthen the paper.
Decision: Accept (Poster)