Title: Bidirectional Attention Flow for Machine Comprehension
Authors: Minjoon Seo
Aniruddha Kembhavi
Ali Farhadi
Hannaneh Hajishirzi
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test.


Review discussion:
Change #1. A non-affiliated researcher pointed out that, after carefully examining our publicly available code, entities in CNN/DailyMail datasets should have been shuffled every time an example (article-query pair) is loaded. We re-ran the experiment with the correction (with some changes in hyperparameters), and we now report the corrected single-run results. We will report our ensemble method results as soon as they are available, which take several days/weeks to train.
Change #2. We retouched and clarified the wording of the paper, mainly Sections 1 and 3.
Change #1: New results (state of the art) on SQuAD, after fixing some bugs in the preprocessing step of the data and the postprocessing step of the answers, and using moving averages instead of raw weights during test.
Also included all recent results in the SQuAD leaderboard as of 12pm 28 Nov 2016.
Change #2: New results on CNN/DailyMail datasets, after using moving averages instead of raw weights during test during test.
"machine comprehension" is both an overhyped and underspecified title.
why not just use existing terms like question answering?
The final ensemble numers in the paper and the SQUAD leaderboard are different. What makes the difference?
Thank you for your questions!
We used the term “machine comprehension” to mean “answering query given context in natural language”, referring to both SQuAD and CNN/DailyMail cloze test. Please note that the terms “machine comprehension” or “reading comprehension” were also used in these dataset papers as well as in their follow-ups. Since QA is a generic term and does not necessarily require “context in natural language” (e.g. QA on databases such as Freebase), we thought MC is a better term for the two tasks.
We just verified that the most recent version of the paper and the leaderboard both report EM=73.3, F1=81.1. Please note that there is usually a delay of 2-3 days between obtaining the test result (by email) and the result being updated on the leaderboard. The difference between the previous model and the current model is highlighted in our Nov 28 revision comment below.
In short, we fixed few bugs and loaded moving average instead of raw weights.
I am interested in knowing how the authors decided the scalar function (equation 1) that encodes the similarity between the phrase-level embeddings of the context (H) and the query (U). In the paper, they mention that it is a linear function of [h; u; h.u]. Did the authors try any other function e.g., a linear function of [h.u]?
Similarly, how was the vector function (equation 2) determined? What happens if this function is an MLP?
Thank you for your questions!
We are aware of a few different approaches for encoding similarity between vectors, such as dot product (Hill et al., 2016) and bilinear term (Chen et al., 2016). During our experiments, we found that concatenating elementwise multiplications (e.g. h.u) before vanilla linear transformation gives better performance than the previous approaches.
As you suggested, we included the comparisons between different definitions of Equations 1 and 2 in the newest (current) revision. Please see Table 5 in Appendix B;
in short, our choice of similarity and vector functions performs similarly or better than other choices.Thank you for the suggestion!
You write "These attentions are derived from the similarity matrix, S".
so the attentions are equal to S, or what is the attention exactly equal to.
Thanks.
Thank you for your question!
The Similarity matrix S is computed once, and is used to compute both directions of attention. Note that S is not the attention itself, because it is an unnormalized 2D similarity measure between the context and the query words.
However, by pooling and normalizing the matrix on one dimension or the other, we obtain the attention weights in either direction. We made our description of attention more explicit in the newest (current) revision.
In response to the questions of the reviewers:
1. Added Appendix B for comparing performance with different choices of similarity and fusion functions.
2. Clarified the description of the similarity matrix and how the attention is obtained from the similarity matrix.
The paper presents an architecture for answering questions about text. The paper proposes a novel architecture which jointly attends over the context and the query.
1.	The paper is clearly written and illustrated.
2.	The architecture is new and incorporates novel and interesting aspects:
2.1.	The attention is not summarized immediately but the features are only weighted with the attention to not loose information.
2.2.	The approach estimates two directions of attention, by maximizing in two directions of the similarity matrix S – towards the context and towards the query.
3.	The paper extensively evaluates the approach on three datasets SQuAD, CNN and Daily Mail. In all cases showing state-of-the-art performance.
It is worth noting that the SQuAD and the CNN/Daily Mail are slightly different tasks and it is positive that the model works well in both scenarios.
3.1.
It is worth noting that the paper even compares mainly favorably to concurrent work (including other ICLR 2017 submissions), recently published/listed on the evaluation server for SQuAD
4.	The paper also includes an ablation study and qualitative results.
5.	I think the paper provides a good discussion of related work and I like that it points out the relations to Visual question answering (VQA). It would be interesting to see how the architecture can be adapted and works on the VQA task.
6.	The authors revised the paper based on the comments from reviewers and others.
7.	It would be interesting to see more qualitative results, e.g. in an appendix.
7.1.	Fig. 3 seems to miss the predicted answer.
7.2.	It would also be interesting to compare the results of different approaches, maybe in a more compact format.
Given the new architecture with novel aspects and the strong experimental evaluation I recommend to accept the paper.
We thank reviewer R2 for insightful comments and suggestions.
Regarding qualitative results, please note that the paper already includes qualitative results for correct (Figure 3) and incorrect (Appendix A) examples. We will add more analysis on success cases in the appendix of a future revision.
Regarding missing predicted answer in Figure 3, we used a red font to denote the the answer words, but perhaps this is not clear in grayscale prints. We have now made this more clear with underlines in the current revision.
After reading the the comments of other reviewers and the answers of the authors I stay with my recommendation to accept the paper.
Paper Summary:
The paper presents a novel approach for machine comprehension. The proposed model, Bi-Directional Attention Flow (BIDAF) network is a multi-stage hierarchical approach that represents the context and query at different levels of granularity and uses bi-directional attention mechanisms to predict an answer. The proposed approach achieves state-of-the-art results on SQuAD dataset and CNN/DailyMail cloze test.
Paper Strengths:
-- The proposed model uses attention in both directions instead of uni-directional attention mechanisms used in previous approaches, prevents early summarization by passing all information to RNNs and uses memory-less attention mechanism, which is a novel combination for machine comprehension task.
-- The proposed model is modular, and can be easily changed for other related tasks, as shown through two types of experiments in the paper.
-- The paper presents thorough ablation study of the proposed model, clearly presenting the importance of all major components in the model. The authors also added further detailed studies as requested by reviewer.
-- Relation of their model to Visual Question Answering approaches, and computing features from context and query at different levels of granularity to multiple layers in Convolutional Neural Networks, are interesting and help in better understanding of the model. It would be very interesting to see how this approach would work for the task of Visual Question Answering.
-- Further visualizations and error analysis can help in identifying failure modes of the model and help in designing next generation of models.
-- The proposed model achieves state-of-the-art result on SQuAD dataset, and CNN/DailyMail cloze test.
-- The paper is well written and the architecture is described in sufficient detail.
Paper Weaknesses / Future Thoughts:
-- As with many deep learning approaches, the overall architecture seems quite complex, and the design choices seem to be driven by performance numbers. As future work, authors might try to analyze qualitative advantages of different modules in the proposed model. What type of questions are correctly answered after adding Context-to-Query attention? What about Query-to-Context attention?
-- Some qualitative examples of success and failure cases should be added to the paper.
Preliminary Evaluation:
Novel, state-of-the-art, and well-studied machine comprehension approach. Paper is well written. In my thoughts, a clear accept.
We thank reviewer R3 for insightful comments and suggestions.
Regarding qualitative examples of success and failures, please note that we have already listed some success cases in Figure 3 and failures cases in Appendix A. We will add a further analysis on success cases in the appendix of a future revision.
This is a solid paper with good results.
However, there aren't many very interesting takeaways (most of the architecture seems a concatenation of standard elements to do well in the leaderboard) and some issues in the writing.
The second paragraph of the introduction is very confusing. It's clear the authors got really deep into their world and made no attempts to actually clearly explain their model, not even to an expert in the field, let a lone somebody who isn't familiar with similar approaches.
The authors keep referring to "previously popular attention paradigms" without any citation and then, I believe, incorrectly describe whatever those are supposed to be by writing that these unknown but popular approaches "summarize each modality into a single vector." That's one of the most incorrect descriptions I've yet seen for attention mechanisms.
First, I don't know what model works over several modalities in a single attention pass. Maybe the authors don't know what a "modality" is? More importantly, the whole point of most attention mechanisms is that one does not simply summarize the whole input but instead can access all elements of it.
So, this paper's supposedly new way of using attention is pretty much exactly the standard way.
Both modeling and modelling spellings are in the text.
I understand the need to sometimes invent new terminology to describe a model but in this paragraph, the authors 3 times talk about a "modeling layer (RNN)"... It's just an RNN, you don't need to give an RNN another name, especially one that's as nondescript as "modeling layer" all layers are part of a model?
Typo: "let's the modeling (RNN) layer to learn"
This paragraph is supposed to give an overview of the model but just confuses readers.
I would delete it.
"Phrase embedding layer" -- terrible word choice as you are not embedding phrases here. It's a standard bidirectional LSTM over words, not phrases. In all subsequent parts of the paper you just give examples of words embedded in context. No phrases. Please change this to "contextual word embedding layer" or something less incorrect.
Your phrase layer embeddings only show single words, as expected in Table 2.
Section 2: point 4.
Second sentence needs citations for "popular"
Typo: "from both *of* the context and query word"
Typo: "aveaged"
It seems like your output layer changes quite substantially so your claim in the abstract/intro of using the same model isn't quite accurate. I'd say you're changing one module or part of your model.
Section 4: attention isn't countable (no "a" in front of "huge attention").
Also, academic writing usually doesn't include such adjectives in the first place.
We thank reviewer R1 for insightful comments, suggestions as well as pointing out some spelling and grammar issues. We have incorporated this feedback in our latest revision and we discuss these changes below.
Novelty of BiDAF:
Most modern neural network models consist of standard elements, and how they are composed is often a major contribution. As pointed out by reviewers R2 and R3, the major novelties of our model are: the memory-less attention flow mechanism and the bi-directionality of the attention, as well as how they can be combined with other standard elements (Char-CNN, LSTM, etc.) to perform well in machine comprehension tasks.
Second paragraph of the introduction is unclear:
We have reworded the first and second paragraphs to make them more clear to readers.
“Most previous work summarizes each modality into a single vector” is incorrect:
We meant that in most previous MC and QA works, the computed attention weights are often used to summarize the context (paragraph or image) into a single fixed-size vector, thus extracting the most relevant information from the context for answering the question. We realize that our previous wording could have been a bit confusing, and we have updated the paper accordingly.
“Modeling layer”:
We named it “modeling layer” to merely indicate its functionality. This is similar to the nomenclature used for others layers in the network (similar to R1’s suggestion on using the name “contextual embedding layer”). We use RNNs in several different layers in the system, but they play different roles in different layers, and the names assigned to the different layers in the system indicate these roles.
“Phrase embedding layer”:
Following the suggestion, we have changed this to “contextual embedding layer”.
Section 2 point 4 needs citation for “popular”:
Please note that Section 3 contains a detailed list of related work. In the current revision, we have now added citations to Memory Networks (Weston et al., 2015) and three recent papers in machine comprehension tasks.
Output layer has changed to adapt to CNN/DailyMail:
In the current revision, we now mention in the introduction that we changed the output module for the CNN/DailyMail dataset.
Typos and grammatical issues:
We have fixed several typos as well as grammatical issues throughout the paper.
Incorporated reviewers' and Kelly Zhang's suggestions:
- Fixed typos / grammatical errors
- "phase embedding layer" -> "contextual embedding layer"
- Added more citation to Section 2
- Reworded abstract and first two paragraphs of intro
- Fixed Stanford AR reader's numbers
- Added WikiQA Experiment (Appendix C for now), giving <2% higher performance than the previous SOTA without seeing any example on WikiQA, i.e. zero-shot scenario.
- Voided the above revision, because zero-shot is a widely different experiment from the other QA experiments.
Final version uploaded:
- fixed typos
- more spacious formatting
- added acknowledgments

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. All reviewers agree that this is a good piece of work that should be accepted to ICLR. Authors are encouraged to incorporate reviewer feedback to further strengthen the paper.

Decision: Accept (Poster)