Title: FusionNet: Fusing via Fully-aware Attention with Application to Machine Comprehension
Authors: Hsin-Yuan Huang
Chenguang Zhu
Yelong Shen
Weizhu Chen
Venue: ICLR 2018
Keywords: Attention Mechanism
Machine Comprehension
Natural Language Processing
Deep Learning

Abstract: This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "History of Word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it identifies an attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

TL;DR: We propose a light-weight enhancement for attention and a neural architecture, FusionNet, to achieve SotA on SQuAD and adversarial SQuAD.

Review discussion:
The paper first analyzes recent works in machine reading comprehension (largely centered around SQuAD), and mentions their common trait that the attention is not "fully-aware" of all levels of abstraction, e.g. word-level, phrase-level, etc. In turn, the paper proposes a model that performs attention at all levels of abstraction, which achieves the state of the art in SQuAD. They also propose an attention mechanism that works better than others (Symmetric + ReLU).
Strengths:
- The paper is well-written and clear.
- I really liked Table 1 and Figure 2; it nicely summarizes recent work in the field.
- The multi-level attention is novel and indeed seems to work, with convincing ablations.
- Nice engineering achievement, reaching the top of the leaderboard (in early October).
Weaknesses:
- The paper is long (10 pages) but relatively lacks substances. Ideally, I would want to see the visualization of the attention at each level (i.e. how they differ across the levels) and also possibly this model tested on another dataset (e.g. TriviaQA).
- The authors claim that the symmetric + ReLU is novel, but  I think this is basically equivalent to bilinear attention [1] after fully connected layer with activation, which seems quite standard.
Still useful to know that this works better, so would recommend to tone down a bit regarding the paper's contribution.
Minor:
- Probably figure 4 can be drawn better. Not easy to understand nor concrete.
- Section 3.2 GRU citation should be Cho et al. [2].
Questions:
- Contextualized embedding seems to give a lot of improvement in other works too. Could you perform ablation without contextualized embedding (CoVe)?
Reference:
[1] Luong et al. Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015.
[2] Cho et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP 2014.
Thank you so much for your thoughtful review!
We acknowledged that our paper is quite long since we hope to best clarify the high-level concepts and the low-level details for fully-aware attention and our model, FusionNet. We are willing to add the visualization of multi-level attention weights in the appendix and will do so during the revision period. To support the generalization of our model, we have also tested on two adversarial datasets in addition to SQuAD and showed significant improvement. We are also working on extending to other datasets such as TriviaQA.
While most machine comprehension models operate at paragraph-level (including ours), TriviaQA requires processing document-level input.
For example, the current state-of-the-art method on TriviaQA [1] uses a pipelined approach where machine comprehension model is only a part of the pipeline.
Hence, we think a more in-depth study is needed to give a solid comparison on TriviaQA.
We agree that the symmetric formulation is only a slightly modified version of standard multiplicative attention.
However, during our research study on various architectures, incorporating history-of-word in attention score calculation only yields marginal improvements when existing attention formulations are used.
On the other hand, when the slightly modified symmetric form is used, fully-aware attention becomes substantially better than normal attention. In the paper, we emphasized the importance of the symmetric attention form in the hope of the future researchers to utilize fully-aware attention better. We have rewritten our manuscript to stress the identification of the proper attention function and give less focus on the proposition of novel formulation. Thank you for pointing this out.
Thank you for the question! CoVe is also helpful in our model. We have conducted some additional ablation study regarding input vectors based on your question.
Performance on Dev set is shown (EM / F1):
FusionNet : 75.3 / 83.6
FusionNet without CoVe (original setting) : 74.1 / 82.5
FusionNet without CoVe (embedding dropout=0.3) : 73.7 / 82.4
FusionNet with GloVe fixed: 75.0 / 83.2
Best documented number [3]: 72.1 / 81.6
Note that we have optimized the hyperparameters in FusionNet with CoVe included. We have also tried our best to simplify the model when CoVe is available since we believe a simpler model is a better model.
Therefore it is not very suitable to directly compare the ablation result with other models without CoVe.
For example, we did not include character embedding (giving 2% improvement in BiDAF [4]) or multi-hop reasoning (giving 1% improvement in Reinforced Mnemonic Reader [3]) in our model.
Minor:
- We will try our best to improve figure 4. It will be very kind of you if you could give us some suggestions.
- Thank you for pointing out the typo in GRU citation.
Reference:
[1] Clark, Christopher, and Matt Gardner. "Simple and Effective Multi-Paragraph Reading Comprehension." arXiv preprint arXiv:1710.10723 (2017)
[2] Jia, Robin, and Percy Liang. "Adversarial examples for evaluating reading comprehension systems." EMNLP (2017).
[3] Minghao Hu, Yuxing Peng, and Xipeng Qiu. "Reinforced Mnemonic Reader for Machine Comprehension." arXiv preprint arXiv:1705.02798 (2017).
[4] Minjoon Seo, et al. "Bidirectional attention flow for machine comprehension." ICLR (2017).
The primary intellectual point the authors make is that previous networks for machine comprehension are not fully attentive.
That is, they do not provide attention on all possible layers on abstraction such as the word-level and the phrase-level. The network proposed here, FusionHet, fixes problem.
Importantly, the model achieves state-of-the-art performance of the SQuAD dataset.
The paper is very well-written and easy to follow. I found the architecture very intuitively laid out, even though this is not my area of expertise.
Moreover, I found the figures very helpful -- the authors clearly took a lot of time into clearly depicting their work! What most impressed me, however, was the literature review. Perhaps this is facilitated by the SQuAD leaderboard, which makes it simple to list related work.
Nevertheless, I am not used to seeing comparison to as many recent systems as are presented in Table 2.
All in all, it is difficult not to highly recommend an architecture that achieves state-of-the-art results on such a popular dataset.
Thank you so much for your encouraging review!
(Score before author revision: 4)
(Score after author revision: 7)
I think the authors have taken both the feedback of reviewers as well as anonymous commenters thoroughly into account, running several ablations as well as reporting nice results on an entirely new dataset (MultiNLI) where they show how their multi level fusion mechanism improves a baseline significantly. I think this is nice since it shows how their mechanism helps on two different tasks (question answering and natural language inference).
Therefore I would now support accepting this paper.
------------(Original review below) -----------------------
The authors present an enhancement to the attention mechanism called "multi-level fusion" that they then incorporate into a reading comprehension system. It basically takes into account a richer context of the word at different levels in the neural net to compute various attention scores.
i.e. the authors form a vector "HoW" (called history of the word), that is defined as a concatenation of several vectors:
HoW_i = [g_i, c_i, h_i^l, h_i^h]
where g_i = glove embeddings, c_i = COVE embeddings (McCann et al. 2017), and h_i^l and h_i^h are different LSTM states for that word.
The attention score is then a function of these concatenated vectors i.e. \alpha_{ij} = \exp(S(HoW_i^C, HoW_j^Q))
Results on SQuAD show a small gain in accuracy (75.7->76.0 Exact Match). The gains on the adversarial set are larger but that is because some of the higher performing, more recent baselines don't seem to have adversarial numbers.
The authors also compare various attention functions (Table 5) showing a particularone (Symmetric + ReLU) works the best.
Comments:
-I feel overall the contribution is not very novel.  The general neural architecture that the authors propose in Section 3 is generally quite similar to the large number of neural architectures developed for this dataset (e.g. some combination of attention between question/context and LSTMs over question/context). The only novelty is these "HoW" inputs to the extra attention mechanism that takes a richer word representation into account.
-I feel the model is seems overly complicated for the small gain (i.e. 75.7->76.0 Exact Match), especially on a relatively exhausted dataset (SQuAD) that is known to have lots of pecularities (see anonymous comment below). It is possible the gains just come from having more parameters.
-The authors (on page 6) claim that that by running attention multiple times with different parameters but different inputs (i.e. \alpha_{ij}^l, \alpha_{ij}^h, \alpha_{ij}^u) it will learn to attend to "different regions for different level".
However, there is nothing enforcing this and the gains just probably come from having more parameters/complexity.
Thank you for your review!
1. Our improvement over best model published is actually ~3% in Exact Match (73. 2 -> 76. 0). We understand that from our Table 2, it seems FusionNet only improve the best "published" model (R-net) by EM 0. 3 (single model). We apologize for not writing this part clear and have updated our paper accordingly. If you look into the ACL2017 paper of R-net [1] or the recent technical report (http://aka. ms/rnet), you will find that the best-published version of R-net only achieves EM: 72. 3, F1: 80. 7. It is lower than our model by near 4% in EM.
It is because the authors of R-net have been designing new models without publishing it while using the same model name (R-net) on SQuAD leaderboard. At the time of ICLR2018 submission, the best-published model is Reinforced Mnemonic Reader [2] (https://arxiv. org/pdf/1705. 02798. pdf), which achieved EM: 73. 2, F1: 81. 8, 1% higher than published version of R-net. Reinforced Mnemonic Reader proposed feature-rich encoder, semantic fusion unit, iterative interactive-aligning self-aligning, multihop memory-based answer pointer, and a reinforcement learning technique to achieve their high performance.
On the other hand, utilizing our simple "HoW" attention mechanism, FusionNet obtained a decent performance (EM: 76.0, F1: 83.9) on original SQuAD with a relatively simplistic model.
For example, in Table 6, by changing S(h_i^C, h_j^Q) to S(HoW_i^C, HoW_j^Q) in a vanilla model (encoder + single-level attention), we observed +8% improvement and achieved EM: 73.3, F1: 81.4 on the dev set. (best documented number on dev set: 72.1 / 81.6)
2. In our paper, we only compare with the official results on adversarial dataset shown in this year EMNLP paper [3]. Adversarial evaluation of more recent higher performing models can be found in a website maintained by the author (Robin Jia). And we still significantly outperform these recent state-of-the-art methods.
https://worksheets.codalab.org/worksheets/0x77ca15a1fc684303b6a8292ed2167fa9/
For example, Robin Jia has compared with another state-of-the-art model DCN+, which is also submitted to ICLR2018. On the AddSent dataset, DCN+ achieved F1: 44.5, and on the AddOneSent dataset, DCN+ achieved F1: 54.3. The results are comparable to the previous state-of-the-art on the adversarial datasets.
But FusionNet is +6% higher than DCN+ on both datasets. We attribute this significant gain to the proposed HoW attention, which can very easily incorporate into other models. We are excited to share this simple idea with the community to improve machines in better understanding texts.
3. SQuAD is a very competitive dataset, so it is unlikely that the significant gain we have (+3% over best-documented models, +5% in adversarial datasets) comes from giving the model more parameters.
Furthermore, existing models can always incorporate more parameters if it helps.
For example, in the high-performing Reinforced Mnemonic Reader, they can increase the number of iterations in iterative aligning or increase the hidden size in LSTM.
Additionally, in our Table 6, we have compared FA All-level and FA Multi-level. FA All-level uses the same attention weight for different levels and fuses all level of representation (including input vector). FA All-level has more parameters than FA Multi-level but performs 2% worse. Based on Reviewer3's comment, we will also include visualization to show that multi-level attention will learn to attend to "different regions for different levels."
References:
[1] Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. "Gated self-matching networks for reading comprehension and question answering." ACL (2017).
[2] Minghao Hu, Yuxing Peng, and Xipeng Qiu. "Reinforced Mnemonic Reader for Machine Comprehension." arXiv preprint arXiv:1705.02798 (2017).
[3] Robin Jia, and Percy Liang. "Adversarial examples for evaluating reading comprehension systems." EMNLP (2017).
We have conducted additional experiments based on Reviewer 3's thoughtful comments and included the results in our paper.
(1) Additional ablation study on input vectors. (Appendix C)
The following is a summary of our additional experimental results. The experiment shows that FusionNet, with or without CoVe, single or ensemble, all yield clear improvement over previous state-of-the-art models on all 3 machine comprehension datasets.
=== SQuAD (Dev EM) ===
> FusionNet: 75.3 (+3.2)
> FusionNet (without CoVe): 74.1 (+2.0)
Previous SotA [1]: 72.1
=== AddSent (Dev EM) ===
> FusionNet (Single): 45.6 (+4.9)
> FusionNet (Single, without CoVe): 47.4 (+6.7)
> FusionNet (Ensemble): 46.2 (+5.5)
Previous SotA [2] (Ensemble): 40.7
=== AddOneSent (Dev EM) ===
> FusionNet (Single): 54.8 (+6.1)
> FusionNet (Single, without CoVe): 55.2 (+6.5)
> FusionNet (Ensemble): 54.7 (+6.0)
Previous SotA [2] (Ensemble): 48.7
(2) Application to Natural Language Inference. (Appendix D)
FusionNet is an improved attention mechanism that can be easily applied to attention-based models. Here, we consider a different task on natural language inference (NLI). We focus on MultiNLI [3], a recent NLI corpus. MultiNLI is designed to be more challenging than SNLI [4] since many models already outperform human performance (87.7%) on SNLI. A state-of-the-art model for NLI is ESIM [5], which performs 88.0% on SNLI and 72.3% (in-domain), 72.1% (cross-domain) on MultiNLI [3]. We implemented a version of ESIM in PyTorch and improved ESIM with our proposed fully-aware multi-level attention mechanisms. For fair comparisons, we reduce the hidden size after adding our enhancements, so the parameter size after attention enhancement is less than or similar to ESIM with standard attention. A summary of the result is shown below. Experiments on natural language inference conform with our observed improvements in machine comprehension tasks.
=== MultiNLI (Dev Cross-domain / In-domain) ===
Our ESIM (d = 300): 73.9 / 73.7
Our ESIM + fully-aware (d = 250): 77.3 / 76.5
Our ESIM + fully-aware + multi-level (d = 250): 78.4 / 78.2
(3) Multi-level attention visualization. (Appendix G)
In Appendix G, we included multi-level attention visualization and a qualitative analysis of the attention variations between low-level and high-level. These visualizations support our original motivation and provide an intuitive explanation for our superior performance, especially on the adversarial datasets.
References:
[1]: Minghao Hu, Yuxing Peng, and Xipeng Qiu. "Reinforced Mnemonic Reader for Machine Comprehension." arXiv preprint arXiv:1705.02798 (2017).
[2]: Robin Jia, and Percy Liang. "Adversarial examples for evaluating reading comprehension systems." EMNLP (2017).
[3]: Adina Williams, Nikita Nangia, and Samuel R. Bowman. "A broad-coverage challenge corpus for sentence understanding through inference." arXiv preprint arXiv:1704.05426 (2017).
[4]: Samuel R. Bowman et al. "A large annotated corpus for learning natural language inference." EMNLP (2015).
[5]: Qian Chen et al. "Enhancing and combining sequential and tree lstm for natural language inference." ACL (2017).

Meta-review:
role: PC
data: comment: State-of-the-art results on Squad (at least at time of submission) with a nice model. Authors have since applied the model to additional tasks (SNLI). Good discussion with reviewers, well written submission and all reviewers suggest acceptance. 

Decision: Accept (Poster)