Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: (Score before author revision: 4)
(Score after author revision: 7)

I think the authors have taken both the feedback of reviewers as well as anonymous commenters thoroughly into account, running several ablations as well as reporting nice results on an entirely new dataset (MultiNLI) where they show how their multi level fusion mechanism improves a baseline significantly. I think this is nice since it shows how their mechanism helps on two different tasks (question answering and natural language inference).

Therefore I would now support accepting this paper.

------------(Original review below) -----------------------

The authors present an enhancement to the attention mechanism called "multi-level fusion" that they then incorporate into a reading comprehension system. It basically takes into account a richer context of the word at different levels in the neural net to compute various attention scores.

i.e. the authors form a vector "HoW" (called history of the word), that is defined as a concatenation of several vectors:

HoW_i = [g_i, c_i, h_i^l, h_i^h]

where g_i = glove embeddings, c_i = COVE embeddings (McCann et al. 2017), and h_i^l and h_i^h are different LSTM states for that word.

The attention score is then a function of these concatenated vectors i.e. \alpha_{ij} = \exp(S(HoW_i^C, HoW_j^Q))

Results on SQuAD show a small gain in accuracy (75.7->76.0 Exact Match). The gains on the adversarial set are larger but that is because some of the higher performing, more recent baselines don't seem to have adversarial numbers.

The authors also compare various attention functions (Table 5) showing a particularone (Symmetric + ReLU) works the best. 

Comments:

-I feel overall the contribution is not very novel.  The general neural architecture that the authors propose in Section 3 is generally quite similar to the large number of neural architectures developed for this dataset (e.g. some combination of attention between question/context and LSTMs over question/context). The only novelty is these "HoW" inputs to the extra attention mechanism that takes a richer word representation into account.

-I feel the model is seems overly complicated for the small gain (i.e. 75.7->76.0 Exact Match), especially on a relatively exhausted dataset (SQuAD) that is known to have lots of pecularities (see anonymous comment below). It is possible the gains just come from having more parameters.

-The authors (on page 6) claim that that by running attention multiple times with different parameters but different inputs (i.e. \alpha_{ij}^l, \alpha_{ij}^h, \alpha_{ij}^u) it will learn to attend to "different regions for different level". However, there is nothing enforcing this and the gains just probably come from having more parameters/complexity.
review_title: Revisions have improved the paper

final_score: rating: 7: Good paper, accept
review_content: The paper first analyzes recent works in machine reading comprehension (largely centered around SQuAD), and mentions their common trait that the attention is not "fully-aware" of all levels of abstraction, e.g. word-level, phrase-level, etc. In turn, the paper proposes a model that performs attention at all levels of abstraction, which achieves the state of the art in SQuAD. They also propose an attention mechanism that works better than others (Symmetric + ReLU).

Strengths:
- The paper is well-written and clear.
- I really liked Table 1 and Figure 2; it nicely summarizes recent work in the field.
- The multi-level attention is novel and indeed seems to work, with convincing ablations.
- Nice engineering achievement, reaching the top of the leaderboard (in early October).


Weaknesses:
- The paper is long (10 pages) but relatively lacks substances. Ideally, I would want to see the visualization of the attention at each level (i.e. how they differ across the levels) and also possibly this model tested on another dataset (e.g. TriviaQA).
- The authors claim that the symmetric + ReLU is novel, but  I think this is basically equivalent to bilinear attention [1] after fully connected layer with activation, which seems quite standard. Still useful to know that this works better, so would recommend to tone down a bit regarding the paper's contribution.


Minor:
- Probably figure 4 can be drawn better. Not easy to understand nor concrete.
- Section 3.2 GRU citation should be Cho et al. [2].


Questions:
- Contextualized embedding seems to give a lot of improvement in other works too. Could you perform ablation without contextualized embedding (CoVe)?


Reference:
[1] Luong et al. Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015.
[2] Cho et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP 2014.
review_title: Nice analysis of literature, interesting model and good results, but a little lack of substance

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The primary intellectual point the authors make is that previous networks for machine comprehension are not fully attentive. That is, they do not provide attention on all possible layers on abstraction such as the word-level and the phrase-level. The network proposed here, FusionHet, fixes problem. Importantly, the model achieves state-of-the-art performance of the SQuAD dataset.

The paper is very well-written and easy to follow. I found the architecture very intuitively laid out, even though this is not my area of expertise. Moreover, I found the figures very helpful -- the authors clearly took a lot of time into clearly depicting their work! What most impressed me, however, was the literature review. Perhaps this is facilitated by the SQuAD leaderboard, which makes it simple to list related work. Nevertheless, I am not used to seeing comparison to as many recent systems as are presented in Table 2. 

All in all, it is difficult not to highly recommend an architecture that achieves state-of-the-art results on such a popular dataset.
review_title: state of the art on SQuAD with FusionNet

Meta-review: State-of-the-art results on Squad (at least at time of submission) with a nice model. Authors have since applied the model to additional tasks (SNLI). Good discussion with reviewers, well written submission and all reviewers suggest acceptance. 
Decision: Accept (Poster)