Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new machine comprehension model, which integrates several contributions like different embeddings for gate function and passage representation function, self-attention layers and highway network based fusion layers. The proposed method was evaluated on the SQuAD dataset only, and marginal improvement was observed compared to the baselines.

(1) One concern I have for this paper is about the evaluation. The paper only evaluates the proposed method on the SQuAD data with systems submitted in July 2017, and the improvement is not very large. As a result, the results are not suggesting significance or generalizability of the proposed method.

(2) The paper gives some ablation tests like reducing the number of layers and removing the gate-specific question embedding, which help a lot for understanding how the proposed methods contribute to the improvement. However, the results show that the deeper self-attention layers are indeed useful (but still not improving a lot, about 0.7-0.8%). The other proposed components contribute less significant. As a result, I suggest the authors add more ablation tests regarding (1) replacing the outer-fusion with simple concatenation (it should work for two attention layers); (2) removing the inner-fusion layer and only use the final layer's output, and using residual connections (like many NLP papers did) instead of the more complicated GRU stuff.

(3) Regarding the ablation in Table 2, my first concern is that the improvement seems small (~0.5%). As a result, I am wondering whether this separated question embedding really brings new information, or the similar improvement can be achieved by increasing the size of LSTM layers. For example, if we use the single shared question embeddings, but increase the size from 128 to some larger number like 192, can we observe similar improvement. I suggest the authors try this experiment as well and I hope the answer is no, as separated input embeddings for gate functions was verified to be useful in some "old" works with syntactic features as gate values, like "Semantic frame identification with distributed word representations" and "Learning composition models for phrase embeddings" etc.

(4) Please specify which version of the SQuAD leaderboard is used in Table 3. Is it a snapshot of the Jul 14 one? Because this paper is not comparing to the state-of-the-art, no specification of the leaderboard version may confuse the other reviewers and readers. By the way, it will be better to compare to the snapshot of Oct 2017 as well, indicating the position of this work during the submission deadline.

Minor issues:

(1) There are typos in Figure 1 regarding the notations of Question Features and Passage Features.

(2) In Figure 1, I suggest adding an "N \times" symbol to the left of the Q-P Attention Layer and remove the current list of such layers, in order to be consistent to the other parts of the figure.

(3) What is the relation between the "PhaseCond, QPAtt+" in Table 2 and the "PhaseCond" in Table 3? I was assuming that those are the same system but did not see the numbers match each other.
review_title: New model architecture with marginal improvement, more ablation tests and analysis may help

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a fairly elaborate model for reading comprehension evaluated on the SQuAD dataset.   The model is shown to improve on the published results but not as-of-submission leaderboard numbers.

The main weakness of the paper in my opinion is that the innovations seem to be incremental and not based on any overarching insight or general principle.  A less significant issue is that the English is often disfluent.

Specific comments: I would remove the significance daggers from table 2 as the standard deviations are already reported and the null hypothesis for which significance is measured seems unclear.  I am also concerned to see test performance significantly better than development performance in table 3.  Other systems seem to have development and test performance closer together.  Have the authors been evaluating many times on the test data?

review_title: Competent work.  This paper seems to represent a legitimate player in the SQuAD leaderboard game.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper introduces "Phase Conductor", which consists of two phases, context-question attention phase and context-context (self) attention phase. Each phase has multiple layers of attention, for which the paper uses a novel way to fuse the layers, and context-question attention uses different question embedding for getting the attention weight and getting the attention vector. The paper shows that the model achieves state of the art on SQuAD among published papers, and also quantitatively and visually demonstrates that having multiple layers of attention is helpful for context-context attention, while it is not so helpful for context-question attention.


Note: While I will mostly try to ignore recently archived, non-published papers when evaluating this paper, I would like to mention that the paper's ensemble model currently stands 11th on SQuAD leaderboard.


Pros:
- The model achieves SOTA on SQuAD among published papers.
- The sequential fusing (GRU-like) of the multiple layers of attention is interesting and novel. Visual analysis of the attention map is convincing.
- The paper is overall well-written and clear.

Cons:
- Using different embedding for computing attention weights and getting attended vector is not entirely novel but rather an expected practice for many memory-based models, and should cite relevant papers. For instance, Memory Networks [1] uses different embedding for key (computing attention weight) and value (computing attended vector).
- While ablations for number of attention layers (1 or 2) were visually convincing, numerically there is a very small difference even for selfAtt. For instance, in Table 4, having two layers of selfAtt (with two layers of question-passage) only increases max F1 by 0.34, where the standard deviation is 0.31 for the one layer. While this may be statistically significant, it is a very small gain nonetheless.
- Given the above two cons, the main contribution of the paper is 1.1% improvement over previous state of the art. I think this is a valuable engineering contribution, but I feel that it is not well-suited / sufficient for ICLR audience. 


Questions:
- page 7 first para: why have you not tried GloVe 300D, if you think it is a critical factor?


Errors:
- page 2 last para: "gives an concrete" -> "gives a concrete"
- page 2 last para: "matching" -> "matched"
Figure 1: I think "passage embedding h" and "question embedding v" boxes should be switched.
- page 7 3.3 first para: "evidence fully" -> "evidence to be fully"


[1] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory Networks. ICLR 2015.
review_title: Improvements on SQuAD, but overall weak

Meta-review: Generally solid engineering work but a bit lacking in terms of novelty and some issues with clarity. At the end of the day the empirical gains are not sufficient for acceptance - the results are state-of-the-art relative to published work, but not in the top 10 based on the official leaderboard (not even at time of submission). Since the technical contributions are small and the engineering contributions have been made obsolete by concurrent work, I suggest rejection.
Decision: Reject