Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The authors propose a multi-hop "gated attention" model, which models the interactions between query and document representations, for answering cloze-style questions. The document representation is attended to sequentially over multiple-hops using similarity with the query representation (using a dot-product) as the scoring/attention function. 
The proposed method improves upon (CNN, Daily Mail, Who-Did-What datasets) or is comparable to (CBT dataset) the state-of-the-art results.


Pros:

1. Nice idea on heirarchical attention for modulating the context (document) representation by the task-specific (query) representation.
2. The presentation is clear with thorough experimental comparison with the latest results.


Comments:

1. The overall system presents a number of architectural elements: (1) attention at multiple layers (multi-hop), (2) query based attention for the context (or gated attention), (3) encoding the query vector at each layer independently.
It is important to breakdown the gain in performance due to the above factors: the ablation study presented in section 4.4 helps establish the importance of Gated Attention (#2 above). However, it is not clear:

  (1) how much multiple-hops of gated-attention contribute to the performance.
  (2) how important is it to have a specialized query encoder for each layer.

Understanding the above better, will help simplify the architecture.


2. The tokens are represented using L(w) and C(w). It is not clear if C(w) is crucial for the performance of the proposed method.
There is a significant performance drop when C(w) is absent (e.g. in "GA Reader--"; although there are other changes in "GA Reader--" which could affect the performance). Hence, it is not clear how much does the main idea, i.e., gated attention contributes towards the superior performance of the proposed method.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: SUMMARY.

The paper proposes a machine reading approach for cloze-style question answering.
The proposed system first encodes the query and the document using a bidirectional gru. These two representations are combined together using a Gated Attention (GA).
GA calculates the compatibility of each word in the document and the query as a probability distribution.
For each word in the document a gate is calculated weighting the query representation according to the word compatibility.
Ultimately, the gate is applied to the gru-encoded document word.
The resulting word vectors are re-encoded with a bidirectional GRU.
This process is performed for multiple hops. After k hops, the probability of a word to be part of the answer is calculated by a log-linear model that take as input the last word representations, and the concatenation of the last query representation before and after the cloze token.
The probability of a candidate being the answer to the question is given by a linear combination of the single word probabilities.

The proposed model is tested on 4 different dataset. 
The authors shown that the proposed model works well (state-of-the-art performance) for 3 out of 4 benchmarks.


----------

OVERALL JUDGMENT
The main contribution of the paper is the gated attention mechanism, that in my opinion, is a simple and interesting idea.
The paper is well thought, and the ablation study on the benefits given by the gated attention are convincing.
The GA reader as whole model outperforms previous state-of-the-art models on 3 benchmarks and seems very promising also on the CBT dataset.
I would have liked to see some discussion on why the model works less well on the CBT dataset, though.


----------

DETAILED COMMENTS

minor. In the introduction, Weston et al., 2014 do not use any attention mechanism.
review_title: null

final_score: rating: null
review_content: (A)
As described in equation (3) and Fig. 1, there is a Bi-GRU for the query Q at each layer indexed by k, i.e. GRU_Q^(k). 
1. Are these K GRU_Qs shared or different from each other? 
2. If different, why did the authors choose to specialize these query GRUs? Since the input (Y) to these GRUs is identical, it would be interesting to explore how is the performance affected by sharing these GRUs (i.e., using a single GRU_Q instead of K specialized ones).

(B)
The GA is applied at all the layers. How important is it to apply it to each layer? Can the authors comment on how the performance is affected if GA is omitted in the lower-layers (or higher layer)? A thorough experimental evaluation would be most insightful.
review_title: Query GRUs and GA Placement

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an interesting idea for iteratively re-weighting the word representations in a document (hence the GRU-coded doc representation as well) with a simple multiplication operation. As the authors correctly pointed out, such an operation serves as a "filter" to reduce the attentions to less relevant parts in the document, hence leading to better performance of the modeling.

The results are or close to the state-of-the-art for a few Cloze-style QA tasks. This paper would deserve an even higher score, if the following limitations could be addressed better:

1. While interesting and conceptually simple (though with significant increased computational overheads), the architecture proposed in the paper is for a very specific task. 

2. The improvement of the main idea of this paper (gated attention) is less significant, comparing GA Reader-- vs. GA Reader, while the latter includes a number of engineering tricks such as adding character embedding and using a word embedding trained from larger corpus (GloVe), as well as some small improvements on the modeling by using token-specific attention in (5).

3. I also wish the authors can shed more lights on what a role the K (number of hops) plays, both intuitively and empirically. I feel more insights could be obtained if we do more deeper analysis of K's impacts to different types of questions for example.
review_title: A simple and interesting idea for iteratively re-weighting word representations 

final_score: rating: null
review_content: Hi,
is there a specific reason why you used a character GRU instead of a convolutional neural network?
Do you have any intuitions on whether the two approaches are equivalent or one can be better than the other?
review_title: character-level embeddings

Meta-review: The paper proposes several extensions to popular attention-enhanced models for cloze-style QA. The results are near state of the art, and an ablation study hows that the different features (multiplicative interaction, gating) contribute to the model's performance. The main concern is the limited applicability of the model to other machine reading problems. The authors claim that unpublished results show applicability to other problems, but that is not sufficient defence against these concerns in the context of this paper. That said, the authors have addressed most of the other concerns brought up by the reviewers (e.g. controlling for number of hops) in the revised version. Overall however, the PCs believe that this contribution is not broad and not novel enough; we encourage the authors to resubmit.
Decision: Reject