Title: Gated-Attention Readers for Text Comprehension
Authors: Bhuwan Dhingra
Hanxiao Liu
Zhilin Yang
William W. Cohen
Ruslan Salakhutdinov
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Supervised Learning

Abstract: In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel attention mechanism, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task--the CNN \& Daily Mail news stories and the Who Did What dataset. The effectiveness of multiplicative interaction is demonstrated by an ablation study, and by comparing to alternative compositional operators for implementing the gated-attention. 


Review discussion:
Hi,
is there a specific reason why you used a character GRU instead of a convolutional neural network?
Do you have any intuitions on whether the two approaches are equivalent or one can be better than the other?
There is some previous work showing the effectiveness of recurrent neural networks for modeling characters for named entity recognition [1,2].
Hence we used an RNN (specifically GRU) as the character encoder for WDW, which has named entities as answers, and CBT, which has either named entities or common nouns as answers.
While both RNNs and CNNs are applicable for character-level feature extraction, we cannot make a general statement whether RNNs or CNNs are more suited for this task, and that itself may be an interesting direction for future research.
[1] Yang et al, 2016, Multi-Task Cross-Lingual Sequence Tagging from Scratch, arxiv.org
[2] Lample et al, 2016. Neural architectures for named entity recognition, arxiv.org
Updated Tables 3,4,5 with new comparisons of GA Reader's performance when word embeddings are either held fixed or updated during training. We observe that for smaller datasets (WDW and CBT) keeping the word embeddings fixed leads to substantial improvements (>1%).
Also, retuned the character GRU hidden state size to 25 for WDW and CBT, which was previously 50.
(A)
As described in equation (3) and Fig. 1, there is a Bi-GRU for the query Q at each layer indexed by k, i.e. GRU_Q^(k).
1. Are these K GRU_Qs shared or different from each other?
2. If different, why did the authors choose to specialize these query GRUs? Since the input (Y) to these GRUs is identical, it would be interesting to explore how is the performance affected by sharing these GRUs (i.e., using a single GRU_Q instead of K specialized ones).
(B)
The GA is applied at all the layers. How important is it to apply it to each layer? Can the authors comment on how the performance is affected if GA is omitted in the lower-layers (or higher layer)? A thorough experimental evaluation would be most insightful.
(A) The query GRUs (and hence the query representations) in our model are different in each layer, i.e. they do not share parameters. In this way, each layer can focus on a particular semantic aspect and/or granularity for the gating operation. We figure that adding diversity to query representations is more effective than reinforcing the same information at every layer repetitively.
In fact, we compared the two schemes (share v.s. not share) experimentally and found them to perform similarly for the datasets we consider. In the paper we present the more general formulation of not sharing parameters.
(B) Thank you for the suggestion. Applying the GA module selectively between the layers is equivalent to replacing the single-layer document GRUs with multilayer versions, and then retuning both the total number of hops (K) and the GRU depth at each hop. We agree this is an interesting direction to explore, though beyond the primary focus of this paper. As shown in our ablation study in Sec 4.4, a statistically significant performance drop is observed when all the GA modules are omitted, which justifies the effectiveness of the gated-attention mechanism. We leave the study regarding the optimum depth of each document GRU for future research.
SUMMARY.
The paper proposes a machine reading approach for cloze-style question answering.
The proposed system first encodes the query and the document using a bidirectional gru. These two representations are combined together using a Gated Attention (GA).
GA calculates the compatibility of each word in the document and the query as a probability distribution.
For each word in the document a gate is calculated weighting the query representation according to the word compatibility.
Ultimately, the gate is applied to the gru-encoded document word.
The resulting word vectors are re-encoded with a bidirectional GRU.
This process is performed for multiple hops. After k hops, the probability of a word to be part of the answer is calculated by a log-linear model that take as input the last word representations, and the concatenation of the last query representation before and after the cloze token.
The probability of a candidate being the answer to the question is given by a linear combination of the single word probabilities.
The proposed model is tested on 4 different dataset.
The authors shown that the proposed model works well (state-of-the-art performance) for 3 out of 4 benchmarks.
OVERALL JUDGMENT
The main contribution of the paper is the gated attention mechanism, that in my opinion, is a simple and interesting idea.
The paper is well thought, and the ablation study on the benefits given by the gated attention are convincing.
The GA reader as whole model outperforms previous state-of-the-art models on 3 benchmarks and seems very promising also on the CBT dataset.
I would have liked to see some discussion on why the model works less well on the CBT dataset, though.
DETAILED COMMENTS
minor. In the introduction, Weston et al., 2014 do not use any attention mechanism.
We thank the reviewer for the valuable feedback. The GA Reader trained with fixed word embeddings achieves state of the art results on CBT-NE but not so on CBT-CN (please see the revisions after 12/2/16). In general, we observed that the small size of the dataset, combined with the open-domain nature of children’s book stories suggests that overfitting was an issue with these datasets. Our model, though conceptually simple, is highly expressive, but a consequence of that is that larger amounts of training data may be required to obtain improvements.
This paper presents an interesting idea for iteratively re-weighting the word representations in a document (hence the GRU-coded doc representation as well) with a simple multiplication operation. As the authors correctly pointed out, such an operation serves as a "filter" to reduce the attentions to less relevant parts in the document, hence leading to better performance of the modeling.
The results are or close to the state-of-the-art for a few Cloze-style QA tasks. This paper would deserve an even higher score, if the following limitations could be addressed better:
1.
While interesting and conceptually simple (though with significant increased computational overheads), the architecture proposed in the paper is for a very specific task.
2. The improvement of the main idea of this paper (gated attention) is less significant, comparing GA Reader-- vs. GA Reader, while the latter includes a number of engineering tricks such as adding character embedding and using a word embedding trained from larger corpus (GloVe), as well as some small improvements on the modeling by using token-specific attention in (5).
3. I also wish the authors can shed more lights on what a role the K (number of hops) plays, both intuitively and empirically. I feel more insights could be obtained if we do more deeper analysis of K's impacts to different types of questions for example.
We thank the reviewer for the valuable feedback. We agree that a deeper analysis of the performance gain w.r.t. different components in the model is important in order to better understand the architecture. To this end, we have added more analyses to Tables 4 and 6 in the revised draft, and related discussions at the end of section 4.4. Below we address the specific points you raised:
1.
While the experiments and architecture presented in the current paper are indeed for the specific task of cloze-style QA, we believe the main idea of GA is applicable in other settings where a sequence needs to be processed conditioned on external context. One possible application is QA over videos. There is already some evidence of the utility of multiplicative interactions in visual QA, see for example [1].
We would also like to point out that other researchers have already successfully applied the preprint version GA Reader-- to challenging datasets for other tasks such as language modeling (LAMBADA [2]) and extractive question answering (SQuAD [3]), with SOTA or competitive results, indicating its generality. The current paper adds further improvements to that unpublished version.
2. Table 6 presents an ablation study on the components added on top of GA Reader--. Character embeddings are the least important, and were not applied for CNN/Daily Mail datasets anyway due to anonymization (Section 4.2). The use of token-specific attentions and glove embeddings are both important additions. Since the strongest baseline, NSE, also uses Glove embeddings we believe that the improvement provided by our main contribution, the GA module (with token-specific attentions), is important.
Further, the significance of this module is established in section 4.4.
3. Our new empirical results in Table 4 show that increasing the number of hops leads to substantial increase in performance up to K=3, which plateaus beyond that. Intuitively, visualization of the attention at intermediate layers in Figures 3-7 shows that the model tends to focus on distinct salient tokens in the query after each layer. This indicates that multiple hops allow the model to combine multiple pieces of information from the query, mimicking a shallow reasoning process. We agree that a deeper analysis on the effect of K on different types of queries would be most informative, unfortunately this is beyond the scope of this work since such query type annotations are not readily available nor easily constructed for any of the datasets we consider.
[1] Fukui, Akira, et al. "Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding." arXiv preprint arXiv:1606.01847 (2016).
[2] Chu, Zewei, et al. "Broad Context Language Modeling as Reading Comprehension." arXiv preprint arXiv:1610.08431 (2016).
[3] Yang, Zhilin, et al. "Words or Characters? Fine-grained Gating for Reading Comprehension." arXiv preprint arXiv:1611.01724 (2016).
Summary:
The authors propose a multi-hop "gated attention" model, which models the interactions between query and document representations, for answering cloze-style questions. The document representation is attended to sequentially over multiple-hops using similarity with the query representation (using a dot-product) as the scoring/attention function.
The proposed method improves upon (CNN, Daily Mail, Who-Did-What datasets) or is comparable to (CBT dataset) the state-of-the-art results.
Pros:
1. Nice idea on heirarchical attention for modulating the context (document) representation by the task-specific (query) representation.
2. The presentation is clear with thorough experimental comparison with the latest results.
Comments:
1. The overall system presents a number of architectural elements: (1) attention at multiple layers (multi-hop), (2) query based attention for the context (or gated attention), (3) encoding the query vector at each layer independently.
It is important to breakdown the gain in performance due to the above factors: the ablation study presented in section 4.4 helps establish the importance of Gated Attention (#2 above).
However, it is not clear:
(1) how much multiple-hops of gated-attention contribute to the performance.
(2) how important is it to have a specialized query encoder for each layer.
Understanding the above better, will help simplify the architecture.
2. The tokens are represented using L(w) and C(w). It is not clear if C(w) is crucial for the performance of the proposed method.
There is a significant performance drop when C(w) is absent (e.g. in "GA Reader--";
although there are other changes in "GA Reader--" which could affect the performance).
Hence, it is not clear how much does the main idea, i.e., gated attention contributes towards the superior performance of the proposed method.
We thank the reviewers for the valuable feedback. We agree that a deeper analysis of the gain of performance due to the different components in the model is important in order to better understand the architecture. To this end, we have added more analyses to Tables 4 and 6 in the revised draft, and a discussion of these at the end of section 4.4. Below we address the specific points you raised:
1. (1) We have added results on the effect of varying the number of hops K to table 4. We can see that there is a significant and stable increase in performance as we go from K=1 (which is the AS reader) to K=3 hops, which then plateaus at K=4 hops. This is in line with our expectation from the visualizations presented in section 4.5 (Figures 3-7), where we show that after each layer the model typically attends to distinct tokens in the query to derive the correct answer.
(2) It is certainly possible to share parameters for different components of the system, including the query encoders at each layer.
However, the decision to do so is dependent on the particular application;
in particular for the datasets we consider we found no benefit in doing so (please see the comment on 4 Dec below). In the paper we present the more general formulation of not sharing the parameters, since that allows the query representations, and hence the filters applied at each layer to be different.
2. Table 6 presents an ablation study on the different components added on top of GA Reader--. We see that even without C(w), the proposed model, despite its relative simplicity, achieves SOTA results on WDW. We would also like to emphasize that C(w) was not used for CNN and Daily Mail datasets where again GA has SOTA performance. The main focus of the paper is the GA module, which we show in section 4.4 has a statistically significant impact on the final performance.
Based on the reviewer feedback we have added the following:
1. Analysis of the effect of number of hops K on the performance to Table 4.
2. Ablation study of the components in the current version added on top of GA Reader-- to Table 6.
A discussion of these is added to section 4.4, and also provided in response to the reviews here.
Note - Mistakenly pushed the same revision twice on 12/21/2016.
Respected Reviewers,
We would like to thank you again for the comments and suggestions. We believe that we have addressed most of the concerns raised in the revision posted on 12-21-2016.
We request you to kindly review the changes and our rebuttal below, and reconsider your scores if satisfied.
Otherwise, kindly let us know how we can improve the paper further (before the review period ends on Jan 20 this Friday).
Best Regards,
Authors.

Meta-review:
role: PC
data: comment: The paper proposes several extensions to popular attention-enhanced models for cloze-style QA. The results are near state of the art, and an ablation study hows that the different features (multiplicative interaction, gating) contribute to the model's performance. The main concern is the limited applicability of the model to other machine reading problems. The authors claim that unpublished results show applicability to other problems, but that is not sufficient defence against these concerns in the context of this paper. That said, the authors have addressed most of the other concerns brought up by the reviewers (e.g. controlling for number of hops) in the revised version. Overall however, the PCs believe that this contribution is not broad and not novel enough; we encourage the authors to resubmit.

Decision: Reject