Venue: ACL 2017

# Attention-over-Attention Neural Networks for Reading Comprehension

Yiming Cui\({}^{\dagger}\), Zhipeng Chen\({}^{\dagger}\), Si Wei\({}^{\dagger}\), Shijin Wang\({}^{\dagger}\), Ting Liu\({}^{\ddagger}\) and Guoping Hu\({}^{\dagger}\)

\({}^{\dagger}\)Joint Laboratory of HIT and iFLYTEK, iFLYTEK Research, Beijing, China

\({}^{\ddagger}\)Research Center for Social Computing and Information Retrieval,

Harbin Institute of Technology, Harbin, China

\({}^{\dagger}\){ymcui,zpchen,siwei,sjwang3,gphu}@iflytek.com

\({}^{\dagger}\)tliu@ir.hit.edu.cn

###### Abstract

Cloze-style reading comprehension is a representative problem in mining relationship between document and query. In this paper, we present a simple but novel model called _attention-over-attention_ reader for better solving cloze-style reading comprehension task. The proposed model aims to place another attention mechanism over the document-level attention and induces "attended attention" for final answer predictions. One advantage of our model is that it is simpler than related works while giving excellent performance. In addition to the primary model, we also propose an N-best re-ranking strategy to double check the validity of the candidates and further improve the performance. Experimental results show that the proposed methods significantly outperform various state-of-the-art systems by a large margin in public datasets, such as CNN and Children's Book Test.

## 1 Introduction

To read and comprehend the human languages are challenging tasks for the machines, which requires that the understanding of natural languages and the ability to do reasoning over various clues. Reading comprehension is a general problem in the real world, which aims to read and comprehend a given article or context, and answer the questions based on it. Recently, the cloze-style reading comprehension problem has become a popular task in the community. The cloze-style query Taylor (Taylor, 1953) is a problem that to fill in an appropriate word in the given sentences while taking the context information into account.

To teach the machine to do cloze-style reading comprehensions, large-scale training data is necessary for learning relationships between the given document and query. To create large-scale training data for neural networks, Hermann et al. (2015) released the CNN/Daily Mail news dataset, where the document is formed by the news articles and the queries are extracted from the summary of the news. Hill et al. (2015) released the Children's Book Test dataset afterwards, where the training samples are generated from consecutive 20 sentences from books, and the query is formed by 21st sentence. Following these datasets, a vast variety of neural network approaches have been proposed Kadlec et al. (2016); Cui et al. (2016); Chen et al. (2016); Dhingra et al. (2016); Sordoni et al. (2016); Trischler et al. (2016); Seo et al. (2016); Xiong et al. (2016), and most of them stem from the attention-based neural network Bahdanau et al. (2014), which has become a stereotype in most of the NLP tasks and is well-known by its capability of learning the "importance" distribution over the inputs.

In this paper, we present a novel neural network architecture, called _attention-over-attention_ model. As we can understand the meaning literally, our model aims to place another attention mechanism over the existing document-level attention. Unlike the previous works, that are using heuristic merging functions Cui et al. (2016), or setting various pre-defined non-trainable terms Trischler et al. (2016), our model could automatically generate an "attended attention" over various document-level attentions, and make a mutual look not only from _query-to-document_ but also _document-to-query_, which will benefit from the interactive information.

To sum up, the main contributions of our work are listed as follows.

* To our knowledge, this is the first time that

the mechanism of nesting another attention over the existing attentions is proposed, i.e. _attention-over-attention_ mechanism.
* Unlike the previous works on introducing complex architectures or many non-trainable hyper-parameters to the model, our model is much more simple but outperforms various state-of-the-art systems by a large margin.
* We also propose an N-best re-ranking strategy to re-score the candidates in various aspects and further improve the performance.

The following of the paper will be organized as follows. In Section 2, we will give a brief introduction to the cloze-style reading comprehension task as well as related public datasets. Then the proposed attention-over-attention reader will be presented in detail in Section 3 and N-best re-ranking strategy in Section 4. The experimental results and analysis will be given in Section 5 and Section 6. Related work will be discussed in Section 7. Finally, we will give a conclusion of this paper and envisions on future work.

## 2 Cloze-style Reading Comprehension

In this section, we will give a brief introduction to the cloze-style reading comprehension task at the beginning. And then, several existing public datasets will be described in detail.

### Task Description

Formally, a general Cloze-style reading comprehension problem can be illustrated as a triple:

\[\langle{\cal D},{\cal Q},{\cal A}\rangle\]

The triple consists of a document \({\cal D}\), a query \({\cal Q}\) and the answer to the query \({\cal A}\). Note that the answer is usually a _single_ word in the document, which requires the human to exploit context information in both document and query. The type of the answer word varies from predicting a preposition given a fixed collocation to identifying a named entity from a factual illustration.

### Existing Public Datasets

Large-scale training data is essential for training neural networks. Several public datasets for the cloze-style reading comprehension has been released. Here, we introduce two representative and widely-used datasets.

#### 2.1.1 CNN/ Daily Mail

Hermann et al. (2015) have firstly published two datasets: CNN and Daily Mail news data 1. They construct these datasets with web-crawled CNN and Daily Mail news data. One of the characteristics of these datasets is that the news article is often associated with a summary. So they first regard the main body of the news article as the _Document_, and the _Query_ is formed by the summary of the article, where one entity word is replaced by a special placeholder to indicate the missing word. The replaced entity word will be the _Answer_ of the _Query_. Apart from releasing the dataset, they also proposed a methodology that anonymizes the named entity tokens in the data, and these tokens are also re-shuffle in each sample. The motivation is that the news articles are containing limited named entities, which are usually celebrities, and the world knowledge can be learned from the dataset. So this methodology aims to exploit general relationships between anonymized named entities within a single document rather than the common knowledge. The following research on these datasets showed that the entity word anonymization is not as effective as expected Chen et al. (2016).

Footnote 1: The pre-processed CNN and Daily Mail datasets are available at [http://cs.nyu.edu/~kcho/DMQA/](http://cs.nyu.edu/~kcho/DMQA/)

\(\bullet\)**Children's Book Test**

There was also a dataset called the Children's Book Test (CBTest) released by Hill et al. (2015), which is built on the children's book story through Project Gutenberg 2. Different from the CNN/Daily Mail datasets, there is no summary available in the children's book. So they proposed another way to extract query from the original data. The document is composed of 20 consecutive sentences in the story, and the 21st sentence is regarded as the query, where one word is blanked with a special placeholder. In the CBTest datasets, there are four types of sub-datasets available which are classified by the part-of-speech and named entity tag of the answer word, containing Named Entities (NE), Common Nouns (CN), Verbs and Prepositions. In their studies, they have found that the answering of verbs and prepositions are relatively less dependent on the content of document, and the humans can even do preposi 

tion blank-filling without the presence of the document. The studies shown by Hill et al. (2015), answering verbs and prepositions are less dependent with the presence of document. Thus, most of the related works are focusing on solving NE and CN types.

## 3 Attention-over-Attention Reader

In this section, we will give a detailed introduction to the proposed Attention-over-Attention Reader (AoA Reader). Our model is primarily motivated by Kadlec et al., (2016), which aims to directly estimate the answer from the document-level attention instead of calculating blended representations of the document. As previous studies by Cui et al. (2016) showed that the further investigation of query representation is necessary, and it should be paid more attention to utilizing the information of query. In this paper, we propose a novel work that placing another attention over the primary attentions, to indicate the "importance" of each attentions.

Now, we will give a formal description of our proposed model. When a cloze-style training triple \(\langle{\cal D},{\cal Q},{\cal A}\rangle\) is given, the proposed model will be constructed in the following steps.

* **Contextual Embedding** We first transform every word in the document \({\cal D}\) and query \({\cal Q}\) into one-hot representations and then convert them into continuous representations with a shared embedding matrix \(W_{e}\). By sharing word embedding, both the document and query can participate in the learning of embedding and both of them will benefit from this mechanism. After that, we use two bi-directional RNNs to get contextual representations of the document and query individually, where the representation of each word is formed by concatenating the forward and backward hidden states. After making a trade-off between model performance and training complexity, we choose the Gated Recurrent Unit (GRU) Cho et al. (2014) as recurrent unit implementation.

\[e(x)=W_{e}\cdot x,~{}where~{}~{}x\in{\cal D},{\cal Q} \tag{1}\] \[\overrightarrow{h_{s}(x)}=\overrightarrow{GRU}(e(x))\] (2) \[\overleftarrow{h_{s}(x)}=\overleftarrow{GRU}(e(x))\] (3) \[h_{s}(x)=[\overrightarrow{h_{s}(x)};\overleftarrow{h_{s}(x)}] \tag{4}\]

We take \(h_{doc}\in\mathbb{R}^{|{\cal D}|*2d}\) and \(h_{query}\in\mathbb{R}^{|{\cal Q}|*2d}\) to denote the contextual representations of document and query, where \(d\) is the dimension of GRU (oneway).

* **Pair-wise Matching Score** After obtaining the contextual embeddings of the document \(h_{doc}\) and query \(h_{query}\), we calculate a pair-wise matching matrix, which indicates the pair-wise matching degree of one document word and one query word. Formally, when given \(i\)th word of the document and \(j\)th word of query, we can compute a matching score by their dot product. \[M(i,j)=h_{doc}(i)^{T}\cdot h_{query}(j)\] (5)

In this way, we can calculate every pair-wise matching score between each document and query word, forming a matrix \(M\in\mathbb{R}^{|{\cal D}|*|{\cal Q}|}\), where the value of \(i\)th row and \(j\)th column is filled by \(M(i,j)\).

* **Individual Attentions** After getting the pair-wise matching matrix \(M\), we apply a column-wise softmax function to get probability distributions in each column, where each column is an individual document-level attention when considering a single query word. We denote \(\alpha(t)\in\mathbb{R}^{|{\cal D}|}\) as the document-level attention regarding query word at time \(t\), which can be seen as a _query-to-document_ attention. \[\alpha(t)=softmax(M(1,t),...,M(|{\cal D}|,t))\] (6) \[\alpha=[\alpha(1),\alpha(2),...,\alpha(|{\cal Q}|)]\] (7)
* **Attention-over-Attention** Different from Cui et al. (2016), instead of using naive heuristics (such as _summing_ or _averaging_) to combine these individual attentions into a final attention, we introduce another attention mechanism to automatically decide the importance of each individual attention.

First, we calculate a reversed attention, that is, for every document word at time \(t\), we calculate the "importance" distribution on the query, to indicate which query words are more important given a single document word. We apply a row-wise softmax function to the pair-wise matching matrix \(M\) to get query-level attentions. We denote \(\beta(t)\in\mathbb{R}^{|{\cal Q}|}\) as the query-level attention regarding document word at time \(t\), which can be seen as a 

document-to-query_ attention.

\[\beta(t)=softmax(M(t,1),...,M(t,|{\cal Q}|)) \tag{8}\]

So far, we have obtained both _query-to-document_ attention \(\alpha\) and _document-to-query_ attention \(\beta\). Our motivation is to exploit mutual information between the document and query. However, most of the previous works are only relying on _query-to-document_ attention, that is, only calculate one document-level attention when considering the whole query.

Then we average all the \(\beta(t)\) to get an averaged query-level attention \(\beta\). Note that, we do not apply another softmax to the \(\beta\), because averaging individual attentions do not break the normalizing condition.

\[\beta=\frac{1}{n}\sum_{t=1}^{|{\cal D}|}\beta(t) \tag{9}\]

Finally, we calculate dot product of \(\alpha\) and \(\beta\) to get the "attended document-level attention" \(s\in\mathbb{R}^{|{\cal D}|}\), i.e. the _attention-over-attention_ mechanism. Intuitively, this operation is calculating a weighted sum of each individual document-level attention \(\alpha(t)\) when looking at query word at time \(t\). In this way, the contributions by each query word can be learned explicitly, and the final decision (document-level attention) is made through the voted result by the importance of each query word.

\[s=\alpha^{T}\beta \tag{10}\]

* **Final Predictions** Following Kadlec et al. (2016), we use _sum attention_ mechanism to get aggregated results. Note that the final output should be reflected in the vocabulary space \(V\), rather than document-level attention \(|{\cal D}|\), which will make a significant difference in the performance, though Kadlec et al. (2016) did not illustrate this clearly. \[P(w|{\cal D},{\cal Q})=\sum_{i\in I(w,{\cal D})}s_{i},~{}~{}w\in V\] (11) where \(I(w,{\cal D})\) indicate the positions that word \(w\) appears in the document \({\cal D}\). As the training objectives, we seek to maximize the log-likelihood of the correct answer. \[{\cal L}=\sum_{i}\log(p(x))~{}~{},x\in{\cal A}\] (12)

Figure 1: Neural network architecture of the proposed Attention-over-Attention Reader (AoA Reader).

 

The proposed neural network architecture is depicted in Figure 1. Note that, as our model mainly adds limited steps of calculations to the AS Reader Kadlec et al. (2016) and does not employ any additional weights, the computational complexity is similar to the AS Reader.

## 4 N-best Re-ranking Strategy

Intuitively, when we do cloze-style reading comprehensions, we often refill the candidate into the blank of the query to double-check its appropriateness, fluency and grammar to see if the candidate we choose is the most suitable one. If we do find some problems in the candidate we choose, we will choose the second possible candidate and do some checking again.

To mimic the process of double-checking, we propose to use N-best re-ranking strategy after generating answers from our neural networks. The procedure can be illustrated as follows.

\(\bullet\)**N-best Decoding**

Instead of only picking the candidate that has the highest possibility as answer, we can also extract follow-up candidates in the decoding process, which forms an N-best list.

\(\bullet\)**Refill Candidate into Query**

As a characteristic of the cloze-style problem, each candidate can be refilled into the blank of the query to form a complete sentence. This allows us to check the candidate according to its context.

\(\bullet\)**Feature Scoring**

The candidate sentences can be scored in many aspects. In this paper, we exploit three features to score the N-best list.

* Global N-gram LM: This is a fundamental metric in scoring sentence, which aims to evaluate its fluency. This model is trained on the document part of training data.
* Local N-gram LM: Different from global LM, the local LM aims to explore the information with the given document, so the statistics are obtained from the test-time document. It should be noted that the local LM is trained sample-by-sample, it is not trained on the entire test set, which is not legal in the real test case. This model is useful when there are many unknown words in the test sample.
* Word-class LM: Similar to global LM, the word-class LM is also trained on the document part of training data, but the words are converted to its word class ID. The word class can be obtained by using clustering methods. In this paper, we simply utilized the _mkcls_ tool for generating 1000 word classes Josef Och (1999).
* **Weight Tuning**

To tune the weights among these features, we adopt the K-best MIRA algorithm Cherry and Foster (2012) to automatically optimize the weights on the validation set, which is widely used in statistical machine translation tuning procedure.

\(\bullet\)**Re-scoring and Re-ranking**

After getting the weights of each feature, we calculate the weighted sum of each feature in the N-best sentences and then choose the candidate that has the lowest cost as the final answer.

## 5 Experiments

### Experimental Setups

The general settings of our neural network model are listed below in detail.

* Embedding Layer: The embedding weights are randomly initialized with the uniform distribution in the interval \([-0.05,0.05]\)

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & \multicolumn{3}{c}{CNN News} & \multicolumn{3}{c}{CBT NE} & \multicolumn{3}{c}{CBT CN} \\  & Train & Valid & Test & Train & Valid & Test & Train & Valid & Test \\ \hline \# Query & 380,298 & 3,924 & 3,198 & 108,719 & 2,000 & 2,500 & 120,769 & 2,000 & 2,500 \\ Max \# candidates & 527 & 187 & 396 & 10 & 10 & 10 & 10 & 10 & 10 \\ Avg \# candidates & 26 & 26 & 25 & 10 & 10 & 10 & 10 & 10 & 10 \\ Avg \# tokens & 762 & 763 & 716 & 433 & 412 & 424 & 470 & 448 & 461 \\ Vocabulary & & 118,497 & & 53,063 & & & 53,185 & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of cloze-style reading comprehension datasets: CNN news and CBTest NE / CN.

 

For regularization purpose, we adopted \(l_{2}\)-regularization to 0.0001 and dropout rate of 0.1 [36]. Also, it should be noted that we do not exploit any pre-trained embedding models.
* Hidden Layer: Internal weights of GRUs are initialized with random orthogonal matrices [48].
* Optimization: We adopted ADAM optimizer for weight updating [57], with an initial learning rate of 0.001. As the GRU units still suffer from the gradient exploding issues, we set the gradient clipping threshold to 5 [49]. We used batched training strategy of 32 samples.

Dimensions of embedding and hidden layer for each task are listed in Table 3. In re-ranking step, we generate 5-best list from the baseline neural network model, as we did not observe a significant variance when changing the N-best list size. All language model features are trained on the training proportion of each dataset, with 8-gram word-based setting and Kneser-Ney smoothing [49] trained by SRILM toolkit [15]. The results are reported with the best model, which is selected by the performance of validation set. The ensemble model is made up of four best models, which are trained using different random seed. Implementation is done with Theano [49] and Keras [10], and all models are trained on Tesla K40 GPU.

### Overall Results

Our experiments are carried out on public datasets: CNN news datasets [10] and CBTest NE/CN datasets [11]. The statistics of these datasets are listed in Table 1, and the experimental results are given in Table 2.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & \multicolumn{2}{c}{CNN News} & \multicolumn{2}{c}{CBTest NE} & \multicolumn{2}{c}{CBTest CN} \\  & Valid & Test & Valid & Test & Valid & Test \\ \hline Deep LSTM Reader [10] & 55.0 & 57.0 & - & - & - & - \\ Attentive Reader [10] & 61.6 & 63.0 & - & - & - & - \\ Human (context+query) [11] & - & - & - & _81.6_ & - & _81.6_ \\ MemNN (window + self-sup.) [11] & 63.4 & 66.8 & 70.4 & 66.6 & 64.2 & 63.0 \\ AS Reader [15] & 68.6 & 69.5 & 73.8 & 68.6 & 68.8 & 63.4 \\ CAS Reader [15] & 68.2 & 70.0 & 74.2 & 69.2 & 68.2 & 65.7 \\ Stanford AR [14] & 72.4 & 72.4 & - & - & - & - \\ GA Reader [16] & 73.0 & 73.8 & 74.9 & 69.0 & 69.0 & 63.9 \\ Iterative Attention [16] & 72.6 & 73.3 & 75.2 & 68.6 & 72.1 & 69.2 \\ EpiReader [16] & **73.4** & 74.0 & 75.3 & 69.7 & 71.5 & 67.4 \\ \hline
**AoA Reader** & 73.1 & **74.4** & **77.8** & **72.0** & **72.2** & **69.4** \\

- & **79.6** & **74.0** & **75.7** & **73.1** \\ \hline \hline MemNN (Ensemble) & 66.2 & 69.4 &
- \\ AS Reader (Ensemble) & 73.9 & 75.4 & 74.5 & 70.6 & 71.1 & 68.9 \\ GA Reader (Ensemble) & 76.4 & 77.4 & 75.5 & 71.9 & 72.1 & 69.4 \\ EpiReader (Ensemble) &
- & 76.6 & 71.8 & 73.6 & 70.6 \\ Iterative Attention (Ensemble) & 74.5 & 75.7 & 76.9 & 72.0 & 74.1 & **71.0** \\ \hline

- & **78.9** & **74.5** & **74.7** & 70.8 \\

- & **80.3** & **75.6** & **77.0** & **74.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results on the CNN news, CBTest NE and CN datasets. The best baseline results are depicted in italics, and the overall best results are in bold face.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt}} \hline \hline  & Embed. \# units & Hidden \# units \\ \hline CNN News & 384 & 256 \\ CBTest NE & 384 & 384 \\ CBTest CN & 384 & 256 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Embedding and hidden layer dimensions for each task.

 

As we can see that, our AoA Reader outperforms state-of-the-art systems by a large margin, where 2.3% and 2.0% absolute improvements over EpiReader in CBTest NE and CN test sets, which demonstrate the effectiveness of our model. Also by adding additional features in the re-ranking step, there is another significant boost 2.0% to 3.7% over AoA Reader in CBTest NE/CN test sets. We have also found that our single model could stay on par with the previous best ensemble system, and even we have an absolute improvement of 0.9% beyond the best ensemble model (Iterative Attention) in the CBTest NE validation set. When it comes to ensemble model, our AoA Reader also shows significant improvements over previous best ensemble models by a large margin and set up a new state-of-the-art system.

To investigate the effectiveness of employing _attention-over-attention_ mechanism, we also compared our model to CAS Reader, which used pre-defined merging heuristics, such as _sum_ or _avg_ etc. Instead of using pre-defined merging heuristics, and letting the model explicitly learn the weights between individual attentions results in a significant boost in the performance, where 4.1% and 3.7% improvements can be made in CNN validation and test set against CAS Reader.

### Effectiveness of Re-ranking Strategy

As we have seen that the re-ranking approach is effective in cloze-style reading comprehension task, we will give a detailed ablations in this section to show the contributions by each feature. To have a thorough investigation in the re-ranking step, we listed the detailed improvements while adding each feature mentioned in Section 4.

From the results in Table 4, we found that the NE and CN category both benefit a lot from the re-ranking features, but the proportions are quite different. Generally speaking, in NE category, the performance is mainly boosted by the \(LM_{local}\) feature. However, on the contrary, the CN category benefits from \(LM_{global}\) and \(LM_{wc}\) rather than the \(LM_{local}\).

Also, we listed the weights of each feature in Table 5. The \(LM_{global}\) and \(LM_{wc}\) are all trained by training set, which can be seen as _Global Feature_. However, the \(LM_{local}\) is only trained within the respective document part of test sample, which can be seen as _Local Feature_.

\[\eta=\frac{LM_{global}+LM_{wc}}{LM_{local}} \tag{13}\]

We calculated the ratio between the global and local features and found that the NE category is much more dependent on local features than CN category. Because it is much more likely to meet a new named entity than a common noun in the test phase, so adding the local LM provides much more information than that of common noun. However, on the contrary, answering common noun requires less local information, which can be learned in the training data relatively.

## 6 Quantitative Analysis

In this section, we will give a quantitative analysis to our AoA Reader. The following analyses are carried out on CBTest NE dataset. First, we investigate the relations between the length of the document and corresponding accuracy. The result is depicted in Figure 2.

As we can see that the AoA Reader shows consistent improvements over AS Reader on the different length of the document. Especially, when the length of document exceeds 700, the improvements become larger, indicating that the AoA Reader is more capable of handling long documents.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{CBTest NE} & \multicolumn{2}{c}{CBTest CN} \\  & Valid & Test & Valid & Test \\ \hline AoA Reader & 77.8 & 72.0 & 72.2 & 69.4 \\ +Global LM & 78.3 & 72.6 & 73.9 & 71.2 \\ +Local LM & 79.4 & 73.8 & 74.7 & 71.7 \\ +Word-class LM & 79.6 & 74.0 & 75.7 & 73.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Detailed results of 5-best re-ranking on CBTest NE/CN datasets. Each row includes all of the features from previous rows. \(LM_{global}\) denotes the global LM, \(LM_{local}\) denotes the local LM, \(LM_{wc}\) denotes the word-class LM.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & CBTest NE & CBTest CN \\ \hline NN & 0.64 & 0.20 \\ Global LM & 0.16 & 0.10 \\ Word-class LM & 0.04 & 0.39 \\ Local LM & 0.16 & 0.31 \\ \hline RATIO \(\eta\) & 1.25 & 1.58 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Weight of each feature in N-best re-ranking step. _NN_ denotes the feature (probability) produced by baseline neural network model.

 

Furthermore, we also investigate if the model tends to choose a high-frequency candidate than a lower one, which is shown in Figure 3. Not surprisingly, we found that both models do a good job when the correct answer appears more frequent in the document than the other candidates. This is because that the correct answer that has the highest frequency among the candidates takes up over 40% of the test set (1071 out of 2500). But interestingly we have also found that, when the frequency rank of correct answer exceeds 7 (less frequent among candidates), these models also give a relatively high performance. Empirically, we think that these models tend to choose extreme cases in terms of candidate frequency (either too high or too low). One possible reason is that it is hard for the model to choose a candidate that has a neutral frequency as the correct answer, because of its ambiguity (neutral choices are hard to made).

## 7 Related Work

Cloze-style reading comprehension tasks have been widely investigated in recent studies. We will take a brief revisit to the related works.

Hermann et al. (2015) have proposed a method for obtaining large quantities of triples through news articles and its summary. Along with the release of cloze-style reading comprehension dataset, they also proposed an attention-based neural network to handle this task. Experimental results showed that the proposed neural network is effective than traditional baselines.

Hill et al. (2015) released another dataset, which stems from the children's books. Different from Hermann et al. (2015)'s work, the document and query are all generated from the raw story without any summary, which is much more general than previous work. To handle the reading comprehension task, they proposed a window-based memory network, and self-supervision heuristics is also applied to learn hard-attention.

Unlike previous works, that using blended representations of document and query to estimate the answer, Kadlec et al. (2016) proposed a simple model that directly pick the answer from the document, which is motivated by the Pointer Network (Vinyals et al., 2015). A restriction of this model is that the answer should be a single word and appear in the document. Results on various public datasets showed that the proposed model is effective than previous works.

Liu et al. (2016) proposed to exploit reading comprehension models to other tasks. They first applied the reading comprehension model into Chinese zero pronoun resolution task with automatically generated large-scale pseudo training data. The experimental results on OntoNotes 5.0 data showed that their method significantly outperforms various state-of-the-art systems.

Our work is primarily inspired by Cui et al. (2016) and Kadlec et al. (2016) , where the latter model is widely applied to many follow-up works (Sordoni et al., 2016; Trischler et al., 2016; Cui et al., 2016). Unlike the CAS Reader (Cui et al., 2016), we do not assume any heuristics to our model, such as using merge functions: \(sum\), \(avg\) etc. We used a mechanism called "attenon

Figure 3: Test accuracy against the frequency rank of the answer. The bar below the figure indicates the number of samples in each rank.

Figure 2: Test accuracy against the length of the document. The bar below the figure indicates the number of samples in each interval.

 

over-attention" to explicitly calculate the weights between different individual document-level attentions, and get the final attention by computing the weighted sum of them. Also, we find that our model is typically general and simple than the recently proposed model, and brings significant improvements over these cutting edge systems.

## 8 Conclusion

We present a novel neural architecture, called attention-over-attention reader, to tackle the cloze-style reading comprehension task. The proposed AoA Reader aims to compute the attentions not only for the document but also the query side, which will benefit from the mutual information. Then a weighted sum of attention is carried out to get an attended attention over the document for the final predictions. Among several public datasets, our model could give consistent and significant improvements over various state-of-the-art systems by a large margin.

The future work will be carried out in the following aspects. We believe that our model is general and may apply to other tasks as well, so firstly we are going to fully investigate the usage of this architecture in other tasks. Also, we are interested to see that if the machine really "comprehend" our language by utilizing neural networks approaches, but not only serve as a "document-level" language model. In this context, we are planning to investigate the problems that need comprehensive reasoning over several sentences.

## Acknowledgments

We would like to thank all three anonymous reviewers for their thorough reviewing and providing thoughtful comments to improve our paper. This work was supported by the National 863 Leading Technology Research Project via grant 2015AA015409.

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Chen et al. (2016) Danqi Chen, Jason Bolton, and D. Christopher Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 2358-2367. [https://doi.org/10.18653/v1/P16-1223](https://doi.org/10.18653/v1/P16-1223).
* Cherry and Foster (2012) Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Montreal, Canada, pages 427-436. [http://www.aclweb.org/anthology/N12-1047](http://www.aclweb.org/anthology/N12-1047).
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1724-1734. [http://aclweb.org/anthology/D14-1179](http://aclweb.org/anthology/D14-1179).
* Chollet (2015) Francois Chollet. 2015. Keras. [https://github.com/fchollet/keras](https://github.com/fchollet/keras).
* Cui et al. (2016) Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. The COLING 2016 Organizing Committee, pages 1777-1786. [http://aclweb.org/anthology/C16-1167](http://aclweb.org/anthology/C16-1167).
* Dhingra et al. (2016) Bhuwan Dingra, Hanxiao Liu, William W Cohen, and Ruslan Salakhutdinov. 2016. Gated-attention readers for text comprehension. _arXiv preprint arXiv:1606.01549_ .
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems_. pages 1684-1692.
* Hill et al. (2015) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representations. _arXiv preprint arXiv:1511.02301_ .
* Och (1999) Franz Josef Och. 1999. An efficient method for determining bilingual word classes. In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_. [http://aclweb.org/anthology/E99-1010](http://aclweb.org/anthology/E99-1010).
* Kadlec et al. (2016) Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 908-918. [https://doi.org/10.18653/v1/P16-1086](https://doi.org/10.18653/v1/P16-1086) 

* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ .
* Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In _International Conference on Acoustics, Speech, and Signal Processing_. pages 181-184 vol.1.
* Liu et al. (2016) Ting Liu, Yiming Cui, Qingyu Yin, Shijin Wang, Weinan Zhang, and Guoping Hu. 2016. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. _arXiv preprint arXiv:1606.01603_ .
* Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. _ICML (3)_ 28:1310-1318.
* Saxe et al. (2013) Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. _arXiv preprint arXiv:1312.6120_ .
* Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2016. Bi-directional attention flow for machine comprehension. _arXiv preprint arXiv:1611.01603_ .
* Sordoni et al. (2016) Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. _arXiv preprint arXiv:1606.02245_ .
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_ 15(1):1929-1958.
* Stolcke (2002) Andreas Stolcke. 2002. Srilm -- an extensible language modeling toolkit. In _Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002)_. pages 901-904.
* Taylor (1953) Wilson L Taylor. 1953. Cloze procedure: a new tool for measuring readability. _Journalism and Mass Communication Quarterly_ 30(4):415.
* Team (2016) Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. _arXiv e-prints_ abs/1605.02688. [http://arxiv.org/abs/1605.02688](http://arxiv.org/abs/1605.02688).
* Trischler et al. (2016) Adam Trischler, Zheng Ye, Xingdi Yuan, Philip Bachman, Alessandro Sordoni, and Kaheer Suleman. 2016. Natural language comprehension with the epireader. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 128-137. [http://aclweb.org/anthology/D16-1013](http://aclweb.org/anthology/D16-1013).
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Advances in Neural Information Processing Systems_. pages 2692-2700.
* Xiong et al. (2016) Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. _arXiv preprint arXiv:1611.01604_ .