# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.3 Context-Question Interaction
Then the outputs of the column-wise SoftMax function can be regarded as query-to-context attention weight \(\alpha\) while context-to-query attention \(\beta\) is calculated by the row-wise SoftMax function. The attention-over-attention reader (AoA Reader) model, the dynamic coattention network (DCN) and the bidirectional attention flow (Bi-DAF) network are representative MRC models with bidirectional attention. In the AoA Reader, [PERSON] et al. [18] compute the dot product between each context embedding and query embedding to obtain a similarity matching matrix \(M(i,j)\). The query-to-context and context-to-query attention are calculated as presented in Figure 12. To combine these two attentions, different from previous work in CAS Reader [53] using naive heuristics such as sum and average over the query-to-context attention, [PERSON] et al. introduce attended attention, computed by the dot product of \(\alpha\) and the average result of \(\beta\), which is later used to predict the answer. To attend to the question and the document simultaneously, [PERSON] et al. [17] fuse the two directional attentions as follows:  
\[C=\alpha[Q;\beta P], \tag{7}\]  
Figure 12: Using bidirectional attention to mine correlation between the context and question. where \(C\) can be regarded as the coattention representations that contains attention information of both context and question. Based on DCN, [PERSON] et al. [22] introduce an extension, DCN+, using residual connections to merge coattention outputs to encode richer information for the input sequences. Compared to the work on AoA Reader, [PERSON] et al. further calculate context representations with two types of directional attentive information, rather than directly using attention weights for answer prediction, which can better extract the correlation between context and question. In contrast to the AoA Reader and DCN, which directly summarize the output of two directional attention flows, [PERSON] et al. [16] let attentive vectors flow into another RNN layer to encode query-aware context representations, which can reduce information loss caused by early summarization. More specifically, after obtaining query-to-context attention weight \(\alpha\) and context-to-query attention weight \(\beta\), [PERSON] et al. compute attended context vector \(\widetilde{P}\) and attended query vector \(\widetilde{Q}\) as follows:  
\[\widetilde{P} =\sum_{i}\alpha P_{i}, \tag{8}\] \[\widetilde{Q} =\sum_{j}\beta Q_{j}.\]  
Then the context embeddings and attention vectors are combined by a simple concatenation:  
\[\alpha_{i}=\frac{\exp S_{i}}{\sum_{j}\exp S_{j}}. \tag{4}\]0  
where \(\circ\) is element-wise multiplication and \(G\) can be regarded as query-aware context representations, which are later fed to bidirectional LSTM to be further encoded. To sum up, MRC systems at the early stage usually used unidirectional attention mechanism, especially query-to-context attention, to highlight the part of the context that was more important to answer the question. However, query-to-context attention is not sufficient to extract the mutual information between the context and the query. Later, bidirectional attention was widely applied to overcome the shortcomings of unidirectional attention, which benefited from context-query correlation and output attentive representations with a fusion of context and question information. (3) One-Hop Interaction  
One-hop interaction is a shallow architecture, where the interaction between the context and the question is computed only once. Earlier context-query interaction was such a one-hop architecture in many MRC systems; for example, the Attentive Reader [5], the attention sum (AS) Reader [13], the AoA Reader [18] and so on. Although this method can do well in tackling simple cloze tests, when the question requires reasoning over multiple sentences in the context, it is hard for this one-hop interaction approach to predict the correct answer.