# Question Difficulty Estimation Based on Attention Model for Question Answering
## 3 Attention-Based Question Difficulty Estimation
### Representing Relationships Using Attention Model  
The attention model is responsible for capturing the relationships between \(q\) and \(A\), and the model consists of two attention modules: a dual multi-head co-attention and a transformer encoder based on the multi-head attention. The proposed model first represents the relationships between \(q\) and every \(a_{i}\in A\) directly since the questionary sentence \(q\) is a key factor in the question-and-answering. Thus, all information components should be represented in accordance with the questionary sentence. However, these representations do not express the relationship among \(a_{i}\)'s sufficiently. Although the inter-information among \(a_{i}\)'s is reflected indirectly and slightly through the relationships between \(q\) and \(a_{i}\)'s, a direct inter-information relationship plays an important role in estimating the question difficulty and thus the second attention module is designed to consider the inter-information relationship directly. In order to identify the bi-directional relationship between \(q\) and \(a_{i}\) (\(1\leq i\leq n\)), the proposed model adopts the dual multi-head co-attention (DUMA) [18]. DUMA is composed of two multi-head attentions where each multi-head attention captures a single directional attention representation. Thus, it captures both representations from \(q\) to \(a_{i}\) and from \(a_{i}\) to \(q\). Then, it fuses these two representations to obtain a final unified representation. That is, the relationship between \(q\) and \(a_{i}\), denoted as \(\mathbf{H}_{i}\), is obtained by applying DUMA to the representations of \(\mathbf{X}^{s}\) and \(\mathbf{A}^{s}_{i}\) in Equation (1). \[\mathbf{H}_{i} = \text{DUMA}(\mathbf{X}^{s},\mathbf{A}^{s}_{i}) \tag{2}\] \[= Fuse(\text{MHA}(\mathbf{X}^{s},\mathbf{A}^{s}_{i},\mathbf{A}^{s }_{i}),\text{MHA}(\mathbf{A}^{s}_{i},\mathbf{X}^{s},\mathbf{X}^{s})), \tag{3}\]  
where \(\text{MHA}(\cdot,\cdot,\cdot)\) denotes a multi-head attention and \(Fuse(\cdot,\cdot)\) is a function for fusing two representations dynamically. The multi-head attention \(\text{MHA}(\cdot,\cdot,\cdot)\) is an attention mechanism to obtain a representation by paying attention jointly to the information from different representations at different positions [33], where the attention is obtained by applying the scaled dot-product attention several times in parallel and then concatenating the results of the attention. Formally, the multi-head attention maps a sequence of query \(\mathbf{Q}\) and a set of key-value pairs of \(\mathbf{K}\) and \(\mathbf{V}\) to a representation by  
\[\text{MHA}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{Concat}(head_{1},\ldots,head_{h})W^{O}\] \[head_{i} = \text{Attention}(\mathbf{Q}W^{\mathbf{Q}}_{i},\mathbf{K}W^{ \mathbf{K}}_{i},\mathbf{V}W^{\mathbf{V}}_{i}),\]  
where \(W^{\mathbf{Q}}_{i}\), \(W^{\mathbf{K}}_{i}\), \(W^{\mathbf{V}}_{i}\), and \(W^{O}\) are all learnable parameters. Here, \(\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})\) represents the scaled dot-product attention. It is a weighted sum of the values of which weight is determined by the dot product of the query with all the keys. Thus, it is defined as  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{softmax}\Bigg{(}\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{ k}}}\Bigg{)}\mathbf{V},\]  
where \(d_{k}\) is a key dimensionality that works for a scaling factor. Among several candidates of \(Fuse(\cdot,\cdot)\) function in Equation (3), the performance of using the concatenation is higher than that of using the element-wise summation according to our experiments below (see Section 4.2). This result complies with the results of the previous study by [PERSON] et al. [18], and thus the concatenation is used as a fuse function in this paper.