# Question Difficulty Estimation Based on Attention Model for Question Answering
## 4 Experiments
### Experimental Results  
We first investigate the reliability of labeling the difficulty level on QuAC dataset. The reliability is measured by the agreement between the labels tagged by multiple QA models and the human-annotated labels. To do this, we first randomly sampled 50 data samples. Then, two annotators labeled the difficulty level manually for each sample. The Kappa coefficient [42] between the annotators is 0.52, which falls under the category of 'Moderate'. This implies that the annotators have an agreement to a degree. To obtain the final level of a question from human annotations, we performed an additional procedure as done in the automatic labeling protocol. That is, a question is labeled as level 2 if at least one annotator labels it as level 2 and is labeled as level 1 if both annotators label it as level 1. We have achieved 76% agreement which implies that the labeling of the difficulty level is reliable. We also investigate the adequateness of the implementation options for \(Fuse(\cdot,\cdot)\) in Equation (3) and the direction of the relationships between a questionary sentence and information components. Both have two options. That is, \(Fuse(\cdot,\cdot)\) can be implemented by the concatenation or the element-wise summation, and the direction of the relationships can be single or dual. Table 3 summarizes the F1-scores according to the options. The F1-score of the concatenation is generally higher than that of the element-wise summation. Even if the F1-score of the concatenation is 0.26 lower in QuAC, it is much higher in both RACE and DramaQA. Thus, the concatenation is used for \(Fuse(\cdot,\cdot)\) in all the experiments below for the sake of consistency. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Parameters** & **RACE** & **QuAC** & **DramaQA** \\ \hline Encoder & Model & & BERT-Base & \\ \hline \multirow{4}{*}{DUMA} & Hidden dim. & 1536 & 1536 & 1536 \\  & No. head & 8 & 6 & 6 \\  & dropout & 0.2 & 0.2 & 0.2 \\  & Fuse & & _concat_ & \\ \hline \multirow{4}{*}{TransEncoder} & Hidden dim. & 3072 & 3072 & 4608 \\  & No. head & 4 & 4 & 4 \\ \cline{1-1}  & No. layers & 6 & 6 & 6 \\ \cline{1-1}  & dropout & 0.2 & 0.2 & 0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameter values used in the experiments. The effectiveness of the bi-directional relationships between a questionary sentence and information components is investigated by replacing DUMA in Equation (2) with a single directional multi-head attention (MHA). That is, \(\mathbf{H}_{i}\), the relationship between a questionary sentence \(q\) and each information component \(a_{i}\), is computed by  
\[\mathbf{H}_{i}=\text{MHA}(\mathbf{X}^{\epsilon},\mathbf{A}_{i}^{s},\mathbf{A}_ {i}^{s}).\]  
As shown in Table 3, the F1-score of DUMA is higher than that of MHA for all data sets, where the largest difference is 0.7 in DramaQA. This result implies that the bi-directional relationships between a questionary sentence and information components are helpful in improving the performance of question difficulty estimation. Table 4 compares the performances of the proposed model and its baselines. The first thing to note is that TACNN shows the worst performance in RACE and DramaQA. This is because TACNN does not utilize any pre-trained contextual representation even if the contextual representation is one of the key factors to improve the performance of the natural language tasks. On the other hand, it achieves slightly higher F1-score and accuracy than other pre-trained language models in QuAC. This is due to the fact that TACNN considers the relationships between a questionary sentence and each information component using an attention model explicitly, while the language models do not. Among the three pre-trained language models, BERT shows the worst performances for all data sets. RoBERTa and XLNet report similar performances on average. Especially, RoBERTa achieves the best performance in RACE with 89.82% of accuracy and 89.84 of F1-score, respectively.