# Question Difficulty Estimation Based on Attention Model for Question Answering
## 4 Experiments
### Ablation Study  
We investigate the effectiveness of DUMA and _TransEncoder_ in the proposed model. Table 5 shows the result of an ablation study over the validation set. The F1-scores of the proposed model over the validation sets of each task are 89.56 for RACE, 70.13 for  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Data Set**} & \multicolumn{2}{c}{**RACE**} & \multicolumn{2}{c}{**QuAC**} & \multicolumn{2}{c}{**DramaQA**} \\ \cline{2-7}  & **Acc. (\%)** & **F1-Score** & **Acc. (\%)** & **F1-Score** & **Acc. (\%)** & **F1-Score** \\ \hline BERT & 87.75 & 87.55 & 58.31 & 58.25 & 87.95 & 87.89 \\ RoBERTa & **89.82** & **89.84** & 58.72 & 58.34 & 88.81 & 88.73 \\ XLNet & 89.02 & 88.22 & 58.51 & 58.48 & 89.07 & 89.05 \\ TACNN & 87.27 & 87.12 & 60.71 & 59.87 & 84.46 & 84.72 \\ \hline Proposed model & 89.81 & **89.84** & **68.23** & **68.37** & **89.53** & **89.59** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performances of question difficulty estimation.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Implementation Option**} & **RACE** & **QuAC** & **DramaQA** \\ \hline \multirow{2}{*}{\(Fuse(\cdot,\cdot)\)} & _concat_ & **89.56** & 70.13 & **89.15** \\  & _summation_ & 88.40 & **70.39** & 88.15 \\ \hline \multirow{2}{*}{Relationship direction} & single (MHA) & 89.26 & 69.74 & 88.45 \\  & dual (DUMA) & **89.56** & **70.13** & **89.15** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The F1-scores according to the implementation options for \(Fuse(\cdot,\cdot)\) and the direction of relationships between a questionary sentence and information components.  
QuAC, and 89.15 for DramaQA. The '-' symbol in front of a module indicates exclusion of the module. Thus, '- DUMA' implies that DUMA is excluded from the proposed model. Without DUMA, the F1-score drops up to 4.91 from that of the proposed model, which implies that the bi-directional relationships between a questionary sentence and information components represented by DUMA helps improve the performance of the proposed model.  
A similar phenomenon is observed with _TransEncoder_ in Equation (4). '- TransEncoder' implies that the concatenation \(\mathbf{H}\) of bi-directional relationships \(\mathbf{H}_{i}\)'s is directly used as an input of the pooling layer of the final classifier. Its F1-score also drops up to 1.32 from that of the proposed model, which proves the consideration of inter-information relationship is helpful in boosting the performance of the proposed model. In order to take a close look at this result, the F1-scores of each difficulty level are further investigated. Figure 4 depicts the F1-scores for every difficulty level of the questions in DramaQA. When comparing F1-scores of the proposed model with those without _TransEncoder_, the improvement in difficult (level 3 and level 4) questions is larger than that in easy (level 1 and level 2) questions. Especially at level 4, the proposed model achieves 97.40 of F1-score, but the model without _TransEncoder_ shows just 94.50. Finally, the model without both DUMA and _TransEncoder_ demonstrates the worst performance for all data sets. From these results, we can conclude that the adoption of DUMA for bi-directional relationships and _TransEncoder_ for inter-information relationship are effective to predicting the level of question difficulty.