# Question Difficulty Estimation Based on Attention Model for Question Answering
## 4 Experiments
### Performance Change according to No. of Information Components  
There are different numbers of information components depending on the QA tasks and the proposed model is designed to consider a various number of information components. Thus, one consequential question about the proposed model is how the performance  
Figure 4: F1-scores for the difficulty levels of the questions in DramaQA.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model Variations** & **RACE** & **QuAC** & **DramaQA** \\ \hline Proposed model & **89.56** & **70.13** & **89.15** \\ \hline – DUMA & 88.54 & 65.22 & 86.77 \\ – TransEncoder & 88.94 & 68.81 & 88.58 \\ – DUMA and TransEncoder & 87.81 & 63.53 & 85.38 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study of the proposed over validation data.  
of the proposed model changes as the number of information components increases. Figure 5 depicts the performance changes according to the number of information components. The X-axis of this figure denotes the information components used and the Y-axis represents F1-score. In the QA tasks of our experiments, a description is the most common and important information component. Thus, it is used as a base information component for all models. The candidate answers and utterances are added consecutively in DramaQA, while only the candidate answers are added in RACE and an answer span is added in QuAC. The performances of all models in RACE increase monotonically as new components of candidate answers are added. This result seems natural because a questionary sentence (QS), a description, and candidate answers all provide somewhat information for predicting the level of question difficulty. On the other hand, the performances do not improve large in QuAC even though a new information component of an answer span is added. This is because the answer span is extracted from a description so that the information of the answer span might be already reflected by the description.  
An interesting fact is found in DramaQA. As in RACE, when a description, candidate answers, and utterances are added in order, the performances of the proposed model and TACNN increase monotonically but those of the pre-trained language models do not. Especially when utterances are newly added, the performances of the language models rather decrease. This is because the language models regard all information components as a single sequence, not as individual sequences. Although the sequence differentiates each information component with a special token [SEP], some individuality among the information components might be lost. Due to this loss of individuality, their performances decrease though the utterances are considered. On the other hand, the proposed model and TACNN treat every information component separately. Furthermore, the proposed model is superior to TACNN because it utilizes the pre-trained contextual representations and considers additional inter-information relationship. These results imply that the proposed model predicts the difficulty of questions well even when the number of information components increases.