# Case Study: Students' Code-Tracing Skills and Calibration of Questions for Computer Adaptive Tests
## 3 Research
### Analysis Based on the Subjective Difficulty Estimation of the Test Questions
#### 4.6.1 Comparing Students' Estimations
Without exception, all students from Year 1 estimated its difficulty as easy, only 12% of students from Year 2, and 10% from Year 3 estimated the question's difficulty as medium difficult (Figure 4). As for the question with the lowest score (Q25), the students' estimations were quite different. The majority of students in Year 1 found the question moderately difficult (medium), while the majority of students in Years 2 and 3 found its difficulty hard (Figure 5). The same trend can be observed with the points achieved on these two questions: the Year 1 students received the highest amount of points, while the Year 2 and Year 3 students received the lowest points. At the same time, the Year 2 and Year 3 students also felt that it was more difficult for them to solve the task, which supports hypothesis H2. When estimating the degree of difficulty of the full scale (the whole test), the majority of students identified it as being of medium difficulty (Table 9). Figure 4: Estimation of difficulty for the highest-score (Q42) question. Figure 5: Estimation of difficulty for the lowest-score (Q25) question. The differences in students' estimation for each year of study are illustrated in Figure 6.  
The results were tested with the chi square probe (Table 10). The data are consistent with the values shown in the descriptive statistics: the majority of students from all three years of the study estimated the test with medium difficulty value. The results show a correlation between the two variables (performance and task difficulty estimation) (\(\chi^{2}=10.73\ p=0.03\)). When students from a higher year of the study estimated a question as medium or hard, then, they achieved a lower score there. This result supports hypothesis H2. Based on year of study, the authors separately examined the five best- and worst-performing questions and the estimation associated with them. For these questions, they also compared the estimated value of the questions given by students with those defined by the instructors. In addition to Questions 42 and 25 already analyzed above, several questions received similar estimations from the students of all three years of study. Question 37 received a high score from the students in their first and second year, and Q39 received a high score from the students in their second and third year. Students in Year 1 and Year 2 achieved low scores with Q14. Questions 5 and 16 were those where students in Year 1 and Year 3 gave fewer correct answers. In the estimation of the difficulty of the five highest-scoring questions, the opinions voiced by students in Year 1 and the instructor are very similar (Figure 7). Estimations were also compared with a nonparametric statistical analysis, where the result of the Mann-Whitney test also supported the agreement of the estimations. \begin{table}
\begin{tabular}{c c c c c} \hline  & **Easy** & **Medium** & **Hard** & **Total** \\ \hline Year 1 & 12 & 66 & 0 & 78 \\ \hline Year 2 & 5 & 53 & 0 & 58 \\ \hline Year 3 & 7 & 36 & 3 & 46 \\ \hline Total & 24 & 155 & 3 & 182 \\ \hline \end{tabular}
\end{table} 
Table 10: Correlations between year of study and estimated difficulty of the questions. Figure 6: Estimating the difficulty of questions by year of study. For the lowest-scoring questions, the estimations of the students and the instructor were no longer so consistent (Figure 8). For Questions 6 and 16, the majority of students and the instructor were of the same opinion. Question 25 was marked as medium by the students and hard by the instructor. Question 43 was rated as hard by the students as opposed to medium by the instructor. Statistically, these differences are not significant. However, for Question 14, where students estimated the question as medium and the instructor estimated the question as easy, there was a significant difference in estimation based on the Mann-Whitney test (\(Z=-1.97\)\(p=0.04\)). This result supports hypothesis H3: it indicates that the Year 1 students gave a different estimation of difficulty than the instructor.