## 4 Results
### Real-World Validation on 100 Human-Created Exam Questions (Task 1)  
To assess the framework's practical utility beyond the experimental conditions, a set of 100 human-authored exam questions was assembled. Each question fell under the theory-oriented course category, was application-based in [PERSON]'s taxonomy, and had a multi-step design. The recommended agent, identified in earlier sections based on the mixed-effects modeling and subsequent evaluations, was applied to these questions for Task 1. A panel of human experts likewise scored each question's alignment (averaging concept coverage, specificity/relevance, and accuracy/correctness), thereby generating a reference for comparison.  
Table 2 presents the key findings, including a paired \(t\)-test of the agent's average scores against the human scores, along with Bland-Altman statistics. The mean difference (\(\approx\)0.23) points) is relatively small, and the \(p\)-value (\(\approx\)0.1224) indicates **no statistically significant difference** in overall mean scores. This suggests that, on average, the agent's alignment evaluations are comparable to those of human experts.  
Figure 10 provides a Bland-Altman plot, illustrating each question's difference (agent minus human) against the mean of the two scores. The \"mean difference\" (red dashed line) is centered slightly above zero, reflecting the 0.23-point bias. The \"95% limits of  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Statistic** & **Value** \\ \hline Number of Questions (N) & 100 \\ Mean Agent Score & 6.88 \\ Mean Human Score & 6.65 \\ Mean Difference (Agent–Human) & 0.23 \\ Std. Dev. of Difference & 1.45 \\ Upper Limit of Agreement (LoA) & 3.08 \\ Lower Limit of Agreement (LoA) & \(-\)2.62 \\ Paired t-Statistic & 1.56 \\ \(p\)-Value & 0.1224 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Paired \(t\)-test and Bland–Altman summary for the 100-question real-world validation set (Task 1).  
Figure 9: Meta evaluator identified bias: (**a**) Grading Bias—Negative values indicate that human evaluators scored lower than the automatic meta evaluator, while positive values indicate higher human scores. Each box summarizes the distribution of bias across all questions in that category. (**b**) Explanation Quality Bias—A negative distribution means human evaluators found lower explanation quality relative to the automatic meta evaluation, and a positive distribution means higher human ratings. Each box represents the variability in bias across multiple questions in the corresponding task–agent grouping.  
agreement\" (green dashed lines) span from \((-2.62)\) to \((3.08)\), suggesting that, while the agent occasionally overestimates or underestimates certain items, most points fall within a reasonable range of agreement.  
These results demonstrate that the discrepancies between the agent and the human experts are generally small and not statistically significant. While occasional outliers remain, the overall trend supports using this best-performing agent to autonomously assess alignment for multi-step, theory-oriented, application-based questions. In practice, instructor oversight continues to play a vital role, particularly for questions lying near the edges of the distribution. Nonetheless, the fact that the system's outputs do not systematically diverge from human judgments underscores the real-world viability of the proposed framework--indicating that it can be confidently deployed for large-scale alignment tasks while still allowing educators to perform final quality checks as needed.