# Evaluating LLMs for Automated Scoring in Formative Assessments
## 3 Results
### Scoring Performance  
Starting with the analysis of the grades assigned by human evaluators and the open-source LLM. The Shapiro-Wilk test indicated significant deviations from normality for both distributions (\(p<0.001\)). Consequently, non-parametric methods were used in the analysis. Descriptive statistics revealed that the median grade for human evaluations was 54.17, with an Interquartile Range (IQR) of 86.11. The open-source LLM exhibited a median grade of 62.50, with an IQR of 100.00. Both distributions displayed bimodal tendencies, with peaks at 0 and 100. A Mann-Whitney U test assessed differences in scoring patterns. The results showed no statistically significant difference between human and open-source LLM grades (U = 20,911.50, \(p\) = 0.666). The effect size, calculated as \(\tau\approx 0.021\), was very small. The mean rank values were 209.98 for human evaluations and 205.02 for the open-source LLM. Table 1 presents the normalized grades for the questions, evaluated by each grader, with the following measures: minimum score (Min), maximum score (Max), the mean score from three evaluations (Mean), and standard deviation (SD), reflecting variability in scores. As  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Question**} & \multirow{2}{*}{**Type**} & \multicolumn{4}{c}{**Human**} & \multicolumn{4}{c}{**Open Source LLM**} & \multicolumn{4}{c}{**Premium LLM**} \\ \cline{2-13}  & & **Min** & **Max** & **Mean** & **SD** & **Min** & **Max** & **Mean** & **SD** & **Min** & **Max** & **Mean** & **SD** \\ \hline
2 & Short-text & 0.00 & 86.36 & 40.97 & 23.41 & 0.00 & 90.91 & 40.91 & 24.97 & 0.00 & 100.00 & 55.01 & 26.55 \\ \hline
3 & Short-text & 0.00 & 1  
shown in the table, the scores assigned by the open-source LLM and human evaluators are generally aligned across most short-text question IDs. However, differences are observed in the scoring of code questions. Now, considering the comparison between grades assigned by human evaluators and the premium LLM. The Shapiro-Wilk test confirmed significant deviations from normality for both distributions (_p_ < 0.001), necessitating the use of non-parametric methods for analysis. Figure 2: Boxplots showing distribution of normalized evaluation scores for each question by evaluator type. Descriptive statistics showed that the median grade for human evaluations was 54.17, whereas the premium LLM assigned a median of 68.18. The IQR was 86.11 for human grades and 82.16 for the premium LLM. Both distributions exhibited bimodal tendencies, with noticeable peaks at scores of 0 and 100. A Mann-Whitney U test was conducted to evaluate differences in scoring patterns between the two evaluators. The results indicated no statistically significant difference (U = 22,444.50, \(p\) = 0.392). The effect size, calculated as r \(\approx\) 0.042, suggested minimal practical differences. The mean rank values were 202.57 for human evaluations and 212.43 for the premium LLM. As shown in Table 1, the normalized grades for the questions reflect the variability in scores across evaluators. Figure 2 complements this by visually illustrating the distribution of grades, offering further insight into the variability and statistical properties observed in the table. For True/False questions ID 4, 9, and 11, which did not require justification for 'True' responses, the normalized scores were analyzed separately, and these questions were excluded from the boxplot analysis. For question ID 4, all evaluators assigned 100 to 19 responses and 0 to four responses. Similarly, for question ID 9, all evaluators assigned 100 to 21 responses and 0 to two responses. However, question ID 11 exhibited greater variability among evaluators, as shown in Figure 3.  
Notably, the alignment between the premium LLM and human evaluators was stronger for code-based questions, as shown in Table 1. However, the premium LLM generally assigned higher scores for text-based questions.