# Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy
## Appendix D Additional Experimental Results
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Grouped**} & \multicolumn{6}{c}{**Individual**} \\ \hline  & **COR.** & **REV.** & **GUL** & **DIA.** & **ENC.** & **COR.** & **REV.** & **GUL** & **DIA.** & **ENC.** \\ \hline
**Feedback Evaluator** & & & & & & & & & & & \\ \hline \multicolumn{10}{c}{**Feedback Generator: GPT-4o**} \\ \hline GPT-4o & 0.875 & 0.911 & 0.675 & 0.446 & 0.736 & 0.868 & 0.593 & 0.718 & 0.679 & 0.929 \\ Llama-3-1-8B & 0.636 & 0.475 & 0.539 & 0.421 & 0.275 & 0.543 & 0.482 & 0.207 & 0.339 & 0.250 \\ Llama-3-1-70B & 0.925 & 0.825 & 0.689 & 0.539 & 0.375 & 0.711 & 0.757 & 0.657 & 0.586 & 0.821 \\ Gemma-9B & 0.996 & 0.996 & 0.832 & 0.671 & 0.654 & 0.975 & 0.979 & 0.904 & 0.854 & 0.911 \\ Gemma-27B & 0.986 & 0.993 & 0.761 & 0.804 & 0.889 & 0.954 & 0.936 & 0.768 & 0.925 & 0.529 \\ \hline
**Feedback Evaluator** & & & & & & & & & & \\ \hline \multicolumn{10}{c}{**Feedback Generator: Claude-3**} \\ \hline GPT-4o & 0.954 & 0.961 & 0.639 & 0.682 & 0.675 & 0.914 & 0.632 & 0.625 & 0.804 & 0.914 \\ Llama-3-1-8B & 0.811 & 0.643 & 0.575 & 0.375 & 0.182 & 0.629 & 0.582 & 0.196 & 0.296 & 0.239 \\ Llama-3-1-70B & 0.993 & 0.932 & 0.721 & 0.546 & 0.407 & 0.864 & 0.857 & 0.650 & 0.664 & 0.868 \\ Gemma-9B & 0.996 & 0.996 & 0.793 & 0.729 & 0.636 & 0.996 & 0.999 & 0.939 & 0.911 & 0.886 \\ Gemma-27B & 0.999 & 0.989 & 0.639 & 0.907 & 0.854 & 0.989 & 0.971 & 0.818 & 0.964 & 0.693 \\ \hline
**Feedback Evaluator** & & & & & & & & & & \\ \hline GPT-4o & 0.882 & 0.818 & 0.721 & 0.654 & 0.754 & 0.936 & 0.789 & 0.768 & 0.846 & 0.932 \\ Llama-3-1-8B & 0.568 & 0.464 & 0.496 & 0.328 & 0.232 & 0.546 & 0.443 & 0.254 & 0.296 & 0.254 \\ Llama-3-1-70B & 0.904 & 0.729 & 0.746 & 0.611 & 0.500 & 0.807 & 0.829 & 0.750 & 0.818 & 0.889 \\ Gemma-9B & 0.986 & 0.975 & 0.850 & 0.775 & 0.704 & 0.999 & 0.961 & 0.975 & 0.964 & 0.914 \\ Gemma-27B & 0.964 & 0.914 & 0.843 & 0.864 & 0.925 & 0.982 & 0.861 & 0.911 & 0.968 & 0.675 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Inner agreement for the generation prompt strategy w/o criteria in the MC160 dataset. \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Grouped**} & \multicolumn{8}{c}{**Individual**} \\ \hline  & **COR.** & **REV.** & **GUI.** & **DIA.** & **ENC.** & **COR.** & **REV.** & **GUI.** & **DIA.** & **ENC.** \\ \hline
**Feedback Generator** & \multicolumn{8}{c}{************w/criteria**} \\ \hline GPT-4o & 0.962 & 0.914 & 0.059 & 0.075 & 0.248 & 0.828 & 0.780 & 0.331 & 0.108 & 0.149 \\ Claude-3 & 0.981 & 0.943 & 0.068 & 0.054 & 0.494 & 0.903 & 0.919 & 0.453 & 0.096 & 0.163 \\ Llama-3.1-70B & 0.980 & 0.914 & 0.107 & 0.046 & 0.246 & 0.897 & 0.849 & 0.571 & 0.125 & 0.062 \\ \hline
**Feedback Generator** & \multicolumn{8}{c}{******w/o criteria**} \\ \hline GPT-4o & 0.866 & 0.808 & 0.238 & 0.098 & 0.216 & 0.678 & 0.536 & 0.305 & 0.104 & 0.308 \\ Claude-3 & 0.958 & 0.913 & 0.125 & 0.100 & 0.203 & 0.847 & 0.574 & 0.253 & 0.213 & 0.163 \\ Llama-3.1-70B & 0.753 & 0.512 & 0.129 & 0.123 & 0.293 & 0.768 & 0.380 & 0.343 & 0.250 & 0.169 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Inter-agreement for the MC500 dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **COR.** & **REV.** & **GUI.** & **DIA.** & **ENC.** & **Avg. Confidence** \\ \hline Annotator 1 & 1 & 1 & 1 & 1 & 1 & 3 \\ Annotator 2 & 1 & 1 & 1 & 1 & 1 & 4.4 \\ Annotator 3 & 1 & 1 & 1 & 1 & 0 & 5 \\ Annotator 4 & 1 & 1 & 1 & 1 & 5 \\ Annotator 5 & 1 & 1 & 1 & 1 & 5 \\ \hline Human Agreement & 1 & 1 & 1 & 1 & 0 & 4.48 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Evaluation results and average confidence score examples for generation prompt strategy w/ criteria by human annotators on the MC160 dataset.