# Evaluating LLMs for Automated Scoring in Formative Assessments
## 5 Conclusions  
This study explored the potential of LLMs to automate scoring in formative assessments, emphasizing their reliability and consistency compared to human evaluators. By developing the introEduAI platform, we demonstrated the feasibility of using LLMs for scalable and precise scoring workflows.  
The evaluation of the open-source LLaMA 3.2 and premium OpenAI GPT-4o revealed distinct strengths and limitations. The premium LLM closely aligned with human scoring patterns, showing high correlations and minimal variability, particularly in coding responses. In contrast, open-source models, while cost-effective and privacy-friendly, exhibited greater variability, underscoring the need for further optimization to match premium model performance. However, we must acknowledge the rapid evolution of new language models, particularly open-source alternatives like DeepSee [31], especially when using reasoning methodologies, which show promising potential to match or exceed the performance of premium commercial models in the near future.  
However, this study focuses on computer programming assessments characterized by objective and structured responses. While these tasks are suitable for testing LLM precision and consistency, the findings may not extend to disciplines where scoring involves interpretative judgment and handling ambiguity.  
Furthermore, the scope of question types used in this study (primarily two code-based questions and seven short-answer text responses) limits our understanding of LLM adaptability. While these formats provided structured insights into LLM performance, they do not fully capture the diversity of tasks commonly encountered in educational assessments.  
A further limitation of this investigation is the restricted sample size (23 university students). Although the study generated 207 data points, it is important to acknowledge that these do not represent independent participants but rather repeated measures from the same 23 individuals across multiple experimental items.  
Therefore, future research should expand the application of generative LLMs to include a broader range of subjects and educational levels, particularly those involving interpretative or subjective elements, and explore a broader range of complex question types to gain deeper insights into how LLMs handle intricate reasoning and extended text-based tasks.  
Conceptualization, P.C.M., F.Q. and F.M.; methodology, P.C.M., F.Q. and F.M.; software, P.C.M.; validation, P.C.M., F.Q. and F.M.; formal analysis, P.C.M., F.Q. and F.M.; investigation, P.C.M., F.Q. and F.M.; resources, P.C.M.; data curation, P.C.M.; writing--original draft preparation, P.C.M.; writing--review and editing, F.Q. and F.M.; visualization, P.C.M.; supervision, F.Q. and F.M.; project administration, F.Q.; funding acquisition, F.Q. All authors have read and agreed to the published version of the manuscript.  
This work was also supported by Interactive Technologies Institute (ITI/Larsys)--Funded by Fundacao para a Ciencia e a Tecnologia (FCT) projects: 10.54499/LA/P/0083/2020; 10.54499/UIDP/50009/2020 & 10.54499/UIDB/50009/2020.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Criterion** & **Sub-Criterion** & **Description** & **Points** \\ \hline \multirow{4}{*}{Identification of Main Structure} & \multicolumn{2}{c}{Mentioning lists as main structure for contact} & \multirow{2}{*}{5} \\  & information & & \\ \cline{1-1} \cline{3-3}  & & information & \\ \cline{1-1} \cline{3-3}  & & Mentioning inappropriate structure (dictionaries, tuples, sets) & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Scoring rubric for Question ID 12, showing criteria and score ranges.