# Evaluating LLMs for Automated Scoring in Formative Assessments
## 4 Discussion  
The designed study contributed to a bimodal distribution of scores, with peaks at 0 and 100, in contrast to studies using public datasets with normal distributions. Despite these differences, the results demonstrate the adaptability of LLMs in effectively handling heterogeneous question types. Addressing RQ1, the examined premium LLMs closely aligned with human evaluators in a programming subject, achieving practical equivalence within defined thresholds (\(-5\) to \(5\)). A strong Spearman correlation coefficient of 0.963 further demonstrated the premium LLM's reliability in replicating human scoring patterns. In contrast, the examined open-source LLM deviated beyond equivalence thresholds, highlighting the need for refinement to achieve similar reliability. These findings establish premium LLMs as reliable tools for educational assessments, particularly in contexts demanding high scoring consistency and objectivity. While these findings align with prior research by [PERSON] et al. [5,26], [PERSON] et al. [18], and [PERSON] and [PERSON] [25], which demonstrated the reliability of premium LLMs across diverse educational tasks, this study provides additional insights into their application. Specifically, the comparative analysis between open-source and premium LLMs in evaluating programming-related, short-text responses represents a novel contribution not extensively explored in prior studies. Unlike research focused on essay scoring or free-form answers, this work highlights the challenges and opportunities in grading structured, code-based assessments. Furthermore, the findings of [PERSON] et al. [27], who identified difficulties in applying LLMs to broader assessment tasks like essay scoring, suggest that the reliability observed here may not extend to disciplines requiring subjective or interpretive evaluation, such as humanities or arts. Additionally, the observed differences in scoring patterns, such as the premium LLM's higher median scores and greater consistency, underscore its suitability for objective, technical evaluations while revealing gaps in open-source models' performance. Distinct scoring trends and consistency were observed across question types. For code-based questions, the premium model closely matched human evaluators, demonstrating its potential for scoring programming assignments with precision. In contrast, the open-source model scored substantially lower in this study, reflecting a need for further optimization in this area. These results highlight the examined premium model suitability for structured, objective assessments and align with prior research on the reliability of generative pre-trained transformer models for educational tasks [5,16]. For short-answer questions, the premium LLM showed leniency in mid-range scoring, while human and open-source evaluations were closely aligned. Variability at score extremes underscored challenges in interpreting subjective responses, contrasting with the consistency observed in code-based evaluations. Nevertheless, these conclusions might not be generalized to disciplines requiring subjective or interpretative grading, such as humanities or arts, where nuanced evaluation poses unique challenges. Further studies are necessary to confirm the findings in such contexts. Another notable finding pertains to scoring consistency. The premium LLM demonstrated the highest consistency across both short-answer and code-based questions, with minimal variation between question types. Human graders showed greater variability, likely reflecting their ability to capture nuanced responses that automated systems may overlook. The premium LLM also assigned higher median grades (68.18) compared to human evaluators (54.17). This aligns with [PERSON] and [PERSON]'s research [28] highlighting that premium LLMs tend to assign higher grades in scoring contexts. However, [PERSON] [29]observed lower grades from GPT-4 in specific domains, emphasizing the importance of contextual factors when interpreting LLM performance.