# Evaluating LLMs for Automated Scoring in Formative Assessments
This approach allows LLMs to retrieve up-to-date and contextually relevant information, substantially improving the accuracy and reliability of their responses [12; 13]. RAG systems use semantic similarity to retrieve relevant data from external databases, such as vector databases like ChromaDB, ensuring the generated outputs align closely with the query context [12; 14]. The implementation of RAG in this study was supported by LangChain, which provided the tools to efficiently manage the retrieval process and integrate relevant information into the LLM's outputs [15]. However, the success of this system depends on the quality and timeliness of the external data, requiring careful management to maximize its effectiveness [12]. While RAG mitigates data limitations, effective prompt design is necessary for maximizing LLM performance. Prompts directly influence response quality, with few-shotlearning emerging as a suitable technique to enable models to adapt to new tasks using minimal data [8, 16]. By incorporating examples like questions, rubrics, and reference answers, few-shot learning can improve accuracy in automated scoring while reducing the need for frequent retraining [6, 17]. However, overfitting--in this context, the model becomes overly reliant on specific examples, limiting generalizability--remains a risk. This challenge, and strategies for its mitigation, including maintaining balanced datasets and using sufficiently large validation sets, have been addressed in recent works [18]. Achieving optimal performance depends on selecting the right LLM, with premium and open-source models offering distinct trade-offs [19]. Premium LLMs typically deliver higher performance but have considerable licensing costs and potential data privacy risks due to remote data processing [11, 19]. On the other hand, open-source models provide greater flexibility and control, often running locally to address privacy concerns. However, they require substantial computational resources, making deployment at scale technically demanding. The choice ultimately involves balancing performance, cost, privacy, and infrastructure requirements [19]. While advancements have been made in employing LLMs for educational purposes, substantial gaps in research remain, particularly in the context of automated grading. Limited studies have explored the feasibility of using LLMs to accurately grade diverse types of student assessments, presenting a critical opportunity for further investigation into their reliability and scalability [7]. Furthermore, while some research highlights the need for comparisons between open-source and premium LLMs, studies directly addressing their grading performance remain scarce, leaving significant gaps in understanding the trade-offs between these models. Addressing these challenges and building on the opportunities highlighted by prior research, this study aims to evaluate the ability of LLMs, both open-source and premium, to perform scoring tasks with reliability and accuracy comparable to human evaluators in the context of a computer programming subject, defining two Research Questions (RQ):  
* RQ1: Are LLM-generated grades as reliable as human evaluations? H\({}_{0}\): There is no significant difference between the two. * RQ2: Can open-source LLMs achieve scoring accuracy comparable to premium models? H\({}_{0}\): There is no significant scoring difference between the models. The findings demonstrate that while both LLMs align closely with human evaluators, the premium LLM achieves statistically significant and practically equivalent scoring performance, particularly for code-based assessments. The premium LLM also demonstrated the highest consistency across all question types, outperforming human evaluators and the open-source model. By addressing these hypotheses, this study evaluates the potential of LLMs to effectively assist human evaluators in grading tasks.