# JorGPT: Instructor-Aided Grading of Programming Assignments with Large Language Models (LLMs)
## 5 Discussion  
The results of this study demonstrate that, under controlled and well-designed conditions, large language models (LLMs) can approximate human grading of beginner programming tasks with a high degree of consistency. The strong correlations observed between LLM-generated and instructor-assigned grades (often above 0.9 for models such as DeepSeek, GPT 4o and o3-mini) suggest that, when robust rubrics and clear cues are provided, automated systems can replicate many aspects of the human assessment process. However, some divergences remain. For example, models such as Gemini 2.0 Flash showed a tendency to assign higher or more lenient ratings compared to instructors. This leniency could stem from several factors, including an inherent 'helpfulness' bias in its fine-tuning, designed to avoid overly negative feedback in conversational contexts, or a different internal mapping of the rubric's qualitative descriptors to the 10-point scale. This divergence not only highlights the 'black box' nature of these models but also reinforces the critical need for human-in-the-loop systems and calibration methods, such as the regression model we propose, before these tools can be reliably used for summative assessment. These differences highlight the importance of continuous human supervision when integrating AI-based grading in academic contexts. From a practical perspective, the implications for educational practice are significant. Automating the assessment process with LLMs can dramatically reduce the workload associated with marking large groups of students, freeing instructors to focus on higher-order teaching tasks, such as personalized support or curriculum development. In addition, the ability of some models to process and evaluate hundreds of submissions in a matter of minutes offers the possibility of near real-time feedback, a feature that can transform student learning. Rapid and practical feedback has been shown to increase student motivation and support more effective formative learning. Importantly, the workflow implemented in this study maintains instructor oversight, allowing for final review and adjustment of grades and feedback before they are delivered to students. This hybrid approach ensures that academic standards are maintained, while taking full advantage of the benefits of automation. To contextualize predictive accuracy, it is important to assess the pedagogical implications of prediction errors on a 0-to-10 grading scale. In educational settings, deviations smaller than 0.5 points are typically considered negligible, as most grading systems incorporate rounding practices or qualitative interpretations that operate within 0.5-point intervals. This assumption is consistent with established educational measurement practices, where minor deviations are considered acceptable if they do not alter the substantive interpretation of student achievement. \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **Time** \\ \hline DeepSeek & 1 h 53\({}^{\prime}\)07\({}^{\prime\prime}\) \\ GPT 4o & 47\({}^{\prime}\)04\({}^{\prime\prime}\) \\ GPT 4.1 & 22\({}^{\prime}\)09\({}^{\prime\prime}\) \\ GPT o3-mini & 2 h 00\({}^{\prime}\)32\({}^{\prime\prime}\) \\ Gemini 2.0 Flash & 10\({}^{\prime}\)47\({}^{\prime\prime}\) \\ Qwen-PLUS & 24\({}^{\prime}\)12\({}^{\prime\prime}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Total execution time for grading 672 student submissions using each LLM. Qualitative analysis corroborates the pedagogical value of LLM-based grading when the assessment process is based on a carefully crafted rubric. The comments generated by the models, while necessarily concise due to their design, were often in line with best practice in formative assessment. In general, the comments were specific, made direct reference to aspects of the code in relation to the rubric categories, and offered concrete suggestions for improvement. However, some limitations remain.