# The Impact of Large Language Models on Programming Education and Student Learning Outcomes
## 4 Results  
### Data Overview  
To provide context for the results outlined in this section, we offer an overview of the code size associated with the tasks completed by participants during the experiment. The estimation of the approximate lines of code (LOC) for each task was derived from solutions prepared in alignment with the curriculum. The approximation of the LOC for each assignment is as follows: assignment 1 has 137 LOC, assignment 2 has 461 LOC,assignment 3 has 427 LOC, assignment 4 has 921 LOC, and the final assignment (controlled phase) has 915 LOC. These values illustrate the approximate code size participants were required to write for each assignment, highlighting the variation in complexity and scope across the tasks. Further on, an overview of the descriptive statistics of the measured variables used in the analysis is provided. Table 1 and the histograms in Figure 2 below offer a comprehensive overview of the extent to which the LLMs were utilized for various aspects of studying and the final grades given. Students reported a mean frequency of LLM usage for generating code at 2.59, with a median of 2.50. This is slightly above the lower midpoint of the scale, indicating that while some students did use LLMs to assist with generating code, it was not the most frequently utilized function. The standard deviation (SD) of 1.103 shows moderate variability in the use of LLMs for this purpose. The mean for seeking additional explanations is higher at 3.75, with a median of 4, suggesting that students were more inclined to use LLMs to gain further understanding of the material. The SD of 1.244 indicates a slightly higher variability \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & & **Mean** & **Median** & **SD** & **Min** & **Max** \\ \hline \multirow{3}{*}{LLM use} & Generating code & 2.59 & 2.50 & 1.10 & 1.00 & 5.00 \\  & Additional explanations & 3.75 & 4.00 & 1.24 & 1.00 & 5.00 \\  & Debugging & 3.78 & 4.00 & 1.16 & 1.00 & 5.00 \\  & Average & 3.38 & 3.67 & 0.94 & 1.00 & 5.00 \\ \hline \multicolumn{2}{c}{Final grade} & 6.72 & 8.00 & 3.10 & 0.50 & 10.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Descriptive statistics of measurement. Figure 2: Histograms of the measurements about LLM use and the final grade. in response, with some students relying heavily on LLMs for explanations while others did not. The usage for debugging has a similar mean of 3.78 and median of 4, which indicates a comparable pattern of reliance on LLMs for this activity as for additional explanations. The SD of 1.157 denotes moderate variability among the students' responses. Overall, the average use of LLM across all activities has a mean of 3.38 and a median of 3.67. This points to a generally moderate use of LLM across the board, with the SD of 0.94 suggesting that the extent of LLM usage did not greatly differ among students. The scatterplots provided in Figure 3 visualize the relationship between the final grades and their reported usage of the LLMs for different study activities. There appears to be a spread of points that tends downward as the frequency of using LLM for generating code increases. This could suggest that students who used LLMs more frequently for code generation tended to receive lower final grades, in line with [PERSON]'s rho value, which will be discussed in subsequent subchapters. For the use of LLMs for additional explanations, the distribution of points shows a less clear pattern, suggesting a weaker or potentially non-significant correlation between the use of LLM for additional explanations and the final grades, which corresponds with the non-significant \(p\)-value found in the [PERSON]'s correlation analysis. Next, for the use of LLMs for debugging purposes, the scatterplot shows a trend where the higher usage of LLMs for debugging corresponds to lower final grades, which aligns with the significant negative correlation identified in the statistical analysis. Looking at the overall use of LLMs, there is a visible trend of decreasing final grades with increased average LLM use.