# The Impact of Large Language Models on Programming Education and Student Learning Outcomes
## 1 Introduction  
In recent years, the integration of Large Language Models (LLMs) into various domains has revolutionized the landscape of artificial intelligence (AI) and machine learning (ML). Particularly in the field of natural language processing, modern, transformed-based LLMs such as OpenAI's ChatGPT and Microsoft's Copilot have demonstrated remarkable capabilities in understanding and generating human-like text. Beyond natural language understanding, LLMs have also found applications in programming education, offering students access to vast repositories of code snippets, explanations, and debugging assistance.  
While the potential benefits of leveraging LLMs in programming education are evident, it is crucial to understand the nuanced impact of their usage on student learning outcomes. Previous studies [1; 2] have begun to explore this relationship, highlighting both the advantages and potential pitfalls of integrating LLMs into educational settings. However, it is essential to note that many of these studies were conducted before the widespread availability of modern transformer-based attention mechanisms [3]. LLM services, including ChatGPT, Claude, Mistral, and Gemini, are open to the non-expert public. Consequently, there is a need for updated research that considers the specific functionalities and implications of these advanced language models.  
This paper seeks to contribute to the expanding body of knowledge on LLMs in education by focusing on the impact of informal LLM usage on undergraduate students' learning outcomes in programming education. It aims to dissect the relationship between LLM usage patterns and student performance, providing fresh insights into the role of LLMs in programming education and guiding pedagogical strategies in this area. To this end, we introduce the following research questions and hypotheses that frame our research:  
**RQ1.**: _What is the overall impact of Large Language Model (LLM) usage on the final grades of undergraduate students in programming courses?_  
**H1.**: _A higher average usage of LLMs for studying is negatively correlated with the final grades of undergraduate programming students._  
**RQ2.**: _How does the use of LLMs for generating code, seeking additional explanations, and debugging specifically impact the final grades of undergraduate students in programming courses?_  
**H2a.**: _The use of LLMs for generating code is negatively correlated with student final grades._  
**H2b.**: _The use of LLMs for seeking additional explanations does not significantly impact student final grades._  
**H2c.**: _The use of LLMs for debugging is negatively correlated with student final grades._  
These research questions and hypotheses steered our exploration into how ChatGPT and similar LLMs impact programming education. Our goal is to provide detailed insights to help shape teaching strategies that effectively integrate LLMs, ensuring they support and promote student learning and skill development in programming.  
The demand for empirical research assessing the impact of these advanced LLMs on programming education is essential. Previous papers, reviewed in the next chapter, largely conducted before these high-capability models were widely accessible, offer limited insights into the effects of LLMs integrated with the latest AI advancements. Thus, the main contribution of this study lies in its empirical analysis of how different usage patterns of modern LLMs--ranging from code generation to seeking explanations and debugging--impact learning programming. This examination not only expands our understanding of the educational implications of LLMs but also provides targeted insights that can guide educators in designing instructional strategies that leverage these tools effectively while nurturing essential problem-solving skills. By backing-up our findings with robust statistical analysis and a controlled experimental setup, this research offers important findings in the integration of AI technologies in programming education.