## 4 Results
### Results and Interpretations
By using two exams of differing difficulty, we aim to control for baseline knowledge and isolate the effect of ChatGPT usage. The difference in grades was calculated as the score from the exam without ChatGPT subtracted from the score on the exam with ChatGPT. This method allows us to focus on how ChatGPT may bridge the performance gap when tackling more complex coding problems, regardless of the developer's inherent knowledge or skills. In this analysis, we examined how different types of queries (tags) related to React.js impact developers' performance on a controlled exam with and without the help of ChatGPT. The Mann-Whitney U test was used to compare the differences in performance between cases where no tag was present and cases where a tag was specified. By examining Tables 1 and 2, we can note that among the tags analyzed, significant differences were found for EF and CU. For EF, developers showed a larger grade difference without ChatGPT compared to with ChatGPT (mean = 3.625, SD = 5.044, N = 6) compared to when a tag was present (mean = 0.287, SD = 3.508, N = 31), with a statistically significant result (U = 135.50, \(p\) = 0.042). A similar trend was seen for CU, where developers without ChatGPT performed substantially better (mean = 4.688, SD = 4.033, N = 4) than those who used ChatGPT with the CU tag (mean = 0.361, SD = 3.695, N = 33), with a significant difference (U = 105.00, \(p\) = 0.030). For other tags, such as FI, BP, DOC, CC, and CO, the differences in performance were not statistically significant (\(p\) > 0.05), indicating that ChatGPT's contribution may not be as impactful in these areas, or the sample sizes were insufficient to draw firm conclusions. These findings suggest that ChatGPT could be particularly helpful for tasks involving code understanding and error fixing, as developers who used ChatGPT in these contexts performed closer to their scores on a more difficult exam, indicating that the assistance potentially mitigated the difficulty gap. Organizations or individual developers could leverage this insight by promoting the use of AI assistance for error resolution and comprehension tasks while focusing on manual learning and problem-solving skills for feature implementation, concept clarification, and other areas where the AI's assistance shows less significant impact. This could lead to more efficient workflows and a tailored use of AI for specific coding tasks. This division of labour aligns with the prompt specificity framework described by [PERSON] et al. [22], which argues that the more constrained and clearly defined a problem is, the more reliable and relevant the model output becomes. In contrast, when prompts are vague, such as in open-ended feature development, the model may generate plausible but ungrounded suggestions, which could confuse rather than assist novice developers. Students who used tagged queries for EF were more likely to submit runnable code compared to those who used untagged queries. This suggests that tagging allows students to specify their problems more precisely, enabling the LLM to generate more targeted and helpful responses. This result highlights the critical role of query clarity in debugging tasks, particularly for novice programmers. From a theoretical perspective, this aligns with the principles of selective attention and anchoring explored by [PERSON] and [PERSON] (2019), who find that prompts that include specific goals and context act as anchors that constrain the model's focus throughout the generation process. This focused output generation increases task success rates, especially in goal-directed tasks like debugging. Similarly, tagged queries for CU were associated with better overall performance. This finding indicates that students who actively sought to deepen their comprehension of code concepts benefited significantly from LLM assistance. The educational implications of this result are notable, as it underscores the potential of LLMs to enhance conceptual understanding alongside practical coding skills.