## 3 Experimental Design
### Instruments  
For the initial and final knowledge tests, exercises similar to those in the guide were used, following the format of code snippets with four non-exclusive alternatives. The pre-test and post-test included six exercises (two associated with each Learning Objective, LO), randomly selected from a pool of 18 exercises (distinct from the guide), with scores ranging from 0.0 to 10.0. These exercises were developed from materials used in prior semesters of the software systems course and aligned with the PHP module's learning objectives. Content validity was ensured through review by two instructors with over five years of PHP teaching experience. The pre- and post-test exercises were extracted from a broader item bank applied in the software systems course for over four academic years (seven semesters), primarily as practice and assessment tools aligned with the course's PHP module learning objectives. These items were developed by instructors with extensive experience in programming instruction and have undergone iterative refinement based on student performance and instructional feedback. While detailed item analysis was not conducted, the sustained application of these items across cohorts supports their empirical reliability and content validity. Post-hoc analysis of the post-test scores showed a Cronbach's alpha of 0.82, indicating good reliability. Figure 6 shows an example of a question used in the pre-test.  
It is important to note that both experimental conditions--Microsoft Copilot and instructional video--worked with the same practice guide and were assessed using pre- and post-tests drawn from a shared item bank. This uniformity ensures that familiarity with item formats or content is applied equally across both groups, minimizing the risk of biased learning gains due to prior exposure.  
On the other hand, for the initial and final perception tests, the HMSAM model was used with its eight dimensions: usefulness, enjoyment, ease of use, intention to use, control, focused immersion, temporal dissociation, and curiosity [16]. The total number of statements for both tests was 24 items, measured on a Likert scale from 1 to 7, where 1 corresponded to \"Strongly Disagree\", 4 to \"Neutral\", and 7 to \"Strongly Agree\". The list of statements by construct is shown in Table 1.  
Figure 5: Explanatory video of the exercises is in the guide.  
Figure 6: Sample question in the knowledge pre-test.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Construct** & \multicolumn{2}{c}{**Question**} \\ \hline  & \(\bullet\) & I will have little control over what I can do (Rev). \\  & \(\bullet\) & I expect to have control while performing the activities. \\ Control (Ctrl) & \(\bullet\) & I expect to be able to freely choose what I want to see or do while performing the activities. \\ \hline Curiosity (Cur) & \(\bullet\) & This experience will stimulate my curiosity. \\  & \(\bullet\) & This experience will spark my imagination. \\  & \(\bullet\) & This experience will make me curious. \\ \hline Temporal Dissociation (TD) & \(\bullet\) & Time will seem to pass very quickly while doing the activity. \\  & \(\bullet\) & I will lose track of time while doing the activities. \\  & \(\bullet\) & Time will “fly” when I do the exercises. \\ \hline  & \(\bullet\) & I believe navigating, writing questions, and \\  & \(\bullet\) & I find that the activity with the ICT resource will be easy to use. \\ Ease of use & \(\bullet\) & I believe that interacting with the ICT resource during the activity will be clear and understandable. \\ \hline \hline \end{tabular}
\end{table}
Table 1: List with questions (statements) by construct of the HMSAM model (for pre-test).