## 4 Assessment Validation
### Quantitative Evaluation  
#### 4.2.1 Methods  
In addition to the qualitative evaluation, the items were administered on a large scale to evaluate students' item responses quantitatively. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline
**Item** & **Att. A** & **Att. B** & **Att. C** & **Att. D** & **Att. E** & **Att. F** & **Att. G** & **Att. H** & **Att. I** & **z** \\ \hline ARTIST\_sc\_MS\_05 & 67\% & - & - & - & - & - & - & - & - & - & - & 3 \\ ARTIST\_sc\_MS\_01 & 100\% & - & - & - & - & - & - & - & - & - & - & 4 \\ ARTIST\_sc\_MC\_05 & 100\% & - & - & - & - & - & - & - & - & - & - & 4 \\ ARTIST\_sc\_MC\_06 & 100\% & - & - & - & - & - & - & - & - & - & 4 \\ ARTIST\_st\_db\_MS\_Q090 & 100\% & - & - & - & - & - & - & - & - & - & 4 \\ CAOS\_14 & 100\% & 100\% & - & - & - & - & - & - & - & - & 4 \\ CAOS\_15 & 100\% & 100\% & - & - & - & - & - & - & - & - & 4 \\ CAOS\_08 & 67\% & 100\% & - & - & - & - & - & - & - & - & 3 \\ CAOS\_09 & 67\% & 100\% & - & - & - & - & - & - & - & - & 3 \\ CAOS\_10 & 100\% & 100\% & - & - & - & - & - & - & - & - & 3 \\ CAOS\_11 & - & 67\% & 100\% & - & - & - & - & - & - & - & 3 \\ CAOS\_12 & - & 100\% & 100\% & - & - & - & - & - & - & - & 3 \\ CAOS\_13 & - & - & - & - & - & - & - & - & - & 3 \\ SRA\_015 & - & 75\% & 100\% & - & - & - & - & 25\% & - & 4 \\ ARTIST\_db\_CC\_Q0840 & 50\% & 100\% & 100\% & - & - & - & - & - & - & 4 \\ ARTIST\_sc\_SV\_01 & - & 67\% & - & 33\% & 33\% & - & - & - & - & 3 \\ CAOS\_17 & - & - & - & 100\% & - & - & - & - & - & - & 4 \\ ARTIST\_sc\_SV\_03 & - & - & - & 100\% & - & 100\% & - & - & - & 4 \\ ARTIST\_sc\_SV\_14 & 67\% & - & - & 100\% & - & 33\% & - & - & - & 3 \\ ARTIST\_sc\_SV\_04 & - & - & - & 60\% & 100\% & - & - & - & - & 5 \\ ARTIST\_sc\_SV\_10 & - & - & - & - & 100\% & - & - & - & - & 4 \\ ARTIST\_sc\_SV\_11 & - & - & - & 25\% & 100\% & 50\% & - & - & - & 4 \\ ARTIST\_sc\_SV\_09 & - & 50\% & - & 50\% & 100\% & 50\% & - & - & - & 4 \\ ARTIST\_st\_BS\_SO061A & - & - & - & 50\% & 100\% & - & - & - & - & 4 \\ ARTIST\_db\_SS\_Q0061B & - & - & - & - & 100\% & 25\% & - & - & - & 4 \\ ARTIST\_db\_SS\_Q0061B & - & - & - & - & 100\% & 25\% & - & - & - & 4 \\ ARTIST\_db\_SS\_Q0061C & - & - & - & - & 100\% & 25\% & - & - & - & 4 \\ ARTIST\_db\_SS\_Q0061B & - & - & - & - & 100\% & 25\% & - & - & -Participants and Procedures  
The items were translated into Dutch and implemented in introductory statistics courses taught during the fall of 2021 in five different bachelor programs at Utrecht University--namely, psychology, educational sciences, pedagogical sciences, sociology, and cultural anthropology. The item set was split into two assessments to reduce the burden on students. The first assessment consisted of 35 items and concerned all topics related to _Samples and spread_, encompassing attributes A-F. The second assessment consisted of 24 items and concerned all topics related to _NHST and confidence intervals (CIs)_, encompassing attributes G-I. The two item sets exclusively measure the attributes that are assessed within each assessment according to the provisional Q-matrix and both Q-matrices meet the conditions for generic identifiability from [40]. The assessments were made available after the relevant topics were addressed in class, but well before the final exam. They were presented as optional, formative assessments. Students were given only one attempt for each assessment to motivate them to make a serious effort. In total, 849 students completed the first assessment and 790 students completed the second assessment. Analysis  
The data were analyzed using the statistical software R [41]. We focused on diagnostic classification models (DCMs) under the log-linear cognitive diagnosis modeling (LCDM) framework because of its modeling flexibility and straightforward interpretation [12]. To validate attribute measurement with DCMs, we first evaluated the assumption of local independence using the local dependence statistic \(\chi^{2}_{L,D}\) for each item pair [42].