SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct experiments evaluating other missing ratios of the Q-matrix on the NIPS2020EC dataset. In Table 2 of the main body, we illustrated the results under  $100\%$ ,  $20\%$  and  $10\%$  settings. Here we additionally add experiments on  $50\%$ ,  $30\%$ , and  $5\%$  Q-matrix scenarios, which are illustrated in 5. The experimental results demonstrate our DCD outperforms all the interpretable baseline models in these few-labeled scenarios. Besides, as the preserved ratio of the Q-matrix decreases (from  $50\%$  to  $5\%$ ), our DCD demonstrates a greater improvement (from  $0.86\%$  to  $1.43\%$ ) compared to the second-ranked model in terms of DOA. This shows the advantage of DCD in label-scarce scenarios.
Table 5: Comparison in other few-labeled scenarios on NIPS2020EC dataset. We divide models into non-interpretable and interpretable models. The best scores are in bold for two kinds of models. Noted that non-interpretable models keep same results in any scenario as they do not utilize Q-matrix.
Model	50% Q-matrix (few-labeled)	30% Q-matrix (few-labeled)	5% Q-matrix (few-labeled)
AUC↑	ACC↑	RMSE↓	DOA↑	AUC↑	ACC↑	RMSE↓	DOA↑	AUC↑	ACC↑	RMSE↓	DOA↑
NIPS2020EC
PMF	0.814	0.760	0.407	-	0.814	0.760	0.407	-	0.814	0.760	0.407	-
IRT	0.819	0.762	0.402	-	0.819	0.762	0.402	-	0.819	0.762	0.402	-
MIRT	0.822	0.765	0.400	-	0.822	0.765	0.400	-	0.822	0.765	0.400	-
DINA	0.766	0.692	0.456	0.807	0.766	0.685	0.455	0.769	0.765	0.675	0.454	0.649
NCDM	0.799	0.739	0.415	0.802	0.801	0.743	0.414	0.765	0.800	0.742	0.414	0.644
β-TCVAE	0.794	0.746	0.416	0.808	0.795	0.748	0.412	0.790	0.801	0.747	0.415	0.770
KSCD	0.800	0.744	0.417	0.800	0.803	0.750	0.413	0.791	0.805	0.752	0.411	0.769
KaNCD	0.807	0.754	0.411	0.810	0.809	0.759	0.406	0.792	0.812	0.761	0.404	0.752
DCD	0.809	0.759	0.407	0.817	0.811	0.760	0.405	0.800	0.814	0.763	0.403	0.781
A.3 Comparison between L1 and L2 loss for unlabeled exercises in alignment module
In section 4.3, we finally adopt the L2 loss to constraint the exercise relevance representation sparsity of unlabeled exercises. Intuitively, it's reasonable that adopt an L1 loss to constraint the value in the last part (i.e.,  $\pmb{\mu}_{v_j}^r$  [min#  $(K - d_1 - d_2)]$ ) to be 0. As shown in 6, in our initial attempt, we first directly try to use L1 loss for numerous unlabeled exercises, and we find it does not show competing performance. We speculate a possible reason is as follows: with L1 loss, after model initialization, it is very hard to revise these incorrectly labeled exercises. In contrast, L2 loss are sensitive to outliers, and would have a larger loss when the corresponding knowledge is incorrectly inferred.
Table 6: Comparison between L1 and L2 loss of alignment in  ${10}\%$  Q-matrix scenario on three datasets.
Model	Matmat	JunYi	NIPS2020EC
AUC↑	ACC↑	RMSE↓	DOA↑	AUC↑	ACC↑	RMSE↓	DOA↑	AUC↑	ACC↑	RMSE↓	DOA↑
DCD with L1	0.747	0.851	0.340	0.777	0.811	0.770	0.398	0.634	0.810	0.758	0.406	0.722
DCD with L2	0.764	0.855	0.334	0.796	0.814	0.771	0.397	0.717	0.813	0.762	0.404	0.786
A.4 Monotonicity Analysis
The monotonicity assumption [38, 39] in CD works means that the probability of correct response to the exercise is monotonically increasing at any dimension of the student's knowledge proficiency, which guarantees the interpretability of student proficiency representation (i.e., the higher the value of the latent factor means the better proficiency on the corresponding knowledge concept). Given a student proficiency representation, each dimension in the representation corresponds to the student's proficiency level on a specific knowledge concept. When we only increase the value of a single dimension in the student representation (improving the student's proficiency on a specific knowledge concept), while keeping the exercise diffi
culty representation and relevance representation unchanged, the student's probability of answering questions related to that knowledge concept correctly will increase, while the probability of answering questions unrelated to that knowledge concept will remain unchanged. In Fig. 6, we present a case where we only increase a single dimension in the student representation, and it shows that the student's probability of answering questions related to that knowledge concept correctly increases.
Figure 4: Example of variation of one factor in student proficiency representation. We randomly select a student and three knowledge concepts.
A.5 Nemenyi Test
The Nemenyi test [9] is conducted to present the comparison between our proposed DCD and the interpretable baselines for all 5-fold cross-validation results. A significant difference is regarded to exist in the Nemenyi tests if the average ranks of two models differ by at least one crucial difference, which is determined using a  $5\%$  significance level. The model metric improves as the ranking score decreases. As illustrated in Fig 5, Fig 6 and Fig 7, which demonstrates the comparison on  $100\%$ ,  $20\%$  and  $10\%$  labeled exercises scenario, respectively. It's obvious that our proposed DCD approach performs best for all metrics in any scenario.
Figure 5: Nemenyi tests for comparison on  $100\%$  labeled exercises scenario
Figure 6: Nemenyi tests for comparison on  $20\%$  labeled exercises scenario
Figure 7: Nemenyi tests for comparison on  $10\%$  labeled exercises scenario
B.1 Reparameterization Trick
In the student encoder module, student proficiency representation  $\mathbf{z}_u$  is sampled from a Gaussian distribution  $\mathcal{N}(\boldsymbol{\mu}_u,\boldsymbol{\sigma}_u)$ . If we sample  $\mathbf{z}_u$  from  $\mathcal{N}(\boldsymbol{\mu}_u,\boldsymbol{\sigma}_u)$  directly, we will lost gradient information as the sampling process is not differentiable. Fortunately, any Gaussian distribution can transform to a standard Gaussian distribution. We can sample a  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$  that has nothing to do with any parameters. By doing so,  $\mathbf{z}_u$  is equal to  $\boldsymbol{\mu}_u + \boldsymbol{\sigma}_u\odot \boldsymbol{\epsilon}$ , where  $\odot$  means element-wise product. However, there is no similar property of Bernoulli distribution corresponding to exercise relevance encoder. In this work, we adopt gumbel-softmax reparameterization [15] to alleviate this problem, which is one of the gradient estimator methods.
B.2 Algorithm for Filling Missing Q-matrix
Existing interpretable baselines cannot directly apply to few-labeled scenarios. We design a filling method by employing similarity of response records in exercise to infer missing knowledge concepts. The detailed procedure is shown in Algorithm 1. We first compute the similarity (detailed in Eq. (10)) among exercises according to response records of exercise. Then for each unlabeled exercise, we vote the  $\text{vote}_k$  most frequently occurring knowledge concepts among the most similar  $\text{exer}_k$  labeled exercises as the filling knowledge concepts. We set  $\text{vote}_k = 2$  and  $\text{exer}_k = 5$  as default.
$$
\mathbf {S} _ {i, j} = \frac {\sum_ {u _ {n} \in \mathcal {U}} I \left(\mathbf {X} _ {n i} \cdot \mathbf {X} _ {n j} = = 1\right)}{\sum_ {u _ {n} \in \mathcal {U}} I \left(\mathbf {X} _ {n j} \neq 0\right)}, \tag {10}
$$
where  $\mathbf{S}_{i,j}$  denotes the similarity between exercise  $v_{i}$  and  $v_{j}$ .  $I(\text{condition}) = 1$  if condition is True, and vice versa  $I(\text{condition}) = 0$ .  $\mathbf{X}$  is the interaction matrix in training set.  $\mathbf{X}_{ni} = 1$ ,  $-1$  represents student  $u_{n}$  answered exercise  $v_{i}$  correctly and incorrectly, respectively.  $\mathbf{X}_{ni} = 0$  represents student  $u_{n}$  did not answer exercise  $v_{i}$ .  $\mathcal{U}$  is the student set.
Algorithm 1 Filling Missing Q-matrix for Interpretable Baselines
Input: labeled exercises set  $\mathcal{V}_1$ , unlabeled exercises set  $\mathcal{V}_2$ , interaction matrix  $\mathbf{X}$  in training set, missing Q-matrix  $\mathbf{Q}$ , number of similar exercises  $exer_k$ , number of filling knowledge concepts  $vote_k$
Output: Filling Q-matrix Q
1:  $M = |\mathcal{V}_1\bigcup \mathcal{V}_2|$
2: Initialize exercise similarity matrix  $\mathbf{S}$  of size  $(M,M)$  with the default value  $-\infty$
3: for all exercise  $v_{i} \in \mathcal{V}_{1}$  do
4: for all exercise  $v_{j} \in \mathcal{V}_{2}$  do
5: Calculate  $\mathbf{S}_{i,j}$  according to Eq. (10)
6: end for
7: end for
8: for all exercise  $v_{j} \in \mathcal{V}_{2}$  do
9:  $s_e =$  the most similar exer  $k$  labeled exercises of  $v_{j}$  according to  $\mathbf{S}[:,j]$
10:  $s_k =$  the vote  $_k$  knowledge concepts with the highest frequency of occurrence in  $s_e$
11: Set corresponding elements to be one in  $\mathbf{Q}[j,:]$  by  $s_k$
12: end for
13: return Q
B.3 Datasets and Preprocessing
Our experiments are conducted on three real-world datasets, i.e., Matmat $^{4}$ , Junyi $^{5}$  [4] and NIPS2020EC $^{6}$  [42], all of which contain knowledge concepts of the tree structure. For all datasets, we preserve the first-time exercise-answering record for same student-exercise pairs to support cognitive diagnosis. The detailed information on datasets and preprocessing method are depicted as following:
- Matmat. This dataset is collected from an intelligent web application for practicing mathematics called Matmat<sup>7</sup>, which includes three-level knowledge concept structure after preprocessing. The first-level knowledge concept is the root node called math. The second-level knowledge concepts are organized as 5 calculational types including addition, multiplication, division and so on. The third-level knowledge concepts are organized by calculational scale. We preserve students with more than 15 response records to guarantee that each student has enough data for diagnosis.
- Junyi. This dataset is collected from an online educational platform called Junyi Academy. There are three-level knowledge concept structure in this dataset. Besides the first-level root node, we apply area and topic field in origin dataset to denote the second-level and third-level knowledge concept, respectively. We preserve students with more than 50 response records to guarantee that each student has enough data for diagnosis.
- NIPS2020EC. This dataset is originated from NeurIPS 2020 Education Challenge [42], which provides students' answers to mathematics questions from Eedi<sup>9</sup>. There are four-level knowledge concept structure in this dataset. We sample a subset from the task 1 of this competition by selecting response records during March 2020. We preserve students with more than 30 response records to guarantee that each student has enough data for diagnosis.
B.4 Implementation Details
We train our model with Python 3.9 and PyTorch 1.12.1 on NVIDIA RTX A5000. The student and exercise encoders are implemented by a multilayer perceptron for all datasets. We set a prior Gaussian distribution with  $\mathcal{N}(0,1)$  for each latent factor in  $\pmb{\mu}_u$  and  $\pmb{\mu}_v^d$ , and a prior Bernoulli distribution with Bernoulli(0.2) for each latent factor in  $\pmb{\mu}_v^r$ . For all interpretable models, we select the epoch with the best DOA for testing and almost set Adam as the default optimizer. The implement of DOA [38] metric is adopted from EduCDM<sup>10</sup>. For more implementation details, please see our public code repository https://github.com/kervias/DCD.
C Knowledge Concept Tree
C.1 Standard Knowledge Concept Tree
The employment of knowledge concept tree in our work is to obtain group based disentanglement according to granularity at different levels. For convenience, we transform the initial knowledge concept tree into a standard tree, which is defined as the tree that the height of all subtrees of any node in the tree is equal in our work. The transformation process is displayed in Fig. 8. For those leaf nodes whose depth has not reached the depth of the tree, we can continuously duplicate this node through inheritance until reaching the depth of the tree.
C.2 Example of Real Knowledge Concept Tree
In this work, we assume that there is less independence among groups of knowledge concepts grouped by their parent nodes in the knowledge concept tree. This viewpoint is inspired by practical applications, and the analysis is as follows. To better clarify the argument, we provide the partial
4https://github.com/adaptive-learning/matmat-web   
5https://ps1cdatashop.web.cmu.edu/DatasetInfo?datasetId  $\equiv$  1198   
6https://eedi.com/projects/neurips-education-challenge   
7https://matmat.cz/   
8https://www.junyiacademy.org/   
9https://eedi.com/   
10https://github.com/bigdata-ustc/EduCDM/
Figure 8: Process of transforming non-standard knowledge concept tree structure into standard tree.
knowledge concept tree of NIPS2020EC dataset in Fig. 9. Firstly, the course chapter hierarchy is a typical example of concepts with a tree structure. Intuitively, the associations among concepts at the high level of the hierarchy are weaker compared to the associations between concepts at the low level. For example, if we adopt the 2nd level concepts (i.e., algebra, data and statistics, ...) to group the last level concepts, the independence among groups is higher than the groups determined by the 3rd level concepts (i.e., inequalities, formula, data collection, ...). Moreover, there is a higher likelihood that concepts within the same chapter are simultaneously assessed in the same exercise. For instance, the concepts whose parent concept is inequalities would have a higher probability occur in the same exercise. However, the solving linear inequalities concept whose parent concept is inequalities and the tally charts concept whose parent concept is data collection would have a lower probability occur in the same exercise.
Figure 9: An example of a four-level knowledge concept tree on NIPS2020EC dataset. Here we only provide details of the hierarchical structure of NIPS2020EC  $\rightarrow$  Algebra  $\rightarrow$  Inequalities and NIPS2020EC  $\rightarrow$  Algebra  $\rightarrow$  Data and Statistics  $\rightarrow$  Data Collection.
There are still some limitations of our proposed method. 1) The latent factors considered in our method are only related to knowledge concepts. However, the student answering process may refer to more factors that are not related to knowledge concepts such as the family economic conditions of students and reading comprehension difficulty of exercise textual information, which results in the incomplete and insufficient disentanglement in our proposed method. The following work we consider is to introduce more side information to help disentangle other factors. 2) The assumption that inter-group knowledge concepts are independent may be too strong. The later work we consider is to model causal relation among knowledge concepts for disentanglement. 3) The alignment module for unlabeled exercises is just to keep sparsity. In the future, we will introduce explicit relation among knowledge concepts to help infer missing knowledge concepts.