Title: Disentangling Cognitive Diagnosis with Limited Exercise Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: Intelligent Education System

Cognitive Diagnosis

Disentangled Representation Learning

Interpretability
Abstract: Cognitive diagnosis is an important task in intelligence education, which aims at measuring students’ proficiency in specific knowledge concepts. Given a fully labeled exercise-concept matrix, most existing models focused on mining students' response records for cognitive diagnosis. Despite their success, due to the huge cost of labeling exercises, a more practical scenario is that limited exercises are labeled with concepts. Performing cognitive diagnosis with limited exercise labels is under-explored and remains pretty much open.
 In this paper, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges of limited exercise labels. Specifically, we utilize students' response records to model student proficiency, exercise difficulty and exercise label distribution.  
 Then, we introduce two novel modules - group-based disentanglement and limited-labeled alignment modules - to disentangle the factors relevant to concepts and align them with real limited labels. 
 Particularly, we introduce the tree-like structure of concepts with negligible cost for group-based disentangling, as concepts of different levels exhibit different independence relationships.
Extensive experiments on widely used benchmarks demonstrate the superiority of our proposed model.

Disentangling Cognitive Diagnosis with Limited Exercise Labels

Abstract

Cognitive diagnosis is an important task in intelligence education, which aims at measuring students' proficiency on specific knowledge concepts. Given a fully labeled exercise-concept matrix, most existing models focused on mining students' response records for cognitive diagnosis. Despite their success, due to the huge cost of labeling exercises, a more practical scenario is that limited exercises are labeled with concepts. Performing cognitive diagnosis with limited exercise labels is under-explored and remains pretty much open. In this paper, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges of limited exercise labels. Specifically, we utilize students' response records to model student proficiency, exercise difficulty and exercise label distribution. Then, we introduce two novel modules - group-based disentanglement and limited-labeled alignment modules - to disentangle the factors relevant to concepts and align them with real limited labels. Particularly, we introduce the tree-like structure of concepts with negligible cost for group-based disentangling, as concepts of different levels exhibit different independence relationships. Extensive experiments on widely used benchmarks demonstrate the superiority of our proposed model.

1 Introduction

In the field of intelligent education systems, cognitive diagnosis (CD) [25, 26] is a fundamental and essential task, which aims at measuring students' proficiency on specific knowledge concepts through the student performance prediction process. CD can serve various applications, such as computerized adaptive testing [2], targeted training [20], and exercise recommendation [24, 14]. We show a toy example of CD in Fig. 1-(a). The exercise-concept matrix (known as Q-matrix) is labeled to indicate which concepts are tested in each exercise. For example, concepts k 3 and k 5 are tested in exercise v 3 . The CD model diagnosed that student u 2 has a higher proficiency on the two concepts than that of u 1 because u 2 answered correctly on the exercise while u 1 answered incorrectly. It's obvious that fully labeled Q-matrix plays an essential role in interpretability (i.e., diagnostic report) for CD models.

Most existing CD models focus on enhancing the mining process of response records to achieve better diagnosis results. These models typically use a fully labeled Q-matrix that has been annotated by domain experts to train their models. For example, the classical DINA model [8] assumes that students must master all knowledge concepts associated with an exercise to answer it correctly,

Full-Labeled Q-matrix

Knowledge concepts

Exercises

v

1

v

2

v

3

v

4

k

1

k

2

k

3

k

4

k

5

Students

v

1

v

2

v

3

v

4

v

1

v

3

v

4

Response Records

Diagnostic Report

k

1

k

2

k

3

k

4

k

5

u

1

u

2

Cognitive

Diagnosis

u

1

u

2

k

1

k

2

k

3

k

4

k

5

Tree-like Structure of

Knowledge Concepts

1

st

Level

2

nd

Level

3

rd

Level

(a)

(b)

except in cases where the student guesses correctly. Representative neural-based CD models, such as NCDM[38], KaNCD [39] and KSCD [30], explicitly model the explicit relationship between exercise and knowledge concepts to constrain student proficiency during diagnostic processes. Despite the success of these models, they share a common assumption that the knowledge concepts of each exercise are fully labeled. However, the annotation of exercises is a labor-intensive task that requires the expertise of professionals. There are always many exercises involved in intelligent education systems, and the workload for annotation significantly increases with the number of exercises. Thus, a more practical scenario is cognitive diagnosis with limited exercise labels.

Performing cognitive diagnosis using limited exercise labels is indeed an under-explored area in current research. When faced with the few labeled exercise, a straightforward approach is to prefill the Q-matrix in order to provide a complete Q-matrix as input to an existing CD model. One limitation of this method is that its effectiveness largely depends on the accuracy of the pre-filling algorithm. If the pre-filling algorithm fails to identify the knowledge concepts associated with each exercise correctly, the resulting filled Q-matrix may be inaccurate or incomplete, which can negatively impact the model's diagnostic performance. Therefore, how to achieve interpretable diagnosis on specific knowledge concepts with limited exercise labels remains a challenge.

In this work, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges posed by limited exercise labels. Specifically, DCD is a semi-supervised disentanglement method to model the students' proficiency with limited exercise labels. To achieve this, we utilize historical practice records to model student proficiency, exercise difficulty, and exercise relevance on each knowledge concept. Then, we introduce two novel modules - the group-based disentanglement module and the limited-labeled alignment module - to disentangle factors relevant to knowledge concepts and align them with real limited exercise labels. Particularly, we introduce the tree-like structure of concepts (shown in Fig. 1-(b)) for group-based disentangling, as concepts of different levels exhibit different independence relationships. The knowledge concept tree reduces the workload required for annotating knowledge concepts, as there are far fewer concepts than exercises to annotate. These modules work together can deal with the limited exercise labels and diagnose the student's proficiency on each concept with interpretability. The main contributions of this work are summarized as follows:

Our work represents one of the few attempts to focus on the problem of limited exercise labels in cognitive diagnosis, which is common in real-world practice.

We propose a semi-supervised disentanglement approach to address the challenge of cognitive diagnosis with limited exercise labels.

We conduct extensive experiments under different few-labeled settings on three real-world datasets, which demonstrates the effectiveness of our proposed DCD model.

2 Related Work

2.1 Cognitive Diagnosis in Intelligent Education Systems

The early CD works like IRT [10] and MIRT [35] focus on modeling students' answering process by predicting the probability of a student answering a question correctly, which utilizes latent factors as

the student's ability. IRT and MIRT are also known as Latent Factor Models (LFM). The obvious limitation of the above models is the lack of interpretability, i.e., the inability to obtain an explicit multidimensional diagnostic report on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of exercises to diagnose students' proficiency in all knowledge concepts [8, 38, 6, 47, 21]. Representative NCDM [38] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works [10, 35]. However, researchers find that the free student proficiency vector learning paradigm in NCDM is not capable of tackling weak-knowledge-coverage scenario and model relation among knowledge concepts implicitly to address the problem [39, 30].

Cognitive Diagnosis in Few-labeled Scenarios. Most existing CD models are unable to directly deal with the few-labeled scenarios, and assume that the exercises are fully labeled. NCDM+ [39] fills the missing knowledge concepts by TextCNN [18] from the exercise's text information, which regards it as a previous task for CD. Therefore, the performance of CD lies in the accuracy of pre-filled knowledge concepts. In this work, we focus on a more challenging scenario of missing exercise textual information and infer missing knowledge concepts from response records only.

2.2 Disentangled Representation Learning

Disentangled Representation Learning (DRL) is an important learning paradigm that aims to disentangle the underlying generative factors hidden in the observed data into the latent variables in the representation [1]. One notable benefit attributed to DRL is its ability to extract latent factors that embody semantic meanings, thus enhancing the interpretability of machine learning models. Both VAE [19] and GAN [12] based DRL are widely adopted techniques. Considering that the training of VAE is more stable than that of GAN [17, 45], in this work, we extend the V AE framework and focus on learning interpretable disentangled representations to help cognitive diagnosis.

VAE-based Disentangled Representation Learning. β -VAE [13] modifies the Evidence Lower Bound (ELBO) of VAE [19] by adding a hyperparamter β on KL term and find that a larger β corresponds to the better disentanglement of latent code. However, the increasing β would also lead to the worse reconstruction error dramatically [17, 36]. To analyze the trade-off between reconstruction accuracy and the quality of disentangled representation, both FactorV AE [17] and β -TCVAE [5] show that the success of β -VAE in learning disentangled representations can be attributed to penalizing the Total Correlation (TC) term. β -TCVAE decomposes the expected KL term in ELBO into index-code MI term, TC term, and dimension-wise KL term. The TC term would decrease the mutual information among factors and improve independence among latent factors, which results in better disentanglement. However, some researchers claim that the above unsupervised learning of disentangled representations is fundamentally impossible without inductive biases as sometimes the target dataset is not semantically clear and well-structured to be disentangled [27]. Hence, more works focus on weak-supervised approaches [28, 37] and semi-supervised [29, 16]. A theoretical framework is provided to assist in analyzing the disentanglement guarantees by weak supervision methods (e.g. restricted labeling, match pairing, and rank pairing) [37]. Some researchers validate that with little and imprecise supervision (e.g. manual labeling of factors) it is possible to reliably learn disentangled representations [29]. In this work, we regard limited exercise labels as a semi-supervision to help disentangle knowledge concepts-related factors.

Disentangled Representation Learning for User Modeling. User modeling [22] aims to capture a number of attributes of each user, with the help of items, item features and/or user-item response matrix [43], etc. DRL has a wide range of applications in user modeling to disentangle attributes. For example, recommendation with several aspects of users' interests [23, 31, 46, 40], fair user representation to disentangle sensitive attributes [7, 34].

3 Problem Formulation

Let U = { u 1 , u 2 , . . . , u N } , V = { v 1 , v 2 , . . . , v M } , and K = { k 1 , k 2 , . . . , k K } denote the sets of students, exercises, and knowledge concepts, respectively, where N , M , and K represent the size of each set. The response records are denoted as X = { X ij } N × M , where X ij equals 1, 0 or -1 representing that the student answered the exercise correctly, incorrectly, or did not answer the exercise, respectively. The relationship between exercises and knowledge concepts is represented by the Q-matrix Q = { Q ij } K × M . Typically, each exercise is related to multiple knowledge concepts.

Q ij equals 1 or 0 representing that exercise v j is related to the knowledge concept k i or not. In this work, we explore the few-labeled situation, that is, there are only a few exercises that are labeled. Thus, we split exercise set V into the labeled set V 1 and the unlabeled set V 2 .

There are correlations and independence among different knowledge concepts. In our work, we introduce the knowledge concept tree [41] to model their relations at different levels. By utilizing this tree structure, we can effectively capture knowledge concepts' interrelationships. For convenience, we transform the knowledge concept tree into a standard tree (the height of all subtrees of any node in the tree is equal). For more details about knowledge concept tree, please see Appendix C. Suppose we construct a standard tree G with L levels, wherein its leaf nodes are denoted by the set K = { k 1 , k 2 , . . . , k K } . Each level of knowledge concept can be regarded as a grouping of the last level of knowledge concepts. The deeper the level, the finer the granularity of the grouping. Let G i = { G i 1 , G i 2 , . . . , G i | G i | } denote grouping method of the i -th level knowledge concepts, where i = 1 , 2 , . . . , L and ∣ ∣ G i +1 ∣ ∣ ≥ ∣ ∣ G i ∣ ∣ . Therefore, G 1 = { G 1 1 = K} and G L = { G L 1 = { k 1 } , G L 2 = { k 2 } , . . . , G L K = { k K }} .

Problem Definition. In an intelligent education system, given student set U , exercise set V = V 1 ∪V 2 , knowledge concepts K , response records X , knowledge concept tree G and incomplete Q-matrix, our goal is to diagnose students' cognitive states on knowledge concept set K with few-labeled exercises, and predict the scores of students doing exercises.

4 The Proposed Model

In cognitive diagnosis, it is crucial to determine whether a student has the proficiency in knowledge concepts required to answer an exercise correctly. This becomes even more challenging when there are limited exercise labels available. To model students' proficiency with few-labeled exercises, we design disentanglement and alignment modules that can effectively utilize the weak supervision of a few labeled exercises.

The overall framework of DCD is illustrated in Fig. 2, including input, encoder, disentanglement, alignment, and decoder. The input of DCD contains the student-exercise interaction matrix X and the few-labeled matrix Q . For diagnosing students' cognitive states, we first utilize the encoder module to model three important components from X : student proficiency, exercise difficulty, and exercise relevance. Then, we leverage the group-based disentanglement module to separate the factors related to knowledge concepts from the three components mentioned above. Next, the limited-labeled alignment module associates each decoupled factor with a knowledge concept semantic with the missing matrix Q . Finally, the decoder module predicts the scores based on the student proficiency, exercise difficulty, and exercise relevance. Overall, DCD provides an effective approach for cognitive diagnosis under limited exercise labels.

4.1 Encoder Module

The interaction matrix X provides valuable information regarding student proficiency, exercise difficulty, and exercise relevance from different perspectives. Specifically, 1) From a student-centric (row-wise) perspective, the response records of a student reflect his proficiency on all knowledge concepts. 2) From an exercise-centric (column-wise) perspective, the response records of an exercise reflect exercise difficulty and exercise relevance on all knowledge concepts. On the one hand, to effectively model from two perspectives, we design the student proficiency encoder, exercise difficulty encoder, and exercise relevance encoder modules. On the other hand, following existing cognitive models, an interaction function (corresponding to our decoder) to predict the answering result with student traits and exercise traits as input (corresponding to our encoders), which ensures the interpretability of student proficiency representation. Traditional single encoder is not applicable for cognitive diagnosis.

Student Encoder. The student encoder module is intended to infer student's proficiency on each knowledge concept from X , which models true posterior distribution p ( z u | x u ) by constructing an inference network f ϕ u ( x u ) corresponding to approximate posterior distribution q ϕ u ( z u | x u ) , parameterized by ϕ u . We assume student's proficiency on all knowledge concepts z u follows a multivariate standard Gaussian distribution prior, i.e., p ( z u ) ∼ N ( 0 , I ) . The approximate posterior

-1

0

1

1

-1

...

1

...

-1

0

0: Incorrect

1: Correct

-1:Unanswered

Input

Interaction matrix

Limited Labeled Q matrix

1

0

0

1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

Exercises

Students

Exercises

Concepts

0

1

0

0

0: Irrelevant

1: Relevant

-1: Unlabeled

0

1

...

-1

0

0

1

...

-1

0

0

1

-1

...

1

0

1

-1

...

1

0

1

-1

...

1

0

1

-1

...

1

Student Proficiency Encoder

Exercise Difficulty Encoder

0

1

0

0

0

1

0

0

Proficiency

Difficulty

Relevance

Decoder

(Section 4.4)

Align-labeled loss

Align-unlabeled loss

Main

loss

(

)

-

*

Disentanglement loss

Group-based Disentanglement Module (Section 4.2)

Labeled

Unlabeled

d

1

d

2

Margin

MSE

1

st

level

2

nd

level

3

rd

level

(Leaf level)

0

1

0

0

r

v

μ

r

v

μ

r

v

μ

d

v

μ

d

v

σ

u

μ

u

σ

BCE

Concept tree

-1

(Root level)

Limited-Labeled Alignment Module (Section 4.3)

Exercise Relevance Encoder

Encoder

Encoder

Encoder

Disentangling at

2

nd

level

Disentangling at

Leaf level

Factors

K

L2

for K -dimensional latent code z u ∈ R K is expressed as follows:

where µ u ∈ R K and σ u ∈ R K > =0 are generated from student encoder f ϕ u ( x u ) . The [ · ] indicates the index selection operation.

Exercise Encoders. We design two encoders at exercise side: exercise difficulty and exercise relevance encoders. The two encoders aim to infer exercise's difficulty and exercise's relevance on each knowledge concept from X , respectively. For exercise difficulty encoder, we assume exercise's difficulty on all knowledge concepts z d v follows a multivariate standard Gaussian distribution prior similar to the student encoder, i.e., p ( z d v ) ∼ N ( 0 , I ) .

We assume exercise's relevance on all knowledge concepts z r v follows a multivariate Bernoulli prior, i.e. p ( z r v ) ∼ Bernoulli ( p ) , where p is a hyperparameter. The exercise relevance encoder models true posterior distribution p ( z r v | x v ) by constructing an inference network f ϕ r v ( x v ) corresponding to approximate posterior distribution q ϕ r v ( z r v | x v ) , parameterized by ϕ r v . The approximate posterior for K-dimensional latent code z r v ∈ { 0 , 1 } K is expressed as:

where µ r v ∈ [0 , 1] K are generated from exercise encoder f ϕ v ( x v ) .

4.2 Group-based Disentanglement Module

The representations obtained by the encoder module do not relates to the knowledge concepts, which makes it difficult to perform an accurate cognitive diagnosis. To address this issue, we assume the entire response records as evolving from three groups of generative factors: student proficiency factors on each knowledge concept, exercise difficulty factors on each knowledge concept, and exercise relevance factors on each knowledge concept. We desire to disentangle these factors following existing DRL methods.

Inspired by the outstanding performance of β -TCVAE [5] in disentanglement, we introduce the structure of β -TCVAE to obtain a better trade-off between the reconstruction accuracy and the quality of disentangled representation. We find the index-code term of β -TCVAE plays a side effect on performance and drop it during optimization like [32, 7]. Take student proficiency factors as an example, the eventual Totoal Correlation (TC) constraint on z u can be expressed as follows:

where q ( z u ) = ∑ N i =1 q ( z u | x u i ) p ( x u i ) is the aggregated posterior of z u . The TC term would decrease the mutual information among latent factors and improve independence among latent factors. The dimension-wise KL constraint on z u can be written as follows:

where p ( z u [ i ]) follows the prior standard Gaussian distribution. The dimension-wise KL term prevents each latent variable from deviating too far from specified priors.

However, some CD works argue that the knowledge concepts are not independent and employ relation among knowledge concepts to improve CD [11, 39]. The direct independence constraint for each knowledge concept may be not applicable for cognitive diagnosis, which is also verified in our experiment. To address the challenge, we propose to utilize the easy-labeled knowledge concept tree to capture underlying independence among knowledge concepts. We assume there is less independence among groups of knowledge concepts grouped by their parent nodes in the knowledge concept tree. Corresponding to the real world, knowledge concepts in the same chapter would be related more than knowledge concepts in different chapters. Therefore, each level concept in the tree is a kind of grouping method for the last level concepts. The deeper the level, the finer the granularity of the grouping. Considering the grouping methods at the i -th level, the Eq. (3) can be rewritten as follows:

where G i j is the knowledge concepts that belong to j -th group according to the grouping method of i -th level in the tree. The Eq. (5) allows dependence in intra-group knowledge concepts and maintains independence among inter-group knowledge concepts.

4.3 Limited-Labeled Alignment Module

After applying the group-based disentanglement module, it is important to correspond each latent factor with an actual knowledge concept to ensure the interpretability. However, there are only a few labeled exercises with knowledge concepts, making it challenging to perform the accurate alignment. Inspired by semi-supervised DRL, to address this issue, we devised an alignment module that aligns both few labeled and numerous unlabeled exercises separately. For convenience, we use q j ∈ { 0 , 1 } K to denote j -th column of matrix Q corresponding to exercise v j that belongs to V 1 .

Few Labeled Exercises. For an exercise v j ∈ V 1 , we utilize the Mean Square Error (MSE) loss to constraint similarity between q j and µ r v j , which can be defined as follows:

where V 1 means labeled exercise set. For semi-supervised DRL in the vision field, it's usually enough with a single alignment module for a few labeled samples. However, for cognitive diagnosis, the annotation vector (i.e., the column of Q-matrix) of exercise is very sparse with only one or two knowledge concepts per exercise. The challenge is that utilizing a few labeled exercises to do alignment is too hard to keep such sparsity in relevance representation for unlabeled exercises. To this end, the alignment of unlabeled exercises is necessary.

Numerous Unlabeled Exercises. For an exercise v j ∈ V 2 , we have no idea any knowledge concept of it. But we know there are only a few knowledge concepts of each exercise. The feature of sparsity

in µ r v j would help infer missing knowledge concepts. We just need to guarantee that there are only a few elements to be one and other elements to be zero in µ r v j . In this work, we employ a margin loss and an L2 loss to achieve this target, which can be expressed as follows:

where m is the margin, λ 1 is the hyperparameter to control margin loss, and λ 2 is the hyperparameter to control L2 loss. We use µ r v [ max @ k ] and µ r v [ max # k ] to denote the k -th largest element of µ r v and the top k largest elements of µ r v , respectively. In Eq. (7), we first sort µ r v j by descending order and then split the sorted one into three parts of length d 1 , d 2 , K -d 1 -d 2 . We argue that knowledge concepts in the last part are the least likely to be the ground-truth knowledge concepts of exercise v j and knowledge concepts in the first part are the most likely to be the ground-truth knowledge concepts of exercise v j . Therefore, we adopt a margin loss between knowledge concepts in the first part and knowledge concepts in the last part, which requires that all values in the first part (i.e., µ r v j [ max # d 1 ] ) is at least m greater than the largest value in the last one (i.e., µ r v j [ max @( d 1 + d 2 +1)] ). For the middle part, we do not constrain anything on it. Intuitively, it's reasonable that adopt an L1 loss to constraint the value in the last part (i.e., µ r v j [ min #( K -d 1 -d 2 )] ) to be 0. In our initial attempt, we first directly try to use L1 loss for numerous unlabeled exercises, and we find it does not show competing performance (detailed in Appendix A.3). We speculate a possible reason is as follows: with L1 loss, after model initialization, it is very hard to revise these incorrectly labeled exercises. In contrast, L2 loss are sensitive to outliers, and would have a larger loss when the corresponding knowledge is incorrectly predicted.

4.4 Decoder Module

̸

After aligning the exercise relevance representation with the real knowledge concepts, the decoder module predicts scores of students doing exercises, which could be described as a triplet set T = { ( u i , v j , X ij ) | X ij = -1 } . The object function can be described as follows:

where BCE ( · , · ) is the binary cross entropy loss function, ⊗ and σ ( · ) denote inner product operation and sigmoid function, respectively.

Similar to existing CD works [38, 39, 21], the interaction function (i.e., the decoder) keeps the monotonicity assumption [39], which guarantees the interpretability of student proficiency representation (i.e., the higher the value of the latent factor means the better proficiency on the corresponding knowledge concept, detailed in Appendix A.4). In addition, the element-wise operation in the decoder module propagates the alignment (i.e., each latent factor corresponds to each real knowledge concept) on exercise relevance representation to both student proficiency representation and exercise difficulty representation, which is also the same as existing works.

Optimization. We summarized all object functions in all modules and combine them for final optimization. We set different hyperparameters to balance each loss function. The final object function can be summarized as follow:

where Θ = [ ϕ u , ϕ d v , ϕ r v ] is the parameter set in the whole model, α is the hyperparameter for alignment of labeled exercises, and β i denotes the weight for disentanglement term corresponding to the i -th level in the knowledge concept tree.

5 Experiments

5.1 Experimental Settings

Datasets. Our experiments are conducted on three real-world datasets, i.e., Matmat 2 , Junyi [4] and NIPS2020EC [42], all of which contain knowledge concepts of the tree structure. The statistics of these datasets are summarized in Table 1. We adopt five-fold cross-validation to avoid randomness. The details about datasets and implementation 3 are depicted in the Appendix B.

Metrics. To evaluate the effectiveness of DCD in terms of score prediction and interpretability, we adopt three widely used prediction metrics: AUC [3], ACC, RMSE, and one interpretability metric: Degree of Agreement (DOA) [38]. DOA is the most commonly used interpretability metric in the CD, which measures the degree of agreement between cognitive results and response records.

Baselines. The baselines including the classical data mining model Probabilistic Matrix Factorization (PMF) [33], the latent factor models IRT [10], MIRT [35], and the representative CD models DINA [8], NCDM [38], KaNCD [39] and KSCD [30]. In addition, we also explore a simplified version of our method. Specifically, we remove the exercise difficulty module and margin loss, and focus solely on the fine-grained disentanglement module. This simplified version serves as a baseline, which we refer to as β -TCVAE in the following analysis.

Existing non-interpretable models such as PMF, IRT, and MIRT cannot obtain a student's proficiency for each knowledge concept [38]. Consequently, DOA cannot be calculated for these models. Interpretable models such as DINA, NCDM, KaNCD, and KSCD cannot directly apply to fewlabeled scenarios. To ensure fairness in comparison, we propose a pre-filling algorithm (detailed in Appendix B.2) for the missing Q-matrix before applying these models.

5.2 Performance Comparison

To evaluate the effectiveness of our proposed DCD model in few-labeled scenarios, we have conducted experiments on datasets with 10% Q-matirx (few-labeled exercises), 20% Q-matirx (few-labeled exercises) and 100% Q-matirx (fully-labeled exercises). The detailed performance comparison on three datasets is displayed in Table 2.

Prediction Comparison. 1) Our proposed model consistently outperforms all interpretable baseline models across the different scenarios, demonstrating its effectiveness. DINA, which only considers binary space, performs the worst. Both KSCD and KaNCD implicitly model relation among knowledge concepts and has a better performance than NCDM. 2) However, all non-interpretable models achieve the best prediction metrics over interpretable models on Junyi and NIPS2020EC datasets except Matmat dataset. This may be because there are far fewer knowledge concepts in the Matmat dataset than in the other two datasets. To meet the interpretability of larger knowledge concept space, interpretable models lose partial prediction accuracy. This explanation is supported by the fact that the prediction results of interpretable models show a noticeable improvement in the few-labeled scenario compared to the fully labeled scenario, indicating that the trade-off between interpretability and accuracy is more significant in a larger knowledge concept space.

Interpretability Comparison. 1) Table 2 demonstrates that our method achieves the best interpretability in all scenarios. Especially, for the 10% and 20% Q-matrix (few-labeled scenarios), our model significantly outperforms interpretable baselines in terms of DOA, demonstrating its superior interpretability in this scenario. 2) In few-labeled scenarios, KaNCD and KSCD show a remarkable improvement in DOA compared to DINA and NCDM. This is because DINA and NCDM can only

diagnose knowledge concepts labeled in the training set. When incorrect knowledge concepts are filled in, KaNCD and KSCD are less affected, while DINA and NCDM cannot accurately diagnose the true knowledge concepts. In contrast, our model avoids this issue by modeling the distribution of students' proficiency overall knowledge concepts.

5.3 Disentanglement Analysis

To investigate the effectiveness of disentanglement, we conducted an experiment where we compared our method at different disentanglement weights β in the few-labeled scenario. Fig. 3 (a-d) illustrates the results, the x-axis shows the value of β from 0 to 2, which corresponds to the strength of disentanglement. The AUC is displayed on the left y-axis and the DOA is displayed on the right y-axis. The AUC(i) and DOA(i) denote the results by disentanglement only according to i-th level of knowledge concept tree. There are some observations from Fig. 3: 1). The disentanglement according to the last level of tree (i.e., the most fine-grained disentanglement) can not have an improvement in terms of DOA as the rise of β . The phenomenon is consistent with the assumption that knowledge concepts are correlated. Despite there being an improvement in AUC, we assume it's the result of interpretability losses, as mentioned in the prediction comparison in section 5.2. 2) The coarse-grained disentanglement could simultaneously improve the AUC and DOA compared to no disentanglement. We observe that the performance increased initially with the implementation of disentanglement and then decreased as the strength of disentanglement became excessive, leading to counterproductivity.

5.4 Alignment Analysis

To evaluate the effectiveness of the limited-labeled alignment module, we conducted an experiment with different hyperparameters on Junyi dataset with 10% Q-matrix, namely α , λ 1 , λ 2 and d 2 , as shown in Fig. 3 (e-h). Due to the number of leaf concepts per exercise being close to 1 for all datasets,

0

0.5

1

2.5

3

4

5

0.81

0.811

0.812

0.813

0.814

0.815

0.816

AUC

0.6

0.65

0.7

0.75

0.8

DOA

0.81

0.811

0.812

0.813

0.814

0.815

0.816

AUC

0.69

0.695

0.7

0.705

0.71

0.715

0.72

DOA

0.81

0.811

0.812

0.813

0.814

0.815

0.816

AUC

0.705

0.71

0.715

DOA

0.81

0.811

0.812

0.813

0.814

0.815

0.816

AUC

0.71

0.712

0.714

0.716

0.718

DOA

0

0.1

0.5

1

2

β

0.806

0.808

0.81

0.812

0.814

0.816

AUC

0.69

0.695

0.7

0.705

0.71

0.715

0.72

DOA

Junyi with 10% Q Matrix

AUC(3)

AUC(2)

DOA(2)

0.806

0.808

0.81

0.812

0.814

AUC

0.71

0.715

0.72

0.725

0.73

0.735

0.74

DOA

Junyi with 20% Q Matrix

0.806

0.808

0.81

0.812

0.814

AUC

0.775

0.78

0.785

0.79

DOA

NIPS2020EC with 10% Q Matrix

AUC(4)

AUC(3)

AUC(2)

DOA(4)

DOA(2)

Junyi with 10% Q Matrix

Junyi with 10% Q Matrix

Junyi with 10% Q Matrix

Junyi with 10% Q Matrix

AUC(3)

AUC(2)

DOA(2)

0

0.1

0.5

1

2

0

0.1

0.5

1

2

0

0.1

0.5

1

2

β

β

β

(a)

(b)

(c)

(d)

0.1

0.5

1

4

6

10

2

8

0

0.25

1

4

6

2

0.5

0.75

1

2

5

7

8

6

3

4

9

λ

2

α

λ

1

d

1+

d

2

(

d

1=1)

(g)

(e)

(f)

(h)

0.8

0.802

0.804

0.806

0.808

0.81

0.812

AUC

0.78

0.785

0.79

0.795

DOA

NIPS2020EC with 20% Q Matrix

AUC(4)

AUC(3)

AUC(2)

DOA(4)

DOA(2)

we set the d 1 = 1 as the default value. The default margin is 0.5. There are some observations: 1) α is the weight of the alignment loss corresponding to labeled exercises. Fig. 3 (e) shows that the performance of our model initially improves and eventually stabilizes as α increases. 2) λ 1 is the weight of margin loss for unlabeled exercises. As illustrated in Fig. 3 (f), the employment of margin loss is effective according to results from zero to non-zero of λ 1 . The excessive λ 1 would decrease the performance. 3) λ 2 is the weight of L2 loss of unlabeled exercises exercise relevance representation. We can observe that the employment of the L2 loss plays an essential role from Fig. 3 (g). 4) We can observe that the employment of d 2 is effective from Fig. 3 (h). As d 2 increases from 0 to 8, the performance increases first and then drops. Overall, the limited-labeled alignment module has a positive effect on performance, and the hyperparameters of the alignment module need to be certainly adjusted.

5.5 Ablation Study

As shown in Table 3, to prove the effectiveness of our proposed disentanglement and alignment modules, we conduct the ablation study on Junyi dataset under the 10% Q-matrix scenario. The margin Loss in alignment module ensures few elements to be one in exercise relevance repre-

sentation, which achieves an improvement of 0.53% in terms of DOA. The L2 loss in alignment module ensures the sparsity of exercise relevance representation, which achieves an improvement of 13.48% in terms of DOA. It proves that utilizing a few labeled exercises to do alignment is too hard to keep such sparsity in relevance representation for unlabeled exercises. The disentanglement module alone achieves 0.56% in terms of DOA. Overall, both the proposed disentanglement and alignment modules achieve an improvement in terms of prediction and interpretability metrics.

6 Conclusion

In this paper, we present a novel approach, called DCD, to tackle the interpretability problem of cognitive diagnosis with limited exercise labels. Inspired by semi-supervised DRL, we learn disentangled representations and align them with real limited labels by group-based disentanglement and limited-labeled alignment. Extensive experiments on widely used benchmarks demonstrate the superiority of our proposed method in few-labeled scenarios.

Acknowledgments and Disclosure of Funding

This work was supported in part by grants from the National Key Research and Development Program of China (Grant No. 2021ZD0111802), the National Natural Science Foundation of China (Grant No. 61972125, 72188101, 61932009, 62376086, and U22A2094).

References

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI , 35(8):1798-1828, 2013.

Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In ICDM , pages 42-51. IEEE, 2020.

Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition , 30(7):1145-1159, 1997.

Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In EDM , pages 532-535, 2015.

Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In NeurIPS , pages 2615-2625, 2018.

Song Cheng, Qi Liu, Enhong Chen, Zai Huang, Zhenya Huang, Yiying Chen, Haiping Ma, and Guoping Hu. Dirt: Deep learning enhanced item response theory for cognitive diagnosis. In CIKM , pages 2397-2400, 2019.

Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In ICML , pages 1436-1445, 2019.

Jimmy De La Torre. Dina model and parameter estimation: A didactic. Journal of educational and behavioral statistics , 34(1):115-130, 2009.

Janez Demšar. Statistical comparisons of classifiers over multiple data sets. JMLR , 7:1-30, 2006.

Susan E Embretson and Steven P Reise. Item response theory . Psychology Press, 2013.

Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In SIGIR , pages 501-510, 2021.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , pages 2672-2680, 2014.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR , 2017.

Zhenya Huang, Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In CIKM , pages 1261-1270, 2019.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Ilyes Khemakhem, Diederik P. Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. In ICAIS , 2019.

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML , pages 2649-2658, 2018.

Yoon Kim. Convolutional neural networks for sentence classification. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, EMNLP , pages 1746-1751. ACL, 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2014.

George D Kuh, Jillian Kinzie, Jennifer A Buckley, Brian K Bridges, and John C Hayek. Piecing together the student success puzzle: Research, propositions, and recommendations: ASHE higher education report , volume 116. John Wiley & Sons, 2011.

Jiatong Li, Fei Wang, Qi Liu, Mengxiao Zhu, Wei Huang, Zhenya Huang, Enhong Chen, Yu Su, and Shijin Wang. Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework. In SIGKDD , pages 904-913, 2022.

Sheng Li and Handong Zhao. A survey on representation learning for user modeling. In IJCAI , pages 4997-5003, 2020.

Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In WWW , pages 689-698, 2018.

Fei Liu, Xuegang Hu, Shuochen Liu, Chenyang Bu, and Le Wu. Meta multi-agent exercise recommendation: A game application perspective. In SIGKDD , page 1441-1452, New York, NY, USA, 2023.

Qi Liu. Towards a new generation of cognitive diagnosis. In IJCAI , pages 4961-4964, 2021.

Yingjie Liu, Tiancheng Zhang, Xuecen Wang, Ge Yu, and Tao Li. New development of cognitive diagnosis models. FCS , 17(1):171604, 2023.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML , pages 4114-4124, 2019.

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In ICML , pages 6348-6359, 2020.

Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In ICLR , 2020.

Haiping Ma, Manwei Li, Le Wu, Haifeng Zhang, Yunbo Cao, Xingyi Zhang, and Xuemin Zhao. Knowledge-sensed cognitive diagnosis for intelligent education platforms. In CIKM , pages 1451-1460, 2022.

Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. In NeurIPS , pages 5712-5723, 2019.

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In ICLR , 2016.

Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. In NeurIPS , pages 1257-1264, 2007.

Sungho Park, Dohyung Kim, Sunhee Hwang, and Hyeran Byun. Readme: Representation learning by fairness-aware disentangling method. arXiv , 2020.

Mark D Reckase and Mark D Reckase. Multidimensional item response theory models . Springer, 2009.

Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In ICML , pages 8655-8664, 2020.

Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In ICLR , 2020.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In AAAI , pages 6153-6161, 2020.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuralcd: A general framework for cognitive diagnosis. TKDE , 2022.

Xin Wang, Hong Chen, and Wenwu Zhu. Multimodal disentangled representation for recommendation. In ICME , pages 1-6, 2021.

Xinping Wang, Caidie Huang, Jinfang Cai, and Liangyu Chen. Using knowledge concept aggregation towards accurate cognitive diagnosis. In CIKM , pages 2010-2019, 2021.

Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobato, Richard E. Turner, Richard G. Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Instructions and guide for diagnostic questions: The neurips 2020 education challenge, 2021.

Le Wu, Xiangnan He, Xiang Wang, Kun Zhang, and Meng Wang. A survey on accuracy-oriented neural recommendation: From collaborative filtering to information-rich recommendation. TKDE , 35(5):4425-4445, 2023.

Le Wu, Peijie Sun, Yanjie Fu, Richang Hong, Xiting Wang, and Meng Wang. A neural influence diffusion model for social recommendation. In SIGIR , pages 235-244, 2019.

Hengshi Yu and Joshua D Welch. Michigan: sampling from disentangled representations of single-cell data using generative adversarial networks. Genome biology , 22(1):1-26, 2021.

Yin Zhang, Ziwei Zhu, Yun He, and James Caverlee. Content-collaborative disentanglement representation learning for enhanced recommendation. In RecSys , pages 43-52, 2020.

Yuqiang Zhou, Qi Liu, Jinze Wu, Fei Wang, Zhenya Huang, Wei Tong, Hui Xiong, Enhong Chen, and Jianhui Ma. Modeling context-aware features for cognitive diagnosis in student learning. In SIGKDD , pages 2420-2428, 2021.

A Additional Experiments

A.1 Comparison on Inferring Missing Knowledge Concepts

To evaluate the degree of alignment from exercises' relevance, we adopt Hit Ratio (HR) [44] to measure whether the top-k ranked elements in exercise relevance representation hit a ground-truth knowledge concept. The algorithm about filling missing Q-matrix for baselines (detailed in Appendix B.2) is compared with our exercise relevance encoder. The filling results of HR@2 is illustrated in Table 4. We can observe that: 1) It's rational that inferring missing knowledge concepts according to response records of exercises. The filling algorithm and our method show a obvious improvement than randomly assigning knowledge concepts for unlabeled exercises. 2) Our method demonstrates the best HR@2 on Matmat dataset, but shows a worse performance on other datasets. This is because the number of knowledge concepts in Matmat has a smaller scale than other datasets and our model has a better inferring performance in small scale knowledge concepts. Despite the worse filling results than filling algorithm for baselines, our method still shows the best performance in prediction metrics and interpretable metrics, which demonstrate the robustness of the proposed DCD approach. This may be attributed to the inference of student proficiency of our student encoder module.

A.2 Experiments under other few-labeled scenarios

We conduct experiments evaluating other missing ratios of the Q-matrix on the NIPS2020EC dataset. In Table 2 of the main body, we illustrated the results under 100%, 20% and 10% settings. Here we additionally add experiments on 50%, 30%, and 5% Q-matrix scenarios, which are illustrated in 5. The experimental results demonstrate our DCD outperforms all the interpretable baseline models in these few-labeled scenarios. Besides, as the preserved ratio of the Q-matrix decreases (from 50% to 5%), our DCD demonstrates a greater improvement (from 0.86% to 1.43%) compared to the second-ranked model in terms of DOA. This shows the advantage of DCD in label-scarce scenarios.

A.3 Comparison between L1 and L2 loss for unlabeled exercises in alignment module

In section 4.3, we finally adopt the L2 loss to constraint the exercise relevance representation sparsity of unlabeled exercises. Intuitively, it's reasonable that adopt an L1 loss to constraint the value in the last part (i.e., µ r v j [ min #( K -d 1 -d 2 )] ) to be 0. As shown in 6, in our initial attempt, we first directly try to use L1 loss for numerous unlabeled exercises, and we find it does not show competing performance. We speculate a possible reason is as follows: with L1 loss, after model initialization, it is very hard to revise these incorrectly labeled exercises. In contrast, L2 loss are sensitive to outliers, and would have a larger loss when the corresponding knowledge is incorrectly inferred.

A.4 Monotonicity Analysis

The monotonicity assumption [38, 39] in CD works means that the probability of correct response to the exercise is monotonically increasing at any dimension of the student's knowledge proficiency, which guarantees the interpretability of student proficiency representation (i.e., the higher the value of the latent factor means the better proficiency on the corresponding knowledge concept). Given a student proficiency representation, each dimension in the representation corresponds to the student's proficiency level on a specific knowledge concept. When we only increase the value of a single dimension in the student representation (improving the student's proficiency on a specific knowledge concept), while keeping the exercise diffi-

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

student proficiency on one specify knowledge concept

0.0

0.2

0.4

0.6

0.8

1.0

Average predicted probability of

answering questions correctly

exercises that contain the concept 1

exercises that contain the concept 2

exercises that contain the concept 3

culty representation and relevance representation unchanged, the student's probability of answering questions related to that knowledge concept correctly will increase, while the probability of answering questions unrelated to that knowledge concept will remain unchanged. In Fig. 6, we present a case where we only increase a single dimension in the student representation, and it shows that the student's probability of answering questions related to that knowledge concept correctly increases.

A.5 Nemenyi Test

The Nemenyi test [9] is conducted to present the comparison between our proposed DCD and the interpretable baselines for all 5-fold cross-validation results. A significant difference is regarded to exist in the Nemenyi tests if the average ranks of two models differ by at least one crucial difference, which is determined using a 5% significance level. The model metric improves as the ranking score decreases. As illustrated in Fig 5, Fig 6 and Fig 7, which demonstrates the comparison on 100%, 20% and 10% labeled exercises scenario, respectively. It's obvious that our proposed DCD approach performs best for all metrics in any scenario.

(a) AUC

(b) ACC

(c) RMSE

(d) DOA

CD=1.95

6

5

4

3

2

1

1 DCD

2.33 KaNCD

3.33

KSCD

3.67

β

-TCVAE

5

NCDM

5.67

DINA

CD=1.95

6

5

4

3

2

1

1.43 DCD

1.57 KaNCD

3.33

β

-TCVAE

4

KSCD

4.67

NCDM

6

DINA

CD=1.95

6

5

4

3

2

1

1.07 DCD

2.47

β

-TCVAE

2.93

KaNCD

4

KSCD

4.53

NCDM

6

DINA

CD=1.95

6

5

4

3

2

1

1.07 DCD

2.93 KaNCD

3

DINA

3.67

NCDM

4.33

β

-TCVAE

KSCD 6

CD=1.95

6

5

4

3

2

1

1.33

DCD

1.8

KaNCD

2.87

KSCD

4.4

NCDM

4.6

β

-

TCVAE

6

DINA

CD=1.95

6

5

4

3

2

1

1.07 DCD

2

KaNCD

3.67

β

-

TCVAE

KSCD 4

NCDM 4.27

DINA 6

CD=1.95

6

5

4

3

2

1

1.4 DCD

1.6 KaNCD

3.67

β

-

TCVAE

3.8

KSCD

NCDM 4.53

6

DINA

(a) AUC

(b) ACC

(c) RMSE

(d) DOA

CD=1.95

6

5

4

3

2

1

1 DCD

2.4 KaNCD

3 KSCD

4.27

β

-

TCVAE

NCDM 5.33

(a) AUC

(b) ACC

(c) RMSE

(d) DOA

CD=1.95

6

5

4

3

2

1

1.33 DCD

1.67 KaNCD

3.4

KSCD

4

NCDM

4.6

β

-TCVAE

6

DINA

CD=1.95

6

5

4

3

2

1

1.33 DCD

1.67 KaNCD

3.6

KSCD

4.13

NCDM

4.27

β

-TCVAE

6

DINA

CD=1.95

6

5

4

3

2

1

1.33

KaNCD

1.67 DCD

3.8

β

-TCVAE

3.87

KSCD

4.33

NCDM

6

DINA

CD=1.95

6

5

4

3

2

1

1

DCD

2.67 KaNCD

3.07 KSCD

3.6

β

-TCVAE

5.2

DINA

5.47

NCDM

B Experimental Details

B.1 Reparameterization Trick

In the student encoder module, student proficiency representation z u is sampled from a Gaussian distribution N ( µ u , σ u ) . If we sample z u from N ( µ u , σ u ) directly, we will lost gradient information as the sampling process is not differentiable. Fortunately, any Gaussian distribution can transform to a standard Gaussian distribution. we can sample a ϵ ∼ N ( 0 , I ) that has nothing to do with any parameters. By doing so, z u is equal to µ u + σ u ⊙ ϵ , where ⊙ means element-wise product. However, there is no similar property of Bernoulli distribution corresponding to exercise relevance encoder. In this work, we adopt gumbel-softmax reparameterization [15] to alleviate this problem, which is one of the gradient estimator methods.

B.2 Algorithm for Filling Missing Q-matrix

Existing interpretable baselines cannot directly apply to few-labeled scenarios. We design a filling method by employing similarity of response records in exercise to infer missing knowledge concepts. The detailed procedure is shown in Algorithm 1. We first compute the similarity (detailed in Eq. (10)) among exercises according to response records of exercise. Then for each unlabeled exercise, we vote the vote k most frequently occurring knowledge concepts among the most similar exer k labeled exercises as the filling knowledge concepts. We set vote k = 2 and exer k = 5 as default.

̸

where S i,j denotes the similarity between exercise v i and v j . I ( condition ) = 1 if condition is True, and vice versa I ( condition ) = 0 . X is the interaction matrix in training set. X ni = 1 , -1 represent student u n answered exercise v i correctly and incorrectly, respectively. X ni = 0 represents student u n did not answer exercise v i . U is the student set.

Algorithm 1 Filling Missing Q-matrix for Interpretable Baselines

Input: labeled exercises set V 1 , unlabeled exercises set V 2 , interaction matrix X in training set, missing Q-matrix Q , number of similar exercises exer k , number of filling knowledge concepts vote k

Output: Filling Q-matrix Q 1: M = |V 1 ⋃ V 2 | 2: Initialize exercise similarity matrix S of size ( M , M ) with the default value -∞ 3: for all exercise v i ∈ V 1 do 4: for all exercise v j ∈ V 2 do 5: Calculate S i,j according to Eq. (10) 6: end for 7: end for 8: for all exercise v j ∈ V 2 do 9: s e = the most similar exer k labeled exercises of v j according to S [: , j ] 10: s k = the vote k knowledge concepts with the highest frequency of occurrence in s e 11: Set corresponding elements to be one in Q [ j, :] by s k 12: end for 13: return Q

B.3 Datasets and Preprocessing

Our experiments are conducted on three real-world datasets, i.e., Matmat 4 , Junyi 5 [4] and NIPS2020EC 6 [42], all of which contain knowledge concepts of the tree structure. For all datasets, we preserve the first-time exercise-answering record for same student-exercise pairs to support cognitive diagnosis. The detailed information on datasets and preprocessing method are depicted as following:

Matmat. This dataset is collected from an intelligent web application for practicing mathematics called Matmat 7 , which includes three-level knowledge concept structure after preprocessing. The first-level knowledge concept is the root node called math. The second-level knowledge concepts are organized as 5 calculational types including addition, multiplication, division and so on. The third-level knowledge concepts are organized by calculational scale.We preserve students with more than 15 response records to guarantee that each student has enough data for diagnosis.

Junyi. This dataset is collected from an online educational platform called Junyi Academy 8 . There are three-level knowledge concept structure in this dataset. Besides the first-level root node, we apply area and topic field in origin dataset to denote the second-level and third-level knowledge concept ,respectively. We preserve students with more than 50 response records to guarantee that each student has enough data for diagnosis.

NIPS2020EC. This dataset is originated from NeurIPS 2020 Education Challenge [42], which provides students' answers to mathematics questions from Eedi 9 . There are four-level knowledge concept structure in this dataset. We sample a subset from the task 1 of this competition by selecting response records during March 2020. We preserve students with more than 30 response records to guarantee that each student has enough data for diagnosis.

B.4 Implementation Details

We train our model with Python 3.9 and PyTorch 1.12.1 on NVIDIA RTX A5000. The student and exercise encoders are implemented by a multilayer perceptron for all datasets. We set a prior Gaussian distribution with N (0 , 1) for each latent factor in µ u and µ d v , and a prior Bernoulli distribution with Bernoulli (0 . 2) for each latent factor in µ r v . For all interpretable models, we select the epoch with the best DOA for testing and almost set Adam as the default optimizer. The implement of DOA [38] metric is adopted from EduCDM 10 . For more implementation details, please see our public code repository https://github.com/kervias/DCD .

C Knowledge Concept Tree

C.1 Standard Knowledge Concept Tree

The employment of knowledge concept tree in our work is to obtain group based disentanglement according to granularity at different levels. For convenience, we transform the initial knowledge concept tree into a standard tree, which is defined as the tree that the height of all subtrees of any node in the tree is equal in our work. The transformation process is displayed in Fig. 8. For those leaf nodes whose depth has not reached the depth of the tree, we can continuously duplicate this node through inheritance until reaching the depth of the tree.

C.2 Example of Real Knowledge Concept Tree

In this work, we assume that there is less independence among groups of knowledge concepts grouped by their parent nodes in the knowledge concept tree. This viewpoint is inspired by practical applications, and the analysis is as follows. To better clarify the argument, we provide the partial

1

A

B

C

D

E

F

G

A

B

C

D

E

F

G

D

F

C

C

knowledge concept tree of NIPS2020EC dataset in Fig. 9. Firstly, the course chapter hierarchy is a typical example of concepts with a tree structure. Intuitively, the associations among concepts at the high level of the hierarchy are weaker compared to the associations between concepts at the low level. For example, if we adopt the 2nd level concepts (i.e., algebra, data and statistics, ...) to group the last level concepts, the independence among groups is higher than the groups determined by the 3rd level concepts (i.e., inequalities, formula, data collection, ...). Moreover, there is a higher likelihood that concepts within the same chapter are simultaneously assessed in the same exercise. For instance, the concepts whose parent concept is inequalities would have a higher probability occur in the same exercise. However, the solving linear inequalities concept whose parent concept is inequalities and the tally charts concept whose parent concept is data collection would have a lower probability occur in the same exercise.

Number

Straight Line Graphs

-Data Processing

QuadraticGraphs

-DataRepresentation

Inequalities onNumber Lines

-Tally Charts

Graphing Linear Inequalities (Shading Regi

Types of Data and Questionnaires

ons)

DataandStatisticsDataCollection

Inequalities

NIPS2020EC

Data Collection-Others

SolvingLinear Inequalities

Algebra

Frequency tables

Inequalities-Others

Probability

Solving Quadratic Inequalities

Data andStatistics-Others

Transformation ofFunctions

Geometry andMeasure

Writing and Simplifying Expressions

D Limitations

There are still some limitations of our proposed method. 1) The latent factors considered in our method are only related to knowledge concepts. However, the student answering process may refer to more factors that are not related to knowledge concepts such as the family economic conditions of students and reading comprehension difficulty of exercise textual information, which results in the incomplete and insufficient disentanglement in our proposed method. The following work we consider is to introduce more side information to help disentangle other factors. 2) The assumption that inter-group knowledge concepts are independent may be too strong. The later work we consider is to model causal relation among knowledge concepts for disentanglement. 3) The alignment module for unlabeled exercises is just to keep sparsity. In the future, we will introduce explicit relation among knowledge concepts to help infer missing knowledge concepts.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe dissociation-based cognitive diagnosis (DisenCD) model proposed in this paper addresses the cognitive diagnosis challenge of limited practice labels by using students' historical practice records to model their proficiency, practice difficulty, and practice label distribution. The model introduces novel modules to disentangle factors related to knowledge concepts and align them with the limited labels available. Experiments on three real datasets demonstrate the effectiveness of the model. Overall, this paper presents a promising approach to the cognitive diagnosis of intellectual education. However, in the process of specific description, the introduction of the two modules is too vague, and some experimental details are not provided in the article.\n\nSTRENGTHS:\nPractical Approach: DisenCD provides an effective approach for cognitive diagnosis under limited exercise labels, representing one of the few attempts to focus on this problem. Extensive experiments on three real-world datasets demonstrate the effectiveness of the model.\n\tGroup-based disentanglement and limited-labeled alignment modules: The DisenCD model introduces two novel modules, group-based disentanglement, and limited-labeled alignment, which disentangle factors relevant to knowledge concepts and align them with the available limited labels. These modules help to overcome the problem of limited exercise labels and improve the accuracy of cognitive diagnosis.\n\tInterpretable: The DisenCD model achieves the best interpretability in all scenarios, especially in the 10% and 20% Q-matrix (label-scarce exercises) scenarios, where it significantly outperforms interpretable baselines in terms of Degree-of-Alignment (DOA). This demonstrates the superior interpretability of the model.\n\nWEAKNESSES:\nLimited Generalizability: Although the effectiveness of the DisenCD model is demonstrated on three real-world datasets, the model's generalizability to other datasets or samples still needs to be established. Further studies are, therefore, needed to establish the generalizability of DisenCD to more extensive and diverse datasets.\n\tLack of Explicit Discussion on Limitations: Although the article describes the proposed model, its training procedures, and experimental results, the limitations of the DisenCD model should be explicitly discussed. This may limit the reader's ability to interpret the results and implications of the model. A more explicit discussion of the model's limitations would have made the study more comprehensive.\n\nQUESTIONS:\n1.Can you please provide more details on the pre-filling algorithm used for missing Q-matrix in the experiments？\n\n2. Could you explain further the two novel modules, group-based disentanglement and limited-labeled alignment, that DisenCD uses to disentangle relevant knowledge concepts and align them with the available limited labels?\n\n3.Can you elaborate on the hyperparameter settings during the experiment?\n\nLIMITATIONS:\nNo, the authors did not provide any discussions for limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper presents an algorithm for cognitive diagnosis (CD) in limited data scenarios. CD aims at labeling questions to knowledge concepts.

As the CD process requires expert tagging, labelling questions to knowlege concepts in time intensive.
This paper addresses the labelling task from  three factors: student proficiency, exercise difficulty, and
exercise relevance.

STRENGTHS:
1. Knowledge concept (KC) labeling in limited data scenario
 2. Disentangled Representation Learning for interpretation
 3. Addresses correlated and independent KCs by a tree-like hierarchy

WEAKNESSES:
1. The authors utilized student interaction distribution to derive exercise relevance and knowledge concepts from exercise relevance. 
 
One limitation of utilizing student interaction is the cold-start problem of newly introduced exercises or a small number of student responses. 
A more robust way to model exercise relation is by considering the question text and student responses. See Pandey et al. [1] for reference.

2. The assumptions of inter-group and correlated KCs may be too strong.  
 
 References.
 1. Pandey S, Srivastava J. RKT: relation-aware self-attention for knowledge tracing. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management 2020 Oct 19 (pp. 1205-1214).

QUESTIONS:
Did the author validate (a subset of) the discovered KCs with domain experts?
Did the author evaluate how their model performs with small student responses to questions in V2?

LIMITATIONS:
Yes, addressed in the supplementary material.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on performing cognitive diagnosis with limited exercise labels. To address the enormous cost of labeling exercises, in this paper, the authors proposed Disentanglement based Cognitive Diagnosis (DisenCD). Specifically, they first used students’ practiced records to model student proficiency, exercise difficulty, and exercise label distribution; Then, group-based disentanglement and limited-labeled alignment modules were proposed to disentangle the factors relevant to concepts and align them with real limited labels. At the same time, a tree-like structure of concepts was proposed for group-based disentangling.\n\nSTRENGTHS:\nThe idea presented in this paper is novel and holds value in addressing the problem of limited exercise labels, as annotating them with domain experts incurs a substantial cost. Meanwhile, in the supplementary materials, it is highly significant for the authors to perform the Nemenyi Test on the experimental results.\n\nWEAKNESSES:\n1.\tFigure 1: The radar chart has some mistakes. For the diagnosis of knowledge concept k2, we can find that the proficiency of student u1 is higher than that of u2, but the answer logs given by the authors show that student u1 answered v1 correctly, answered v2 incorrectly, and student u2 answered v1 correctly.\n2.\tIn line 28, the word 'The' is written incorrectly.\n3.\tIn line 70: MIRT has been enhanced based on IRT and can provide an assessment of subjects or students from multiple perspectives.\n4.\tIn the experimental section: \n(1)\tIt is clear that the model proposed by the authors did not achieve the best results in all scenarios. \n(2)\tInsufficient preparation of experimental parts, i.e., lack of ablation experiments to verify the effectiveness of each part.\n\nQUESTIONS:\n1.\tIn Section 2.1, The authors believed that the TextCNN method proposed by NCDM+ contained errors in extracting knowledge concepts from the semantic information of the exercise text. Could the authors provide verification of this claim? Alternatively, could the authors compare their method with the TextCNN approach to assess its effectiveness in extracting knowledge concepts?\n2.\tIn Section 4.1, initially, the authors explained that the interaction matrix X contains significant information, including student proficiency. However, it is later mentioned that the response records of a student conceal his proficiency regarding the knowledge concepts. This raises the question of why such a contradiction exists.\n3.\tTo the best of my knowledge, most cognitive diagnosis (CD) [1][2][3] works typically exclude students with less than 15 answer logs to ensure the plausibility of the results. However, I am curious as to why the authors chose to employ three different criteria for removing students in the three selected datasets.\n4.\tRegarding the 10% Q matrix or the 20% Q matrix, what criteria do the authors employ to retain 10% or 20% of the original Q matrix?\n\n[1] Wang F, Liu Q, Chen E, et al. Neural cognitive diagnosis for intelligent education systems[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 6153-6161.\n[2] Gao W, Liu Q, Huang Z, et al. Rcd: Relation map driven cognitive diagnosis for intelligent education systems[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021: 501-510.\n[3] Ma H, Li M, Wu L, et al. Knowledge-Sensed Cognitive Diagnosis for Intelligent Education Platforms[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022: 1451-1460.\n\nLIMITATIONS:\n1.\tCurrently, in the field of cognitive diagnosis, many datasets lack a tree structure between knowledge concepts. Therefore, the method proposed in this paper may not be suitable for the majority of datasets, i.e., the ASSISTments2009 dataset[1], and the ASSISTments2012 dataset[2].\n2.\tAs can be seen from Table 2 in the experimental part and Table 2 in the supplementary materials, the method proposed in this paper is not well suited for datasets[1][2] with a large number of knowledge concepts. This would greatly limit the applicability of the method.\n\n[1] https://sites.google.com/site/assistmentsdata/home/assistment-2009-2010-data/skill-builder-data-2009-2010\n[2] https://sites.google.com/site/assistmentsdata/datasets/2012-13-\nschool-data-with-affect", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an innovative approach called DisenCD, aimed at enhancing the performance of cognitive diagnosis when exercise labels are limited. The proposed method incorporates two newly developed modules: the group-based disentanglement module and the limited-labeled alignment module. These modules effectively identify relevant concept factors and align them with existing labels. By leveraging limited exercise labels, the DisenCD method maximizes their potential for improved cognitive diagnosis.\n\nSTRENGTHS:\n1. The paper addresses an intriguing and relevant problem: cognitive diagnosis with limited exercise labels, which has practical implications.\n2. The paper effectively utilizes a tree-structure to disentangle concept-related factors and align them with existing labels, providing a valuable solution to the aforementioned problem.\n3. The extensive experiments conducted in this paper convincingly showcase the effectiveness of the proposed method, further strengthening its validity.\n\nWEAKNESSES:\n1. Although DisenCD aims to enhance interpretability in cognitive diagnosis, the model itself may lack interpretability. The inclusion of disentanglement and alignment modules could introduce additional complexity, making it challenging to comprehend the model's decision-making process.\n2. In Section 4.2, the authors claim that there is reduced independence among knowledge concept groups in high-level tree nodes. However, the absence of necessary experiments or references weakens the support for this argument.\n3. Section 5.1 lacks the necessary details regarding the implementation, which leaves gaps in understanding the practical aspects of applying the proposed method.\n\nQUESTIONS:\nSince the tree structure of knowledge concepts is very important for the proposed method, The details of the tree structure in each dataset should be provided.\n\nLIMITATIONS:\n1. DisenCD is a complex model that involves multiple modules and hyperparameters. This complexity may make it difficult to implement and tune, especially for practitioners who are not familiar with the underlying techniques.\n2. DisenCD may not be scalable to large datasets due to its complexity. This may limit its applicability in real-world scenarios where large-scale cognitive diagnosis is required.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to perform cognitive diagnosis. The method leverages notions of disentanglement representation learning to achieve better interpretability while avoiding sacrificing performance of predicting the students' answers. Experiments were conducted on three popular datasets and quantitative empirical evidence appears to support the above claim (no loss of predictive performance; improved interpretability).\n\nSTRENGTHS:\nThe idea of leveraging disentangled representation learning in the context of cognitive diagnosis seems new. \n\nThe execution of this idea is interesting as the authors made a few changes to solve the technical challenges of applying disentangled representation learning to modeling students' skills and predict their answer correctness.\n\nThe experiments cover three popular datasets and seem comprehensive, including numerical results comparing multiple baselines and various analyses.\n\nWEAKNESSES:\nMy main concern is on the notion of \"interpretability\". After reading the paper, I fail to understand what is \"interpretability\" in the authors' context. For disentangled representation learning in the vision domain, interpretability mostly means disentangling factors of variation, such that when you perform generation, you can manipulate the encoded latent vector (e.g., change a specific dimension of the vector that corresponds to one variation such as shape, size, viewpoint, etc.) to control the generation to be of a particular shape, size, viewpoint, and so on.\n\nThe authors seem to apply a similar notion of \"interpretability\" on the latent student knowledge/skill vector. However, I do not see any evidence or discussion on how such interpretability is manifested, e.g., what are the \"disentangled factors\" that the model learns? What do these factors mean and how do they connect to the knowledge concepts in the Q matrix? Do the model learn different disentanglement for different students? I believe an illustrative example would be extremely useful. Reporting only the DOA metric is not very convincing to me as a measure of \"interpretability\". \n\nBecause the paper makes a big claim on \"interpretability\" and the proposed methodology revolves around it, I am unsatisfied with the empirical evidence and analyses around it. I am open to raise my score if the authors can provide more in-depth discussion and definition around interpretability in their context, as well as illustrative examples to demonstrate the interpretability that their model learns.\n\n\n----\npost-rebuttal update: authors answered most of my questions and I have updated my scores accordingly.\n\nQUESTIONS:\nWhat do the authors mean by interpretability? Does it mean disentangling factors of variation on the students' latent vector? If so, what is being disentangled? Does the different dimension of the students' latent vector mean anything, or correspond to one or few concepts defined in the Q matrix?\n\nIn line 228, what's the rationale to use L2 loss? From my understanding, this loss aims to encourage sparsity, so why not using an L1 loss?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}