Disentangling Cognitive Diagnosis with Limited Exercise Labels
Authors: Xiangzhi Chen
Le Wu
Fei Liu
Lei Chen
Kun Zhang
Richang Hong
Meng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Intelligent Education System
Cognitive Diagnosis
Disentangled Representation Learning
Interpretability

Abstract: Cognitive diagnosis is an important task in intelligence education, which aims at measuring students’ proficiency in specific knowledge concepts. Given a fully labeled exercise-concept matrix, most existing models focused on mining students' response records for cognitive diagnosis. Despite their success, due to the huge cost of labeling exercises, a more practical scenario is that limited exercises are labeled with concepts. Performing cognitive diagnosis with limited exercise labels is under-explored and remains pretty much open.
 In this paper, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges of limited exercise labels. Specifically, we utilize students' response records to model student proficiency, exercise difficulty and exercise label distribution.  
 Then, we introduce two novel modules - group-based disentanglement and limited-labeled alignment modules - to disentangle the factors relevant to concepts and align them with real limited labels. 
 Particularly, we introduce the tree-like structure of concepts with negligible cost for group-based disentangling, as concepts of different levels exhibit different independence relationships.
Extensive experiments on widely used benchmarks demonstrate the superiority of our proposed model.

Disentangling Cognitive Diagnosis with Limited Exercise Labels
Xiangzhi Chen'
Le Wu 3,
Fei Liu'
Lei Chen?
Kun Zhang'
Richang Hong
Meng Wang
Hefei University of Technology Tsinghua University
3 Institute of Dataspace, Hefei Comprehensive National Science Center
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center {cxz. bfut, lewu. @gmall com feiliu@mail .ffut. edu. en {chenlei hfut, zhang1028kun, hongrc.hfut eric mengvang) com
Abstract
Cognitive diagnosis is an important task in intelligence education, which aims at measuring students' proficiency on specific knowledge concepts. Given a fully labeled exercise-concep matrix, most existing models focused on mining students' response records for cognitive diagnosis. Despite their success, due the huge cost of labeling exercises, more practical scenario is that limited exercises are labeled with concepts. Performing cognitive diagnosis with limited exercise labels is under-explored and remains pretty much open. In this paper, we propose Disen- tanglement based Cognitive Diagnosis (DCD) to address the challenges of limited exercise labels. Specifically, utilize students' response records to model student proficiency, exercise difficulty and exercise label distribution. Then, we introduce novel modules group-based disentanglement and limited-labeled alignment modules to disentangle the factors relevant to concepts and align them with real limited labels. Particularly, introduce the tree-like like structure of concepts with negligible cost for group-basee disentangling, as concepts of different levels ex- hibit different independence relationships. Extensive experiments widely used benchmarks demonstrate the superiority of our proposed model.
Introduction
In the field intelligent education systems, cognitive diagnosis (CD) [25] 26 is a fundamental and essential task, which aims at measuring students' proficiency specific knowledge concepts through the student performance prediction process. CD can serve various applications, such as computerized adaptive testing 2]. targeted training [20]. and exercise recommendation 24 14]. We show toy example of CD in Fig. (a). The exercise-concept matrix (known as Q-matrix) is labeled to indicate which concepts
tested in each exercise. For example, concepts ka and kş are tested in exercise dg.
The CD model diagnosed that student u2 has a higher proficiency on the two concepts than that of uj because U2 answered correctly the exercise while uı answered incorrectly. It's obvious that fully labeled Q-matrix plays an essential role interpretability (i.e., diagnostic report) for CD models.
Most existing CD models focus enhancing the mining process of response records to achieve better diagnosis results. These models typically use a fully labeled Q-matrix that has been annotated by domain experts to train their models. For example, the classical DINA model [8] assumes that students must master all knowledge concepts associated with an exercise to answer correctly,
"Corresponiing Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fuil-I abeled Q-matrix
Response Records
Diagnestic Report
Tree-like Structrre of Knowledge Concepts
vs
%
1 Level
Bordlee
Stukkts
2 Level
Cognitive Diagnosi
3 Leva
k
1
14
(b)
Figure 1: Example of cognitive diagnosis. (a) We show an example to demonstrate the CD process, using fully-labeled Q Q-matrix annotated by domain experts. (b) Tree-like structure of concepts is easily annotated with negligible cost as the number of concepts is far fewer than that of exercises.
except in cases where the student guesses correctly. Representative neural-based CD) models, such as NCDM 38 KaNCD [39] and KSCD 30,, explicitly model the explicit relationship between exercise and knowledge concepts to constrain student proficiency during diagnostic processes. Despite the success of these models, they share a common assumption that the knowledge concepts of each exercise are fully labeled. However, the annotation of exercises is labor-intensive task that requires the expertise of professionals. There are always many exercises involved in intelligent education systems, and the workload for annotation significantly increases with the number of exercises. Thus, a more practical scenario cognitive diagnosis with limited exercise labels.
Performing cognitive diagnosis using limited exercise labels is indeed an under-explored area in current research. When faced with the few labeled exercise, straightforward approach to prefill the Q-matrix in order to provide a complete Q-matrix as input to an existing CD model. One limitation of this method is that its effectiveness largely depends on the accuracy of the pre-filling algorithm. If the pre-filling rilliilingg algorithm fails to identify the knowledge concepts associated with each exercise correctly, the resulting filled Q-matrix may be inaccurate or incomplete, which can negatively impact the model's s diagnostic performance. Therefore, how to achieve interpretable diagnosis on specific knowledge concepts with limited exercise labels remains a challenge.
In this work, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges posed by limited exercise labels. Specifically, DCD is semi-supervised disentanglement method to model the students' proficiency with limited exercise labels. To achieve this, we utilize historical practice records to model student proficiency, exercise difficulty, and exercise relevance on each knowledge concept. Then, we introduce two novel modules the group-based disentanglement module and the limited-labeled alignment module to disentangle factors relevant to knowledge concepts and align them with real limited exercise labels. Particularly, we introduce the tree-like structure of concepts (shown in Fig. [T-) for group-based disentangling, as concepts of different levels exhibit different independence rel relationships. The knowledge concept tree reduces the workload required for annotating knowledge concepts, as there are far fewer concepts than exercises to annotate. These modules work together deal with the limited exercise labels and diagnose the student's s proficiency on each concept with interpretability. The main contributions of this work are summarized follows:
Our work represents one of the few attempts to focus on the problem of limited exercise labels cognitive diagnosis, which is common in real-world practice.
We propose a semi-supervised disentanglement approach to address the challenge of cogni- tive diagnosis with limited exercise labels.
We conduct extensive experiments under different few-labeled settings three real-world datasets, which demonstrates the effectiveness of our proposed DCD model.
2 Related Work
2.1
Cognitive Diagnosis in Intelligent Education Systems
The early CD works like IRT 10] and MIRT 335 focus on modeling students' answering process by predicting the probability of a student answering a question correctly, which utilizes latent factors as

---

the student's ' ability. IRT and MIRT are also known as Latent Factor Models (LFM). The obvious limitation of the above models is the lack of interpretability, i... the inability obtain an explicit multidimensional diagnostic report on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of exercises to diagnose students' proficiency in all knowledge concepts 88] 38, 6 47 21]. Representative NCDM [8] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works 10 35 However, researchers find that the free student proficiency vector learning paradigm kinoweedge-ooverage scenario and model relation among
in NCDM is not capable of tackling
knowledge concepts implicitly to address the problem [39 30].
Cognitive Diagnosis in Few-labeled Scenarios. Most existing CD models are unable to directly deal with the few-labeled scenarios, and assume that the exercises are fully labeled. NCDM+ (39] fills the missing knowledge concepts by TexCCNN [8] from the exercise's s text information, which regards it as a previous task for CD. Therefore, the performance of CD lies in the accuracy of pre-filled knowledge concepts. In this work, we focus on more challenging scenario of missing exereise
textual information and infer missing knowledge concepts from response records only.
2.2 Disentangled Representation Learning
Disentangled Representation Learning (DRL) is an important learning paradigm that aims to disen- tangle the underlying generative factors hidden in the observed data into the latent variables in the representation [1. One notable benefit attributed to DRL is its ability to extract latent factors that embody semantic meanings, thus enhancing the interpretability of machine learning models. Both VAE [9] and GAN [12] based DRL are widely adopted techniques. Considering that the training of VAE is more stable than that of GAN 17 45 in this work, we extend the VAE framework and focus on learning interpretable disentangled representations to help cognitive diagnosis.
VAE-based Disentangled Representation Learning. 3-VAE [13] modifies the Evidence Lower Bound (ELBO) of VAE 19 by adding a hyperparamter 3 on KL term and find that a larger / corresponds the better disentanglement of latent code. However, the increasing / would also lead to the worse reconstruction error dramatically 17 36 To analyze the trade-off between reconstruction accuracy and the quality of disentangled representation, both FactorVAE [17 and 8-- TCVAE [5] show that the success of 3-VAE in learning disentangled representations can be attributed penalizing the Total Correlation (TC) term. 3-TCVAE decomposes the expected KL term in ELBO into index-code MI term, TC term, and dimension-wiss KL term. The TC term would decrease the mutual information among factors and improve independence among latent factors, which results in better disentanglement. However, some researchers claim that the above unsupervised learning of disentangled representations is fundamentally impossible without inductive biases as sometimes the target dataset is not semantically clear and well-structured to be disentangled [27). Hence, more works focus on weak- saak-supervised approaches 28 37 and -upervied 29 16 A theoretical framework is provided to assist in analyzing the disentanglement guarantees by weak supervision methods (e.g. restricted labeling, match pairing, and rank pairing) [37. Some researchers validate that with little and imprecise supervision (e.g. manual labeling of factors) it is possible to reliably learn disentangled representations (29] In this work, we regard limited exercise labels as a semi-supervision to help disentangle knowledge concepts-related factors.
Disentangled Representation Learning for User Modeling. User modeling [22 aims to capture a number of attributes of each user, with the help of items, item features and/or user-item response matrix 43), etc. DRL has wide range of applications in user modeling to disentangle attributes.
For example, recommendation with several aspects of users' interests 23
31
46
40),
fair user
representation to disentangle sensitive attributes 7
3 Problem Formulation
Let {ui, u....., uuy)...uy... V= (ui U2... ......, J. and K. [Ki,, kg...., kgk denote the sets of students, exercises, and knowledge concepts, respectively, where N, M, and K represent the size of each set. The response records are denoted as X = Nxxj where X,, equals 1, or -1 representing that the student answered the exercise correctly, incorrectly, or did not answer the exercise, respectively. The relationship between exercises and knowledge concepts is represented by the Q-matrix Q {Qu)kxM Typically, each exercise is related to multiple knowledge concepts.
3

---

Qij equals I or O representing that exercise by is related to the knowledge concept k; or not. In this work, we explore the few-labeled situation, that is, there are only a few exercises that are labeled. Thus, we split exercise set V into the labeled set V1 and the unlabeled set V22
There are correlations and independence among different knowledge concepts. In our work, introduce the knowledge concept tree [41] to model their relations at different levels. By utilizing this tree structure, we can effectively capture knowledge concepts' interrelationships. For convenience, we transform the knowledge concept tree into a standard tree (the height of all subtrees of any node in the tree equal). For more details about knowledge concept tree, please see Appendix C Suppose we construct a standard tree G with L levels, wherein its leaf nodes are denoted by the set kx}.
K = {k1. kg...., ...,..,,, }. Each level of knowledge concept can be regarded as a grouping of the last level of knowledge concepts. The deeper the level, the finer the granularity of the grouping. Let G G denote grouping method of the i-th level knowledge concepts, where
G [Gi, G3.
i 1,2, ...,, L and | 2 G*I. Therefore, G¹ { K} and G [ {11). G3 [K2)..... Gk {kr}}.
Problem Definition. In an intelligent education system, given student set U, exercise set V V2V22 knowledge concepts K, response records X. knowledge concept tree G and incomplete Q-matrix, goal is to diagnose students' cognitive states on knowledge concept set with few-labeled exercises, and predict the scores of students doing exercises.
The Proposed Model
In cognitive diagnosis, it is crucial to determine whether student has the proficiency in knowledge concepts required to answer an exercise correctly. This becomes even more challenging when there are limited exercise labels available. To model students' proficiency with few-labeled exercises, we design disentanglement and alignment modules that can effectively utilize the weak supervision of a few labeled exercises.
The overall framework of DCD is illustrated in Fig. []] including input, encoder, disentanglement, alignment, and decoder. The input of DCD contains the student-exereiss interaction matrix X and the few-labeled matrix Q. For diagnosing students' cognitive states, we first utilize the encoder module to model three important components from X: student proficiency, exercise difficulty, and exercise relevance. Then, we leverage the group-based disentanglement module to separate the factors related to knowledge concepts from the three components mentioned above. Next, the limited-labeled alignment module associates each decoupled factor with a knowledge concept semantic with the missing matrix Q. Finally, the decoder module predicts the scores based the student proficiency, exercise difficulty, and exercise relevance. Overall, DCD provides an effective approach for cognitive diagnosis under limited exercise labels.
4.1 Encoder Module
The interaction matrix X provides valuable information regarding student proficiency, exereise difficulty, and exercise relevance from different perspectives. Specifically, 1) From a student-centric (row-wise) perspective, the response records of student reflect his proficiency on all knowledge concepts. 2) From an exercise-centric (column-wise) perspective, the response records of an exercise reflect exercise difficulty and exercise relevance on all knowledge concepts. On the one hand, to effectively model from two perspectives, we design the student proficiency encoder, exercise difficulty encoder, and exercise relevance encoder modules. On the other hand, following existing cognitive models, an interaction function (corresponding to decoder) to predict the answering result with student traits and exercise traits as input (corresponding to our encoders), which ensures the interpretability of student proficiency representation. Traditional single encoder is not applicable for cognitive diagnosis.
Student Encoder.
The student encoder module is intended to infer student's 's proficiency on each knowledge concept from X, which models true posterior distribution p(z.. xx,) by constructing
an inference network fo_ (x.) corresponding to approximate posterior distribution do, (z.,l (x))
parameterized by &u. We assume student's proficiency on all knowledge concepts z, follows a multivariate standard Gaussian distribution prior, i.c., p(z.) I), The approximate posterior

---

Input
Encuder Module (Section 4.1)
Group-hased Disentanglement Module (Section 4.2)
Decodar (Section 1.4)
Interaction matrix
Student Prufitiency Encoder
Proficiency
Disestanglezeett loss
Exunises
Concept
[*Jerel (Root (Rvot leval)
Disentangling at
Encader
Stakktt
2dd Level
Py Erp
Tevel (Leaf level)
Disentangling at Leaf level
Main loss
Exercise Difficultt Encodor
Ditticulty
Factors
BCE
0
Limited- Seiltedd-labbe Labrled lignment M oddule (Section 4.3)
-:Unnonseerrd
Encoder
L
imiited Labrled matrix
A Align- unlan-edl loss
Execcises
My
by
Margin
Exercise Relevance Encoder
Canaye
Relevance
A Aiggl- loss
0
MSE
Encoder
-1
-1
0: Imelevant
1: Relevant
-1: Unlabeled
Labeled
Unlabeled
Figure 2: The overall framework of the proposed DCD. We first encode student proficiency, exercise difficulty, and relevance vectors from students' historical response records. Then, devise two novel modules (i.e., the group-based disentanglement and the limited-labeled alignment modules) to disentangle factors relevant to knowledge concepts and align them with real limited exercise labels. Finally, the decoder module predicts the scores based on the student proficiency, exercise difficulty, and exercise relevance.
for K-dimensional latent code zu R.K is expressed as follows:
Go. (z xx)
II
k=1
where n4, E R and a. RRK index selection operation.
==0 generated from student encoder So., (xu). The [+] indicates the
Exercise Encoders. We design two encoders at exercise side: exercise difficulty and exercise relevance encoders. The two encoders aim to infer exercise's difficulty and exercise's relevance on each knowledge concept from X, respectively. For exercise difficulty encoder, we assume exercise difficulty on all knowledge concepts z follows a multivariate standard Gaussian distribution prior similar student encoder, i.e., p(z.) ~ (O,
We assume exercise's relevance on all knowledge concepts z follows multivariate Bernoulli prior, iec. p(z)) Bernoulli(p) p).. where p a hyperparameter. The exercise relevance encoder models true posterior distribution ((,) by constructing an inference network før (xx) corresponding to approximate posterior distribution qaz (x)), parameterized by o". The approximate posterior for K-dimensional latent code z {00, 1] is expressed as:
K
goz (x,
II [errnoull((2] A=1
(2)
where μb [0, 11x generated from exercise encoder for (x.).
4.2
Group-based Disentanglement Module
The representations obtained by the encoder module do relates the knowledge concepts, which makes difficult to perform an accurate cognitive diagnosis. To address this issue, we assume the entire response records as evolving from three groups of generative factors: student proficiency factors on each knowledge concept, exercise difficulty factors on each knowledge concept, and exercise relevance factors on each knowledge concept. We desire disentangle these factors following existing DRL methods.
5

---

Inspired by the outstanding performance of 3-TCVAE [ in disentanglement, we introduce the structure of 3-TCVAE to obtain a better trade-off between the reconstruction accuracy and the quality of disentangled representation. We find the index-code term of 3-TCVAE plays side effect on performance and drop it during optimization like [32 77. Take student proficiency factors as an example, the eventual Totoal Correlation (TC) constraint on Zu can be expressed as follows:
K
Ld(zu) = DKL(q(Zu)l II (z.
(3)
where g(z.) Σ i=1 g(Z,)) Xx. )xx,.)) is the aggregated posterior of Zu. The TC term would decrease the mutual information among latent factors and improve independence among latent factors. The dimension-wise KL constraint Zu can be written follows:
K
Cp(Zu) = Σ (kilalzuli)lllaaa
where p(z,[i)) [i]) follows the prior standard Gaussian distribution. The dimension-wise KL term prevents each latent variable from deviating too far from specified priors.
However, some CD works argue that the knowledge concepts are not independent and employ relation among knowledge concepts to improve CD (3). The direct independence constraint for each knowledge concept may be not applicable for cognitive diagnosis, which is also verified in our experiment. To address the challenge, we propose to utilize the casy-labeled knowledge concept tree to capture underlying independence among knowledge concepts. We assume there is less independence among groups of knowledge concepts grouped by their parent nodes in the knowledge concept tree. Corresponding to the real world, knowledge concepts in the same chapter would be related more than knowledge concepts in different chapters. Therefore, each level concept in the tree is a kind of grouping method for the last level concepts. The deeper the level, the finer the granularity of the grouping. Considering the grouping methods at the i-th level, the Eq. (3) can be rewritten as follows:
G
(2().) D (g(z)) )ll II (( (GH)))
where G3 is the knowledge concepts that belong to j-th group according to the grouping method of i-th level in the tree. The Eq. (5) allows dependence in intra-group knowledge concepts and maintains independence among inter-group knowledge concepts.
4.3 Libited-Labeled Alignment Module
After applying the group based disentanglement module, it is important to correspond each latent factor with an actual knowledge concept to ensure the interpretability. However, there are only few labeled exercises with knowledge concepts, making it challenging to perform the accurate alignment. Inspired by semi-supervised DRI, to address this issue, we devised an alignment module that aligns both few labeled and numerous unlabeled exercises separately. For convenience, q E {0, 13 to denote j-th column of matrix Q corresponding to exercise ey that belongs to V1.
Few Labeled Exercises. For exercise vj E Vi, we utilize the Mean Square Error (MSE) loss to constraint similarity between q and p which can be defined as follows:
C1 = Σ CE
| - 1., 11
(6)
where Vi means labeled exercise set. For semi-supervised DRL in the vision field, it's usually enough with single alignment module for a few labeled samples. However, for cognitive diagnosis, the annotation vector (i.e., the column of Q-matrix) of exercise is very sparse with only one or two knowledge concepts per exercise. The challenge is that utilizing few labeled exercises to do aligmment is too hard to keep such sparsity in relevance representation for unlabeled exercises. To this end, the alignment of unlabeled exercises is necessary.
Numerous Unlabeled Exercises. For an exercise ny E V2, we have no idea any knowledge concept of it. But we know there are only a few knowledge concepts of each exercise. The feature of sparsity

---

in Ha; would help infer missing knowledge concepts, We just need to guarantee that there are only few elements to be one and other elements to be zero in pezz In this work, we employ a margin loss and an L2 loss to achieve this target, which can be expressed as follows:
Lut= Σ (Aimar(O, m (1he [maa Mo [mar@( + 1)))) d2 X2 RyEVs
(7)
where m is the margin, A1 is the hyperparameter to control margin loss, and X2 is the hyperparameter control L2 loss. We use pt [mar@k] and Jr [maa #k] ki] to denote the k-th largest element of μe and the top & largest elements of μr,, respectively. In Eq. (7). we first sort Mi; by descending order and then split the sorted one into three parts of length d1 d2, K dj d2. We argue that knowledge concepts the last part are the least likely to be the ground-truth knowledge concepts of exercise Uj and knowledge concepts in the first part are the most likely to be the ground-truti knowledge concepts of exercise Vy. Therefore, we adopt a margin loss between knowledge concepts in the first part and knowledge concepts in the last part, which requires that all values in the first part (i.e., My, [max##di]) least m greater than the largest value in the last one (i.e., Mey [mar@(() (d1 + d2 + 1)]). For the middle part, we do not constrain anything it. Intuitively, it's reasonable that adopt an L1 loss to constraint the value in the last part (i.e., Ne, [min# (K - d1 - dz)]) to be In our initial attempt, we first directly try to use LI loss for numerous unlabeled exercises, and we find it does not show competing performance (detailed in [Appendix A.3). We speculate a possible reason is as follows: with LI loss, after model initialization, it is very hard to revise these incorrectly labeled exercises. In contrast, L2 loss sensitive to outliers, and would have larger loss when the corresponding knowledge is incorrectly predicted.
4.4 Decoder Module
After aligning the exercise relevance representation with the real knowledge concepts, the decoder module predicts scores of students doing exercises, which could described as triplet set {(u,, By, X1y Xx # - The object function can be described follows:
Cm -Egou (Z. (2ulxx)). (x. (x.) [log p( Zu 2 x!
Σ
BCE(o((Zu, z z X1z).
(8)
where BCF(.)) .) is the binary cross entropy loss function, @ and a(.) denote inner product operation and sigmoid function, respectively.
the interaction function (i.e., the decoder) keeps the monotonicity assumption 39, which guarantees the interpretability of student proficiency represen- tation (i.e., the higher the value of the latent factor means the better proficiency on the corresponding knowledge concept, detailed in Apppnddix A.4). In addition, the element- wise operation in the decoder module propagates the alignment (1.e., each latent factor corresponds to each real knowledge concept) on exercise relevance representation to both student proficiency representation and exercise difficulty representation, which is also the same existing works.
Similar to existing CD works 38 39
Optimization. We summarized all object functions in all modules and combine them for final optimization. We set different hyperparameters to balance each loss function. The final object function can be summarized as follow:
I.
arg min e= (ow.dd
C Lm + aali + Cut
Σ (
Σ
Cx(2))
Σ
Cz)
(9)
ZE (M m
ZE 2E ad..
where θ =
[0,, dd [.] is the parameter set in the whole model, is the hyperparameter for alignment of labeled exercises, and f denotes the weight for disentanglement term corresponding to the i-th level in the knowledge concept tree.
7

---

Experiments
5.1 Experimental Settings
Datasets. Our experiments are conducted on three real-world datasets, i.c., Matma Junyi 4 and NIPS2020EC [42], all of which contain knowledge concepts of the tree structure. The statistics of these datasets are summarized in [ We adopt five-fold valsddaaio to avoid randomness. The details about datasets and implementation are depicted in the Appendix B
Table I: The statistics three datasets.
#concepts
#leaf concepts exercise
#right records:
dataset
#students
#exercises
sparsity
at different level
#error records
Matmat
7,067
1,847
[1,5,11|
1.00
5.50
97.7%
Junyi
8,852
720
[1,8,99]
1.00
2.16
87.4%
NIPS2020EC
11,857
6,509
[1,4,29.64]
1.05
2.00
99.1%
Metrics. To evaluate the effectiveness of DCD in terms of score prediction and interpretability. we adopt three widely used prediction metries: AUC RI, ACC, RMSE, and one interpretability metric: Degree of Agreement (DOA) [38] DOA is the most commonly used interpretability metric in the CD, which measures the degree of agreement between cognitive results and response records.
Baselines. The baselines including the classical data mining model Probabilistic Matrix Factorization (PMF) [33]. the latent factor models IRT [10), MIRT [35), and the representative CD models DINA (8), NCDM 38) KaNCD [39] and KSCD 30 In addition, we also explore a simplified version of our method. Specifically, we remove the exercise difficulty module and margin loss, and focus solely on the fine-grained disentanglement module. This simplified version serves as a baseline, which we refer as 3-TCVAE in the following analysis.
Existing non-interpretable models such as PMF, IRT, and MIRT cannot obtain a student's proficiency for each knowledge concept (38). Consequently, DOA cannot be calculated for these models. Interpretable models such as DINA, NCDM, KaNCD, and KSCD cannot directly apply to few- labeled scenarios. To ensure fairness in comparison, propose pre-filling algorithm (detailed in Appendix B.2) for the missing Q-matrix before applying these models.
5.2 Performance Comparison
To evaluate the effectiveness of our proposed DCD model in few-labeled scenarios, we have conducted experiments on datasets with 10% Q-matirx (few-labeled exercises), 20% Q-matirx (few-labeled exercises) and 100% Q-matirx (fully-labeled exereises). The detailed performance comparison on three datasets is displayed in Table
Prediction Comparison. 1) Our proposed model consistently outperforms all interpretable baseline models across the different scenarios, demonstrating its effectiveness. DINA, which only considers binary space, performs the worst. Both KSCD and KaNCD implicitly model relation among knowl- edge concepts and has a better performance than NCDM. 2) However, all non-interpretable models achieve the best prediction metrics over interpretable models on Junyi and NIPS2020EC datasets except Matmat dataset. This may be because there are far fewer knowledge concepts in the Matmat dataset than the other two datasets. To meet the interpretability of larger knowledge concept space, interpretable models lose partial prediction accuracy. This explanation is supported by the fact that the prediction results of interpretable models show a noticeable improvement in the few-labeled scenario compared to the fully labeled scenario, indicating that the trade-off between interpretability and accuracy is more significant in a larger knowledge concept space.
Interpretability Comparison. 1) Table 2 demonstrates that our method achieves the best inter- pretability in all scenarios. Especially, for the 10% and 20% Q-matrix (few-labeled scenarios), our model significantly outperforms interpretable baselines in terms of DOA, demonstrating its superior interpretability this scenario. 2) In few-labeled scenarios, KaNCD and KSCD show remarkable improvement in DOA compared to DINA and NCDM. This is because DINA and NCDM can only
"Our code is available at htttps://github. http:s//ggshuub..co/k/erraaass/ www..////oo/eeeeaa///o//g
8

---

Table 2: Comparison in both fully-labeled and few-labeled scenarios. We divide models into non- interpretable and interpretable models. The best scores are in bold for two kinds of models. Noted that bon-interpretable models keep same results in any scenario as they do not utilize Q-matrix.
Q-maitix (iully laheled)
20% Q-matrix (few- Labeleb
Q Q-matiix (lew laheled) ACC
Model
AUCT
ACC1
RMSF) DOAT
AUC
ACC1
RMSEL
DOAT
AUCT
RMSEL DOA
Matmat
PMF
0.759
0.857
0.334
0.759
0.857
0.334
0.759
0.857
0.334
0.747
0.339
MIRT
0.750
0,853
0.337
0,750
0,853
0,337
0.750
0,853
0.337
DINA
0.696
0.816
0.368
0.828
0.698
0.774
0.380
0.766
0.700
0.770
0.372
0.735
NCDM
0.742
0.839
0.348
0.850
0.738
0.837
0,350
0.781
0.740
0.846
0.346
0.738
8-TVVAE
0.337
0.847
0.730
0.733
KSCD
0.753
0.824
0.348
0.833
0.745
0,828
0,348
0.789
0,744
0.838
0.345
0.769
KaNCD
0.760
0.857
0.335
0.860
0.752
0.852
0.338
0.800
0.751
0.852
0.338
0.783
DCD
0.763
0.857
0.334
0.861
0.767
0.857
0.819
0.764
0.855
0.334
0.796
Junyi
PMF
0.817
0.776
0.394
0.817
0,776
0.394
0.817
0.776
0.394
IRT
0.822
0.779
0.391
0.822
0.779
0.391
0.822
0.779
0.391
0.392
0.777
0.392
DINA
0.737
0.684
0.462
0.862
0.748
0.658
0.459
0.638
0.754
0.663
0.456
0.599
NCDM
0.760
0.715
0.445
0.851
0.791
0.743
0.418
0.632
0.804
0.753
0.409
0.594
8-CCAE
0.770
0.732
0.423
0.827
0.803
0.764
0.406
0.699
0.805
0,763
0.407
0.673
0,804
0.754
0.760
0.405
KaNCD
0.775
0.754
0.422
0.835
0,807
0.767
0.400
0.719
0.815
0.773
0,398
0.691
DCD
0.787
0.754
0.414
0.873
0.811
0.768
0.400
0.733
0.814
0.771
0.397
0.717
NIPS2020EC
0.407
0.760
0.407
IRT
0.819
0.762
0.402
0.819
0.762
0.402
0.819
0.762
0.402
MIRT
0.822
0.765
0.400
0.822
0.765
0.400
0.822
0.765
0.400
DINA
0.764
0,705
0.451
0.856
0,767
0.687
0.453
0.740
0.766
0.684
0.453
0.693
NCDM
0.795
0.730
0.423
0.853
0.803
0,751
0,411
0.736
0.805
0.750
0.411
0.687
8-TCVAE
0.792
0,744
0.417
0.854
0.797
0.740
0,416
0.781
0.801
0.745
0.413
0.774
KSCD
0.798
0.716
0.421
0.830
0.809
0.754
0,410
0.787
0.809
0.755
0.410
0.777
KaNCD
0.797
0.423
0.856
0.812
0.760
0,404
0.761
DCD
0.501
0.752
0.415
0.861
0.812
0.761
0.405
0.793
0.813
0.762
0.404
0,786
diagnose knowledge concepts labeled in the training set. When incorrect knowledge concepts are filled in, KaNCD and KSCD are less affected, while DINA and NCDM cannot accurately diagnose the true knowledge concepts. In contrast, model avoids this issue by modeling the distribution of students' proficiency overall knowledge concepts.
5.3 Disentanglement Analysis
To investigate the effectiveness of disentanglement, we conducted an experiment where we compared our method at different disentanglement weights ,3 in the few-labeled scenario. Fig, (a-d) illustrates the results, the x-axis shows the value of from 0 to 2, which corresponds to the strength of disentanglement, The AUC is displayed on the left y-axis and the DOA is displayed on the right y-axis. The AUC()) and (OA(i) denote the results by disentanglement only according to i-th level of knowledge concept tree. There are some observations from Fig. 3 1). The disentanglement according to the last level of tree (i.e., the most fine-grained disentanglement) can not have an improvement in terms of DOA the rise of B3. The phenomenon is consistent with the assumption that knowledge concepts are correlated. Despite there being an improvement in AUC, we assume it's the result of interpretability losses, as mentioned in the prediction comparison in section 5.2] 2) The coarse-grand grained disentanglement could simultaneously improve the AUC and DOA compared to no disentanglement. We observe that the performance increased initially with the implementation of disentanglement and then decreased as the strength of disentanglement became excessive, leading to counterproduccivityy.
5.4 Alignment Analysis
To evaluate the effectiveness of the limited-labeled alignment module, we conducted an experiment with different hyperparameters on Junyi dataset with 10% Q-matrix, namely a, X1, X2 and d2, as shown in Fig. (c-h). Due to the number of leaf concepts per exercise being close to I for all datasets,

---

Lint wet 10% Monie
0.2
Lund with 20% Mariic
0.7
NIPS2200F 1096-
NPSPIROO C with 20% Mentt
0.80
07155
0775
0,502
0.811
677
6773
DOON
1005
07725
01000
ALV3
0.808
AUGI E0OM22
07
A0002
0.72
0716
0100
DICAT
Al30221
0:78
0.000
DOM6
ALX13
0.005
0,805
DIO412
A776
08
0.1
[c]
Lural with 10% () Merie
Juryt mith
Jurat wet 10% Mente
Juny with 10% Nonts
0710
0m0
071
0715
0,01
00715
0.11
0275
0m
07
07165
0.02
0m
0/17
0712
0.8
01
05
009
5:01
(125
0.5
005
0.01
25
2
(g)
Figure 3: AUC and DOA results of DCD with Hyper-parameten in the group-based disentanglement (a-d) and limited-labeled alignment (e-h) modules.
we set the d1 as the default value. The default margin is 0.5. There are some observations: 1) is the weight of the alignment loss corresponding to labeled exercises. Fig. 3 (e) shows that the performance of our model initially improves and eventually stabilizes as a increases. 2) is the weight of margin loss for unlabeled exercises. As illustrated in Fig- 3 (f), the employment of margin loss is effective according to results from zero to non zon-zero of λ1 The excessive d₁ would decrease the performance. 3) A2 the weight of L2 loss of unlabeled exercises exercise relevance representation. We can observe that the employment of the L2 loss plays an essential role from Fig. (g). 4) We can observe that the employment of de is effective from Fig. (h). As d2 increases from to 8, the performance increases first and then drops. Overall, the limited-labeled alignment module has positive effect performance, and the hyperparameters of the alignment module need to be certainly adjusted.
5.5 Ablation Study
As shown in Table 3 to prove the effectiveness of our proposed disentanglement and aligmment modules, we conduct the ablation study on Junyi dataset under the 10% Q-matrix scenario. The margin Loss in alignment module ensures few elements to be one in exercise relevance repre- sentation, which achieves module ensures the sparsity of exercise relevance representation, which achieves an improvement of 13.48% in terms of DOA. It proves that utilizing few labeled exercises do alignment is too hard to keep such sparsity in relevance representation for unlabeled exercises. The disentanglement module alone achieves 0.56% terms of DOA. Overall, both the proposed disentanglement and alignment modules achieve an improvement in terms of prediction and interpretability metrics.
Table 3: Ablation study of DCD on Junyi.
Model
AUC1
10%
ACC1
(-maiix/Jow-lakkk5)
RMSEJ
DOAT
DCD w/o Alignment- Margin
0.8140
0.7692
0.3976
0.7136
DCD w/o Alignment 12 loss
0.8101
0.7681
0.3977
0.6322
Disentanglemeet
0.8130
DCD
0.8143
0,7713
0.3965
0.7174
improvement of 0.53% in terms of DOA. The 1.2 loss in alignment
6 Conclusion
In this paper, we present a novel approach, called DCD, to tackle the interpretability problem of cognitive diagnosis with limited exercise labels. Inspired by semi-supervised DRL, we learn disentangled representations and align them with real limited labels by group-based disentanglement and limited-labeled alignment. Extensive experiments on widely used benchmarks demonstrate the superiority our proposed method in few-labeled scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by grants from the National Key Research and Development Program of China (Grant No. 2021ZD01 I 1822, the National Natural Science Foundation of China (Grant No. 61972125, 72188101, 61932009, 62376086, and U22A2094).
References
[1] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 35(8):1798-1888 2013.
[2] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In ICDM, pages 42-51. IEEE, 2020.
[3] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Putlern recognition, 30(7):1 145-1159 1997.
[4] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In EDM, pages 532-535, 2015.
[5] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In NeurlPS, pages 2615-2625,
2018. 2018.
[6] Song Cheng, Qi Liu, Enhong Chen, Zai Huang, Zhenya Huang, Yiying Chen, Haiping Ma, and Guoping Hu. Dirt: Deep learning enhanced item response theory for cognitive diagnosis. In CIKM, pages 2397-2400, 2019.
[7] Elliot Creager, David Madras, Jörn-Herrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In ICML, pages 1436- 1445, 2019.
[8] Jimmy De La Torre. Dina model and parameter estimation: didactic. Journal of educational and behavioral statistics, 34(1):115-13)) 2009.
[9] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. JMLR, 7:1-30, 2006.
[10] Susan E Embretson and Steven P Reise. Item response theory. Psychology Press, 2013.
[11] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma. Shijin Wang, and Yu Su. Red: Relation map driven cognitive diagnosis for intelligent education systems. In SIGIR, pages 501-510, 2021.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pages 2672-2680, 2014.
[13] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[14] Zhenya Huang. Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In CIKM, pages 1261-1270, 2019.
[15] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 16Xiv:1611.01144 2016.
[16] Ilyes Khemakhem, Diederik P. Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. In ICAIS, 2019.
[17] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML., pages 2649-2658. 2018.

---

[18] Yoon Kim. Convolutional neural networks for sentence classification. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, EMNLP, pages 1746-1751, ACL, 2014.
[19] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In JCLR, 2014.
[20] George D Kuh, Jillian Kinzie, Jennifer A Buckley, Brian K Bridges, and John C Hayek. Piecing together the student success puzzle: Research, propositions, and recommendations: ASHE higher education report, volume 116. John Wiley & Sons, 2011.
[21] Jiatong Li, Fei Wang, Qi Liu, Mengxiao Zhu, Wei Huang, Zhenya Huang, Enhong Chen, Yu Su, and Shijin Wang. Hieredf: A bayesian network-bass hierarchical cognitive diagnosis framework. In SIGKDD, pages
2022.
[22] Sheng Li and Handong Zhao. A survey on representation learning for user modeling. In IJCAI, pages 4997-5003, 2020.
[23] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoen- coders for collaborative filtering. In WWW, pages 689-698, 2018.
[24] Fei Liu, Xuegang Hu, Shuochen Liu, Chenyang Bu, and Le Wu. Meta multi-agent exereise recommendation: game application perspective. In SIGKDD, page 1441-1452, New York, NY, USA, 2023.
[25] Qi Liu. Towards generation of cognitive diagnosis. In IJCAI, pages
2021.
[26] Yingjie Liu, Tiancheng Zhang, Xuecen Wang, Ge Yu, and Tao Li. New development of cognitive diagnosis models. FCS, 17(1):171604, 2023.
[27] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, pages 4114-4124, 2019.
[28] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In ICML, pages 6348-6359, 2020.
[29] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In ICLR, 2020.
[30] Haiping Ma, Manwei Li, Le Wu, Haifeng Zhang, Yunbo Cao, Xingyi Zhang, and Xuemin Zhao. Knowledge-sensee cognitive diagnosis for intelligent education platforms. In CIKM, pages 1451-1460, 2022.
[31] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. In NeurlPS, pages 5712-5723, 2019.
[32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. In ICLR, 2016.
[33] Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. In NeurIPS, pages 1257-1264, 2007.
[34] Sungho Park, Dohyung Kim, Sunhee Hwang, and Hyeran Byun. Readme: Representation learning by fairness-aware disentangling method. arXiv, 2020.
[35] Mark D Reckase and Mark D Reckase. Multidimensional item response theory models. Springer, 2009.
[36] Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In ICMI., pages 8655-8664, 2020.
[37] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In ICLR, 2020.
12

---

[38] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In AAAI, pages 6153-6161, 2020.
[39] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuraled: A general framework for cognitive diagnosis. TKDE, 2022.
[40] Xin Wang, Hong Chen, and Wenwu Zhu. Multimodal disentangled representation for recom- mendation. In ICME, pages 1-6, 2021.
[41] Xinping Wang, Caidie Huang, Jinfang Cai, and Liangyu Chen. Using knowledge concept aggregation towards accurate cognitive diagnosis. In CIKM, pages 2010-2019, 2021.
[42] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmnnna Cameron, Yordan Zaykov, José Miguel Hernández-lLobato, Richard Turner, Richard G. Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Instructions and guide for diagnostic questions: The neurips 2020 education challenge, 2021.
[43] Le Wu, Xiangnan He, Xiang Wang. Kun Zhang, and Meng Wang. A survey on accuracy-oriented neural recommendation: From collaborative filtering to information-rich recommendation. TKDE, 35(5):4425-44455 2023.
[44] Le Wu, Peijie Sun, Yanjie Fu, Richang Hong, Xiting Wang, and Meng Wang. A neural influence diffusion model for social recommendation. In SIGIR, pages 235-244, 2019.
[45] Hengshi Yu and Joshua Welch. Michigan: sampling from disentangled representations of single-cell data using generative adversarial networks. Genome biology, 22(1):1-22 2021.
[46] Yin Zhang, Ziwei Zhu, Yun He, and James Caverlee. Content-collaboortiiv disentanglement representation learning for enhanced recommendation. In RecSys, pages 43-52, 2020.
[47] Yuqiang Zhou, Qi Liu, Jinze Wu, Fei Wang, Zhenya Huang, Wei Tong, Hui Xiong, Enhong Chen, and Jianhui Ma. Modeling context-aware features for cognitive diagnosis in student learning. In SIGKDD, pages 2420-2428, 2021.
13

---

A Additional Experiments
A.1 Comparison on Inferring Missing Knowledge Concepts
To evaluate the degree of alignment from exercises' relevance, we adopt Hit Ratio (HR) 44 to measure whether the top-k ranked elements in exercise relevance representation hit ground-truth knowledge concept. The algorithm about filling missing Q-matrix for baselines (detailed in Appendixx B.2 is compared with our exercise relevance encoder. The filling results of HR @2 is illustrated in Table We can observe that: 1) It's rational that inferring missing knowledge concepts according to response records of exercises. The filling algorithm and our method show obvious improvement than randomly assigning knowledge concepts for unlabeled exercises. 2) Our method demonstrates the best HR @2 on Matmat dataset, but shows a worse performance on other datasets. This because the number of knowledge concepts in Matmat has a smaller scale than other datasets and our model has better inferring performance in small scale knowledge concepts. Despite the worse filling results than filling algorithm for baselines, our method still shows the best performance in prediction metrics and interpretable metrics, which demonstrate the robustness of the proposed DCD approach. This may be attributed to the inference of student proficiency of our student encoder module.
Table 4: HR @ 2 comparison on inferring missing knowledge concepts
Matmat
Junyi
NIPS2020EC
20% Q-matrix
10% Q-matrix
20% Q-matrix
10% Q-matrix
20% Q-matrix
10% Q-matrix
Random
0.1818
0.1818
0.0513
0.0513
0.0313
0.0313
Filling Algorithm
0.6876
0.6006
0.2483
0.2382
0.5864
0.4548
DCD
0.9292
0.8693
0.2892
0.1898
0.4943
0.3849
A.2 Experiments under other few-labeled scenarios
We conduct experiments evaluating other missing ratios the Q-matrix on the NIPS2020EC dataset. In Table@of the main body, illustrated the results under 100%, 20% and 10% settings. Here we additionally add experiments on 50%, 30%, and 5% Q-matrix scenarios, which are illustrated in 5 The experimental results demonstrate DCD outperforms all the interpretable baseline models in these few-labeled scenarios. Besides, as the preserved ratio of the Q-matrix decreases (from 50% to 5%), our DCD demonstrates a greater improvement (from 0.86% to 1 1.43%) compared to the second-ranked model in terms of DOA. This shows the advantage of DCD in label-scarce scenarios.
Table 5: Comparison in other lew-labeled scenarios on NIPS2020EC dataset. We divide models into non-interpretable and interpretable models. The best scores are in bold for two kinds of models. Noted that interpetreee models keep same results in any scenario as they do not utilize Q-matrix.
50% Q-matrix labeld)
30% Q-matrx Laweled)led) ACC1
3% Q-matri (lew-labele labeled))
Model
AUCT
ACCT
RMSEL DOAT
AUC
RMSEL
DOAT
AUCT
ACC
RMSEL DOA
NIPS2020EC
PMF
0.814
0.760
0.407
0.814
0.760
0.407
0.814
0.760
0.407
IRT
0.819
0.762
0.402
0,819
0.762
0.402
0.819
0.762
0.402
MIRT
0.822
0.765
0.400
0.822
0.765
0.400
0.822
0.765
0,400
DINA
0.766
0.692
0.456
0.807
0.766
0.685
0.455
0.769
0.765
0.675
0.454
0.649
NCDM
0.799
0.415
0.802
0.743
0.765
0.800
0.742
0.644
8-TCVAE
0.794
0.746
0.416
0.808
0.795
0.748
0.412
0.790
0.801
0.747
0.415
0.770
KSCD
0.800
0.744
0.417
0.800
0.803
0.750
0.413
0.791
0.805
0.752
0.411
0.769
KaNCD
0.807
0.754
0.411
0.810
0.809
0.759
0.406
0.792
0.812
0.761
0.404
0.752
DCD
0.809
0.759
0.407
0.817
0.811
0.760
0.405
0.800)
0.814
0.763
0.403
0.781
A.3
Comparison between L1 and L2 loss for unlabeled exercises in alignment module
In section 4.3 we finally adopt the I.2 loss to constraint the exercise relevance representation sparsity of unlabeled exercises. Intuitively. it's reasonable that adopt an L1 loss to constraint the value in the last part (i.e., pby [min (K di - dz) )} to be O. As shown in 6 in our initial attempt, we first directly try to use Ll loss for numerous unlabeled exercises, and we find does not show competing performance. We speculate possible reason is as follows: with L1 loss, after model initialization, it very hard revise these incorrectly labeled exercises. contrast, 1.2 loss are sensitive to outliers, and would have larger loss when the corresponding knowledge is incorrectly inferred.
14

---

Table 6: Comparison between L1 and L2 loss of alignment in 10% Q-matrix scenario on three
datasets.
Maimat
JunYi
NIPS2020EC
Model
AUCT
ACC:
RMSFL
DOAT
AUCT
ACC
RMSEL
DOAT
AUCT
ACC1
RMSE
DOA1
DCD with
0.747
0.851
0.340
0.777
0.811
0.770
0.398
0.634
0.810
0.758
0.406
0.722
DCD with L2
0.764
0,855
0.334
0.796
0.814
0.771
0.397
0.717
0.813
0.762
0.404
0,786
A.4
Monotonicity Analysis
The monotonicity assumption 38 in CD works means that the probability of correct re- sponse to the exercise monotonically increas- ing at any dimension of the student's s knowledge proficiency, which guarantees the interpretabil- ity of student proficiency representation (i.e., the higher the value of the latent factor means the the corresponding knowl-
coardaa cenedter that mrok concagr
Aueragge prodlcttd probactee anasa fng questiinns con coreecty
better proficiency edge concept). Given a student proficiency rep- resentation, each dimension in the representa- tion corresponds to the student's proficiency level specific knowledge concept. When we only increase the value of a single dimen- sion in the student representation (improving the student's proficiency on specific knowl- edge concept), while keeping the exercise diffi- culty representation and relevance representation unchanged, the student's probability of answering questions related that knowledge concept correctly will increase, while the probability of answer- ing questions unrelated to that knowledge concept will remain unchanged. In Fig. [6] we present case where we only increase a single dimension the student representation, and it shows that the student's probability of answering questions related to that knowledge concept correctly increases.
student proficiency on one speccfffy knowledge concept
Figure 4: Example of variation of one factor in student proficiency representation. We randomly select student and three knowledge concepts.
A.5 Nemenyi Test
The Nemenyi test [9] is conducted to present the comparison between our proposed DCD and the interpretable baselines for all 5-fold volsdvvali results. significant difference is regarded to exist in the Nemenyi tests if the average ranks of two models differ by at least one crucial difference, which is determined using 5 5% significance level. The model metric improves as the ranking score decreases. As illustrated in Fig 5 Fig 6 and Fig 7 which demonstrates the comparison on 100%, 20% and 10% labeled exercises scenario, respectively. It's obvious that our proposed DCD approach performs best for all metrics in any scenario.
1- I-TCVAE
NOTM
KSED
KSCO
KINCO
DBIA KSC7
JB DCD KANCO
ICCN
KSCD
1ee DCD DBA
ALC
ACC
(
60
Figure 5: Nemenyi tests for comparison on 100% labeled exercises scenario
DCD
JOVAE
ACC
(3 RMSE
640 DOA
Figure 6: Nemenyi tests for comparison on 20% labeled exercises scenario
15

---

CONES
Dha WEDM
123 DCD
CAM MCOM
121 KANCO KSCD
3.3 KANCO
MCDM 5-47
DCD GMDD
21 ESED
REDMES
DENA 3F
Figure 7: Nemenyi tests for comparison on 10% labeled exercises scenario
B Experimental Details
B.1 Reparameterization Trick
In the student encoder module, student proficiency representation 2, is sampled from a Gaussian distribution N ( ow). we sample Zu from N (Huu σu) directly, we will lost gradient information as the sampling process is not differentiable. Fortunately, any Gaussian distribution can transform standard Gaussian distribution. we can sample a (O I) that has nothing to do with any parameters. By doing so, Z.a is equal to f4u a. 6, where @ means element- wise product. However, there no similar property of Bernoulli distribution corresponding to exercise relevance encoder. In this work, we adopt gumbel-softma reparameterization [15] to alleviate this problem, which is one of the gradient estimator methods.
B.2 Algorithm for Filling Missing Q-matrix
Existing interpretable baselines cannot directly apply to few-labeled scenarios. We design a filling method by employing similarity of response records in exercise to infer missing knowledge concepts. The detailed procedure is shown in Algorithm[]] We first compute the similarity (detailed in Eq. (IO) among exereises according to response records of exercise. Then for each unlabeled exereise, vote the voles most frequently occurring knowledge concepts among the most similar everr labeled exercises the filling knowledge concepts. We voles and ezerk
default.
S.j = En,eu T(Xmi Xnj 1=)==1)
(10)
En I [(Xnj # 0)
where S denotes the similarity between exercise v; and uy. 1 (condition) 1 if condition is True, and vice versa (condition) = X the interaction matrix in training set. X = 1, represent student un answered exercise correctly and incorrectly, respectively. X,, represents student u, did not answer exercise v; 2/ the student set.
Algorithm 1 Filling Missing Q-matrix for Interpretable Baselines
Input: labeled exercises set V1, unlabeled exercises set V22 interaction matrix X in training set, missing Q-matrix Q, number of similar exercises ezer number of filling knowledge concepts vote:
Output: Filling Q-matrix Q
1: M - VV2
2: Initialize exercise similarity matrix S of size (M, M) with the default value -o
3: for all exercise v E V1 do
4: for all exercise 8y 1/2 do
5: Calculate Si according to Eq. 10
6: end for
7: end for
8: for all exercise Vy E V22 do
9: 8, the most similar eλerı labeled exercises of Un according to S|:,7]]
St the voles knowledge concepts with the highest frequency of occurrence in 8.
11: Set corresponding elements to be one in Q :] by 8g
end for
13: return Q
16

---

B.3 Datasets and Preprocessing
Our experiments are conducted on three real-world datasets, i.e., Matma Juny and NIPS2020EC [42, all of which contain knowledge concepts of the tree structure. For all datasets, we preserve the first-time diagnosis. The detailed information on datasets and preprocessing method are depicted as following:
record for same student-exercise pairs to support cognitive
Matmat. This dataset is collected from an intelligent web application for practicing mathe- matics called Matma which includes three-level knowledge concept structure after prepro- cessing. The first-level knowledge concept the root node called math. The second-level knowledge concepts are organized 5 calculational types including addition, multiplication, division and so on. The third-level knowledge concepts are organized by calculational scale. We preserve students with more than 15 response records to guarantee that each student has enough data for diagnosis.
Junyi. This dataset is collected from an online educational platform called Junyi Academy There are three-level knowledge concept structure in this dataset. Besides the first-level root node, we apply area and topic field in origin dataset to denote the second-level and third-level knowledge concept rrspeettvvlyy. We preserve students with more than 50 response records to guarantee that each student has enough data for diagnosis.
NIPS2020EC. This dataset originated from NeurIPS 2020 Education Challenge 42] which provides students' answers to mathematics questions from Eed There are four-level knowledge concept structure in this dataset. We sample subset from the task 1 of this competition by selecting response records during March 2020. We preserve students with more than 30 response records to guarantee that each student has enough data for diagnosis.
B.4 Implementation Details
We train our model with Python 3.9 and PyTorch 1.12.1 on NVIDIA RTX A5000. The student and exereise encoders are implemented by multilayer perceptron for all datasets. We prior Gaussian distribution with A(0, 1) for each latent factor in ft, and μ and a prior Bernoulli distribution with Bernoulli (0.2) for each latent factor in μ.. For all interpretable models, we select the epoch with the best DOA for testing and almost set Adam as the default optimizer. The implement of DOA [38] metric is adopted from EduCDM TT)
For more implementation details, please see our public code repository htttps://githie com/ kervias/DCD
C Knowledge Concept Tree
C.1 Standard Knowledge Concept Tree
The employment of knowledge concept tree in our work is to obtain group based disentanglement according to granularity at different levels. For convenience, we transform the initial knowledge concept tree into a standard tree, which is defined as the tree that the height of all subtrees of any node in the tree is equal in our work. The transformation process is displayed in Fig. [ For those leaf nodes whose depth has not reached the depth of the tree, through inheritance until reaching the depth of the tree.
continuously duplicate this node
C.2 Example of Real Knowledge Concept Tree
In this work, we assume that there is less independence among groups of knowledge concepts grouped by their parent nodes the knowledge concept tree. This viewpoint inspired by practical applications, and the analysis is as follows. To better clarify the argument, we provide the partial
https://palccarasshpp cmu wtt.:/m.....aaaaaaamaeaa.a wtw..cc.ccc/laaaaaeettttaaaaaaaa
https://matmat:
https://www.junyiaccaa
httpa://eedi.com.
17

---

D
E
D
E
F
Figure 8: Process of transforming non-standard knowledge concept tree structure into standard tree.
knowledge concept tree NIPS2020EC dataset in Fig. 9 Firstly, the course chapter hierarchy is typical example of concepts with a tree structure. Intuitively, the associations among concepts at the high level of the hierarchy are weaker compared the associations between concepts at the low level. For example, if we adopt the 2nd level concepts (i.e., algebra, data and statistics, ...) to group the last level concepts, the independence among groups higher than the groups determined by the 3rd level concepts (i.e., inequalities, formula, data collection, ...). Moreover, there is higher likelihood that concepts within the same chapter are simultaneously assessed the same exercise. For instance, the concepts whose parent concept is inequalities would have higher probability occur in the same exercise. However, the solving linear inequalities concept whose parent concept is inequalities and the tally charts concept whose parent concept is data collection would have lower probability occur in the same exercise.
Number
Siriight Graple
Data Processing
Quadratic Graphs-
Representation
Inequalities Number Lines
Tally Charts
Graphing Lincar Inequalities (Shading Ragi
Types Data and Quotiooniree
Data and Statistios
Data Collection
Solving Linear Inequalities-
Inquulitics
Algehes
NIPS2120EC
Froquency tibles
Cellection-Oteer
Inequalities Otthers
Probability
Solving Quadratic Inequliiies-
Dota and Staritics-Other::
TransSommtion of Functions
Writing Simplifying Expressions
Geometry and Messure
Formula
Figure 9: An example of four-level knowledge concept NIPS2020EC dataset. Here we only provide details the hierarchical structure NIPS2020EC -Allgea -Ae---lra--neeuuaittiiee and -Aleoee Dula and Statistics Daata Collection.
D Limitations
There are still some limitations of our proposed method. 1) The latent factors considered in our method are only related to knowledge concepts. However, the student answering process may refer to more factors that are not related to knowledge concepts such as the family economic conditions of students and reading comprehension difficulty of exercise textual information, which results in the incomplete and insufficient disentanglement in our proposed method. The following work we consider is introduce more side information help disentangle other factors. The assumption that inter-group knowledge concepts are independent may be too strong. The later work we consider model causal relation among knowledge concepts for disentanglement, 3) The alignment module for unlabeled exercises is just to keep sparsity. In the future, we will introduce explicit relation among knowledge concepts help infer missing knowledge concepts.
18


Reviews:
[{"review_id": "aknGiRfmd4", "reviewer": "Reviewer_d3Wh", "summary": "The paper proposes a method to perform cognitive diagnosis. The method leverages notions of disentanglement representation learning to achieve better interpretability while avoiding sacrificing performance of predicting the students' answers. Experiments were conducted on three popular datasets and quantitative empirical evidence appears to support the above claim (no loss of predictive performance; improved interpretability).", "questions": "What do the authors mean by interpretability? Does it mean disentangling factors of variation on the students' latent vector? If so, what is being disentangled? Does the different dimension of the students' latent vector mean anything, or correspond to one or few concepts defined in the Q matrix?\n\nIn line 228, what's the rationale to use L2 loss? From my understanding, this loss aims to encourage sparsity, so why not using an L1 loss?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of leveraging disentangled representation learning in the context of cognitive diagnosis seems new. \n\nThe execution of this idea is interesting as the authors made a few changes to solve the technical challenges of applying disentangled representation learning to modeling students' skills and predict their answer correctness.\n\nThe experiments cover three popular datasets and seem comprehensive, including numerical results comparing multiple baselines and various analyses.", "weaknesses": "My main concern is on the notion of \"interpretability\". After reading the paper, I fail to understand what is \"interpretability\" in the authors' context. For disentangled representation learning in the vision domain, interpretability mostly means disentangling factors of variation, such that when you perform generation, you can manipulate the encoded latent vector (e.g., change a specific dimension of the vector that corresponds to one variation such as shape, size, viewpoint, etc.) to control the generation to be of a particular shape, size, viewpoint, and so on.\n\nThe authors seem to apply a similar notion of \"interpretability\" on the latent student knowledge/skill vector. However, I do not see any evidence or discussion on how such interpretability is manifested, e.g., what are the \"disentangled factors\" that the model learns? What do these factors mean and how do they connect to the knowledge concepts in the Q matrix? Do the model learn different disentanglement for different students? I believe an illustrative example would be extremely useful. Reporting only the DOA metric is not very convincing to me as a measure of \"interpretability\". \n\nBecause the paper makes a big claim on \"interpretability\" and the proposed methodology revolves around it, I am unsatisfied with the empirical evidence and analyses around it. I am open to raise my score if the authors can provide more in-depth discussion and definition around interpretability in their context, as well as illustrative examples to demonstrate the interpretability that their model learns.\n\n\n----\npost-rebuttal update: authors answered most of my questions and I have updated my scores accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions and concerns one-by-one as follows:\n\n### Questions\n\n**Q1 & Weakness - Brief: Confusion on interpretability and DOA.**\n\n**A1**: Thanks for your question. In cognitive diagnosis, different dimensions of the students' latent vector are required to correspond to specific knowledge concepts. In few-labeled exercise scenario, it is hard to meet the interpretability requirement of cognitive diagnosis. Therefore, we propose a novel DisenCD to enchance this type of interpretability through Disentangled Representation Learning (DRL). In DisenCD, interpretablity also means disentangling factors of variation. As shown in Figure 2 in the rebuttal global PDF, we randomly select three concepts, we observe that increasing the value of the correpoding concept, all students' predicted ratings on the exercises with the corresponding concpet would increase.  \n\nTo better illustrate the interpretability of our DisenCD, we will elaborate from two perspectives: the data generation process and the variation of disentangled factors. we will draw an analogy on DRL between the visual domain and cognitive diagnosis, and answer your questions during the process.\n\n1.Vision Domain\n\n(1) The data generation process\n\nWe can view the entire image dataset as evolving from a series of generative factors (such as color, shape, size,viewpoint, etc). For a single image, it is determined by the specific states of all the generative factors.\n\n(2) The variation of disentangled factors\n\nGiven a disentangled representation, changing the value of one dimension, such as color, will result in a change in the color of the image only.\n\n2.Cognitive Diagnosis\n\n(1) The data generation process\n\nWe can view the entire response records as evolving from three groups of generative factors: student proficiency factor on each knowledge concept ($\\mu_u$), exercise difficulty factor on each knowledge concept ($\\mu_v^d$), and exercise relevance factor on each knowledge concept ($\\mu_v^r$). \nFor a response record, result of the student response to exercise $v$ depends on factors: knowledge concepts contained in the exercise $v$ (corresponding to exercise relevance factors), exercise difficulty on each knowledge concept contained in the exercise $v$ (corresponding to exercise difficulty factors) and student proficiency on each knowledge concept contained in the exercise $v$ (corresponding to student proficiency factors). \n\n(2) The variation of disentangled factors\n\n- The variation of disentangled factors in student proficiency representation.\nGiven a student proficiency representation, each dimension in the representation corresponds to the student's proficiency level on a specific knowledge concept. The higher the value of a dimension, the higher the proficiency level on that knowledge concept.\nWhen we only increase the value of a single dimension in the student representation (improving the student's proficiency on a specific knowledge concept), while keeping the exercise difficulty representation and relevance representation unchanged, the student's probability of answering questions related to that knowledge concept correctly will increase, while the probability of answering questions unrelated to that knowledge concept will remain unchanged. \nIn Figure 2 in the rebuttal global pdf file, we present a case where we only increase a single dimension in the student representation, and it shows that the student's probability of answering questions related to that knowledge concept correctly increases.\n- The variations of disentangled factors in exercise difficulty representation disentangled factors in exercise relevance representation have similar phenomenon.  \n\nAs for the DOA metric, the interpretability of cognitive diagnosis primarily aims to reflect the student's proficiency in various knowledge concepts when providing prediction results. Therefore, diagnostic models should provide reasonable diagnostic reports based on the student's historical response data. To meet this requirement, DOA metric is proposed to measure the degree of agreement between diagnostic report (i.e. student proficicency representation) and the whole students' historical response data. In other words, DOA metric currently is the most commonly used metric to measure the interpretability in cognitive diagnosis[1,2,4]. Perhaps this type of metric is deficient. We are also eager for more researchers to participate in cognitive diagnosis and come up with more interpretability metrics. Thanks for your comment sincerely.\n\n**Q2: In line 228, what's the rationale to use L2 loss? From my understanding, this loss aims to encourage sparsity, so why not using an L1 loss?**\n\n**A2**: Thank you for your question. In our preliminary attempt, we first directly try to use L1 loss for numerous unlabeled exercises, and we find it does not show competing performance and it is harder to reach convergence in the training process. We speculate a possible reason is as follows: with L1 loss, after model initialization, it is very hard to revise these incorrectly labeled exercises. In contrast, L2 loss are sensitive to outliers, and would have a larger loss when the corresponding knowledge is incorrectly predicted.  We show the experimental results of L1 and L2 loss in Table 2 in the global PDF file.\n\nBesides, to ensure sparsity , we use a margin-based loss as shown in the first part of Eq.(7). In the margin-based loss, after sorting each predicted exercise encoder in a descending order, we encourage that the top-ranked concepts have a large margin compared to the least-likely concepts, i.e. ,the (d1+d2+1)-th largest value to the K-th values . \nAs shown in Figue 3(h), d1+d2 are set to small values. Therefore, the margin-based loss can encourage the predicted concepts to be sparse. Sorry for the confusion again, and we would explain Eq.(7) clearer in the revised version."}, {"review_id": "Q74OKD2tsn", "reviewer": "Reviewer_xiTN", "summary": "This paper introduces an innovative approach called DisenCD, aimed at enhancing the performance of cognitive diagnosis when exercise labels are limited. The proposed method incorporates two newly developed modules: the group-based disentanglement module and the limited-labeled alignment module. These modules effectively identify relevant concept factors and align them with existing labels. By leveraging limited exercise labels, the DisenCD method maximizes their potential for improved cognitive diagnosis. ", "questions": "Since the tree structure of knowledge concepts is very important for the proposed method, The details of the tree structure in each dataset should be provided. ", "limitations": "1. DisenCD is a complex model that involves multiple modules and hyperparameters. This complexity may make it difficult to implement and tune, especially for practitioners who are not familiar with the underlying techniques.\n2. DisenCD may not be scalable to large datasets due to its complexity. This may limit its applicability in real-world scenarios where large-scale cognitive diagnosis is required.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an intriguing and relevant problem: cognitive diagnosis with limited exercise labels, which has practical implications.\n2. The paper effectively utilizes a tree-structure to disentangle concept-related factors and align them with existing labels, providing a valuable solution to the aforementioned problem.\n3. The extensive experiments conducted in this paper convincingly showcase the effectiveness of the proposed method, further strengthening its validity.", "weaknesses": "1. Although DisenCD aims to enhance interpretability in cognitive diagnosis, the model itself may lack interpretability. The inclusion of disentanglement and alignment modules could introduce additional complexity, making it challenging to comprehend the model's decision-making process.\n2. In Section 4.2, the authors claim that there is reduced independence among knowledge concept groups in high-level tree nodes. However, the absence of necessary experiments or references weakens the support for this argument.\n3. Section 5.1 lacks the necessary details regarding the implementation, which leaves gaps in understanding the practical aspects of applying the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions and concerns one-by-one as follows:\n\n### Weaknesses\n**W1-Brief: Although DisenCD aims to enhance interpretability in cognitive diagnosis, the model itself may lack interpretability.**\n\n**A1**: Thanks for your comment. In cognitive diagnosis, \"Interpretability\" means that the each dimension of learned student proficiency and exercise difficulty/relevance representation should correspond to a specific knowledge concept. Thus, it can be used to analyze the student proficiency. However, it will become difficult when there are few labeled exercises, which is the focus of our paper. We design the disentanglement and alignment modules to construct DisenCD, such that each dimension responds to a specific concept, and the variation of each dimension can control the prediction results of corresponding concepts. As such, the requirements of cognitive diagnosis can be satisfied in few-labeled exercise scenario.  This is the reason why we claimed that our proposed DisenCD is able to enhance interpretability in cognitive diagnosis.\n\nMoreover, we can understand the model's decision-making process by drawing analogies to the encoder and decoder processes in VAE-based methods. The decision process of the model involves two main stages: 1) In the encoder stage, the disentanglement module enforces the independence of the joint distribution of latent variables within each group of representations, resulting in student proficiency, exercise difficulty, and exercise relevance. The alignment module then ensures that each dimension of the representation aligns with a specific knowledge concept, giving it practical meaning. 2) In the decoder stage, prediction scores are generated based on the aforementioned three representations, similar to existing prediction methods in cognitive diagnosis.  \n\nWe will further clarify the model's decision-making process in the manuscript. \n\n**W2: In Section 4.2, the authors claim that there is reduced independence among knowledge concept groups in high-level tree nodes. However, the absence of necessary experiments or references weakens the support for this argument.**\n\n**A2**:\nThanks for your question. We believe that the independence among high-level concepts (coarse-grained) is higher than low-level concepts (fine-grained). This viewpoint is inspired by practical applications, and the analysis is as follows. To better clarify the argument, we provide the partial knowledge concept tree of NIPS2020EC dataset in Figure 1 in the rebuttal global pdf . Firstly, the course chapter hierarchy is a typical example of concepts with a tree structure. Intuitively, the associations among concepts at the high level of the hierarchy are weaker compared to the associations between concepts at the low level. For example, if we adopt the 2nd level concepts (i.e., algebra, data and statistics, ...) to group the last level concepts, the independence among groups is higher than the groups determined by the 3rd level concepts (i.e., inequalities, formula, data collection, ...). Moreover, there is a higher likelihood that concepts within the same chapter are simultaneously assessed in the same exercise. For instance, the concepts whose parent concept is inequalities would have a higher probability occur in the same exercise. However, the solving linear inequalities concept whose parent concept is inequalities and the tally charts concept whose parent concept is data collection would have a lower probability occur in the same exercise.  Based on your valuable suggestions, in the revised version, we would add the group-based structure to make it clearer.\n\n**W3-Brief: Section 5.1 lacks the necessary details regarding the implementation.**\n\n**A3**:\nThank you. We presented the implementation details in the supplementary materials, specifically in sections C.4 and C.5. Besides, we also provide the code including baselines in the supplementary material zip file. Based on your valuable suggestion, we would add a link to the codes in the main body to facilitate easy reimplementation. \n\n### Questions\n**Q1-Brief: The details of the tree structure in each dataset should be provided.**\n\n**A1**: \nThank you for your valuable comment. According to your suggestion, we add the partial tree structure for NIPS2020EC dataset, as shown in Figure 1 in the rebuttal global pdf file. Displaying the tree structure on these datasets helps readers to have an intuitive understanding of the operation process of DisenCD. We will update the relevant content of all datasets in the supplementary materials accordingly.\n\n### Limitations\n\n**L1 & L2 - Brief: DisenCD is a complex model that involves multiple modules and hyperparameters. This complexity may make it difficult to implement and tune. DisenCD may not be scalable to large datasets due to its complexity.**\n\n**Answer**: \nThanks for your comment. We agree with that the DisenCD involves multiple modules and hyperparameters and introduces additional time complexity. In this paper, we aim to design a cognitive diagnosis for numerous unlabeled exercises scenario, which significantly reduces the cost of expert annotation. Therefore, we believe that the increased complexity is acceptable compared to the huge expert annotation cost.Due to the task scenario becoming more complex and challenging, it is inevitable for the model's complexity to increase and the number of hyperparameters to grow. Besides, we have provided the code including baselines in the supplementary material zip file. If the paper is luckily to be accepted, we promise to release our project to facilitate reproducible research.  In the future, we will explore methods with lower complexity in this scenario. This is a promising direction, and we very much appreciate your suggestion.\n\n"}, {"review_id": "qxP416vznM", "reviewer": "Reviewer_B6tk", "summary": "This paper focuses on performing cognitive diagnosis with limited exercise labels. To address the enormous cost of labeling exercises, in this paper, the authors proposed Disentanglement based Cognitive Diagnosis (DisenCD). Specifically, they first used students’ practiced records to model student proficiency, exercise difficulty, and exercise label distribution; Then, group-based disentanglement and limited-labeled alignment modules were proposed to disentangle the factors relevant to concepts and align them with real limited labels. At the same time, a tree-like structure of concepts was proposed for group-based disentangling. ", "questions": "1.\tIn Section 2.1, The authors believed that the TextCNN method proposed by NCDM+ contained errors in extracting knowledge concepts from the semantic information of the exercise text. Could the authors provide verification of this claim? Alternatively, could the authors compare their method with the TextCNN approach to assess its effectiveness in extracting knowledge concepts?\n2.\tIn Section 4.1, initially, the authors explained that the interaction matrix X contains significant information, including student proficiency. However, it is later mentioned that the response records of a student conceal his proficiency regarding the knowledge concepts. This raises the question of why such a contradiction exists.\n3.\tTo the best of my knowledge, most cognitive diagnosis (CD) [1][2][3] works typically exclude students with less than 15 answer logs to ensure the plausibility of the results. However, I am curious as to why the authors chose to employ three different criteria for removing students in the three selected datasets.\n4.\tRegarding the 10% Q matrix or the 20% Q matrix, what criteria do the authors employ to retain 10% or 20% of the original Q matrix?\n\n[1] Wang F, Liu Q, Chen E, et al. Neural cognitive diagnosis for intelligent education systems[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 6153-6161.\n[2] Gao W, Liu Q, Huang Z, et al. Rcd: Relation map driven cognitive diagnosis for intelligent education systems[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021: 501-510.\n[3] Ma H, Li M, Wu L, et al. Knowledge-Sensed Cognitive Diagnosis for Intelligent Education Platforms[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022: 1451-1460.\n", "limitations": "1.\tCurrently, in the field of cognitive diagnosis, many datasets lack a tree structure between knowledge concepts. Therefore, the method proposed in this paper may not be suitable for the majority of datasets, i.e., the ASSISTments2009 dataset[1], and the ASSISTments2012 dataset[2].\n2.\tAs can be seen from Table 2 in the experimental part and Table 2 in the supplementary materials, the method proposed in this paper is not well suited for datasets[1][2] with a large number of knowledge concepts. This would greatly limit the applicability of the method.\n\n[1] https://sites.google.com/site/assistmentsdata/home/assistment-2009-2010-data/skill-builder-data-2009-2010\n[2] https://sites.google.com/site/assistmentsdata/datasets/2012-13-\nschool-data-with-affect\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea presented in this paper is novel and holds value in addressing the problem of limited exercise labels, as annotating them with domain experts incurs a substantial cost. Meanwhile, in the supplementary materials, it is highly significant for the authors to perform the Nemenyi Test on the experimental results.", "weaknesses": "1.\tFigure 1: The radar chart has some mistakes. For the diagnosis of knowledge concept k2, we can find that the proficiency of student u1 is higher than that of u2, but the answer logs given by the authors show that student u1 answered v1 correctly, answered v2 incorrectly, and student u2 answered v1 correctly.\n2.\tIn line 28, the word 'The' is written incorrectly.\n3.\tIn line 70: MIRT has been enhanced based on IRT and can provide an assessment of subjects or students from multiple perspectives.\n4.\tIn the experimental section: \n(1)\tIt is clear that the model proposed by the authors did not achieve the best results in all scenarios. \n(2)\tInsufficient preparation of experimental parts, i.e., lack of ablation experiments to verify the effectiveness of each part.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions and concerns one-by-one as follows:\n \n### Weaknesses\n\n**W1 and W2 : The radar chart in Figure 1 has some mistakes...., 'The', …**\n \n**A1 and A2**: \nVery sorry for the mistake and typos. In fact, the proficiency on concept k2 of student u1 should be lower than that of u2. We will update Figure 1, and correct typos in the manuscript accordingly.  \n \n**W3: In line 70: MIRT has been enhanced based on IRT and can provide an assessment of subjects or students from multiple perspectives.**\n \n**A3**:\nThank you. We agree with you that MIRT transforms the student representation from a scalar in multiple dimensions. However, in MIRT, each dimension of the multidimensional vector representing students does not correspond to a specific knowledge concept. We will rewrite this part to avoid confusion. \n \n**W4-(1): It is clear that the model proposed by the authors did not achieve the best results in all scenarios.**\n \n**A4-(1)**: \nThank you for your comment. We compared the results of the model in terms of prediction metrics (i.e., AUC, ACC, and RMSE) and interpretability metric DOA, as shown in Table 2 of the manuscript. Only in the 10% Q-matrix scenario of the Junyi dataset, our model achieves prediction metrics comparable to the state-of-the-art model KaNCD. Moreover, for the interpretability metric DOA, the proposed model significantly outperforms the best baseline of KaNCD model. Therefore, the experimental results demonstrate the effectiveness of the proposed model.\n \n \n**W4-(2)-Brief: Insufficient preparation of experimental parts.**\n \n**A4-(2):** \nThanks for your suggestion. We have presented the results of the ablation experiments in Figure 3 of the original submission.  Due to the page limit, we demonstrate this by hyperprameter analysis. Specifically, the disentanglement module is influenced by hyperparameter $\\beta$. When $\\beta$ is equal to 0, it means DisenCD without disentanglement. The margin loss and L2 loss in alignment module is influenced by $\\lambda_1$ and $\\lambda_2$ , respectively. When $\\lambda_1$=0 and $\\lambda_2$ =0 , it means DisenCD without alignment without margin loss and L2 loss, respectively. We present the ablation experiments in Table 3 in our rebuttal global pdf file. The experimental results validate the effectiveness of the disentanglement and alignment modules. We will include the separate ablation experiment results in the supplementary materials. \n\n### Questions\n\n**Q1-Brief: Provide verification of the claim about NCDM+.... Compare DisenCD with TextCNN approach…**\n \n**A1**:\nThanks for your question and sorry for the confusion. Extracting knowledge concepts from the exercise text is a good idea to deal with the few-labelled exercise scenario. We additionally focus on a more challenging scenario of missing exercise text scenario and infer knowledge concepts from student interaction records. According to your valuable suggestion, we would rewrite this part to avoid misunderstanding. \n\n**Q2-Brief: There exists contradiction in statement…**\n \n**A2**:\nSorry for the confusion due to the typo. In line 154, the word “…conceal..” should be “reflect”.  We will correct this error.\n \n\n**Q3-Brief: The choice of filtering threshold for removing students.**\n \n**A3**: \nThanks for your professional question.\n \nWe conducted a survey of related literature on student filtering methods, including setting a certain filtering threshold (e.g., 15 [1], 50 [2] ) or selecting a certain number of students (e.g., 1000 [3], 10000 [4]). These methods have slight deviations in threshold settings due to different dataset sizes. Therefore, we have also designed different filtering thresholds according to the different dataset sizes. The student scales of the three datasets we used, from small to large, are Matmat, NIPS2020, and Junyi, respectively. Therefore, we set the filtering thresholds as 15, 30, and 50 in ascending order based on the student scale.\n \nAdditionally, we conducted an experiment where we set the filtering threshold to 15 for NIPS2020EC dataset. The experimental results are shown in Table 1 of our global rebuttal pdf file. DisenCD outperforms the baselines, demonstrating the effectiveness of the DisenCD model. \n\n**Q4-Brief: The choice of the 10% Q matrix or the 20% Q matrix.**\n \n**A4**:  \nThanks for your question. We randomly selected two representative cases where there are 10% and 20% labeled exercises. We can also adopt other ratios.\n\n### Limitations\n**L1-Brief:  DisenCD requiring the knowledge concept tree may not be suitable for the majority of datasets.**\n \nA1: \nThanks for your valuable comment. Some publicly available datasets in the field of cognitive diagnosis do not include tree structure information. For such datasets, we can also constraint independence among all dimensions. \n \nMore importantly, we believe that obtaining and annotating tree-structured knowledge concepts is relatively easy due to the following reasons: 1) Tree-structured knowledge concepts are typically reflected in the structural divisions of course chapters, and when designing a smart education system, chapter divisions are inevitably involved, making this data readily available. 2) Compared to cognitive diagnostic models that utilize prerequisite relationships among knowledge concepts, tree-structured knowledge concept information is easier to annotate and obtain. \n \n**L2-Brief: DisenCD is not well suited for datasets with a large number of knowledge concepts.**\n\n**A2**: \nThanks for your insightful comment. We agree that our model may not be well suited for datasets with a large number of knowledge concepts. This is due to the Beta-TCVAE that DisenCD rely on for disentanglement, which has a high computational complexity when calculating total correlation. In the future, we will further explore the complexity issue of the model to alleviate this limitation. \n"}, {"review_id": "9anuXpBs0X", "reviewer": "Reviewer_zBu1", "summary": "This paper presents an algorithm for cognitive diagnosis (CD) in limited data scenarios. CD aims at labeling questions to knowledge concepts.\n\nAs the CD process requires expert tagging, labelling questions to knowlege concepts in time intensive.\nThis paper addresses the labelling task from  three factors: student proficiency, exercise difficulty, and\nexercise relevance. \n\n", "questions": "Did the author validate (a subset of) the discovered KCs with domain experts?\nDid the author evaluate how their model performs with small student responses to questions in V2? ", "limitations": "Yes, addressed in the supplementary material.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": " 1. Knowledge concept (KC) labeling in limited data scenario\n 2. Disentangled Representation Learning for interpretation\n 3. Addresses correlated and independent KCs by a tree-like hierarchy\n ", "weaknesses": "1. The authors utilized student interaction distribution to derive exercise relevance and knowledge concepts from exercise relevance. \n \nOne limitation of utilizing student interaction is the cold-start problem of newly introduced exercises or a small number of student responses. \nA more robust way to model exercise relation is by considering the question text and student responses. See Pandey et al. [1] for reference.\n\n2. The assumptions of inter-group and correlated KCs may be too strong.  \n \n References.\n 1. Pandey S, Srivastava J. RKT: relation-aware self-attention for knowledge tracing. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management 2020 Oct 19 (pp. 1205-1214).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions and concerns one-by-one as follows:\n\n### Weaknesses\n**W1: The authors utilized student interaction distribution to derive exercise relevance and knowledge concepts from exercise relevance.\nOne limitation of utilizing student interaction is the cold-start problem of newly introduced exercises or a small number of student responses. A more robust way to model exercise relation is by considering the question text and student responses.** \n\n**A1**:\nThanks for your suggestions. Our algorithm focuses on inferring the exercise relevance from student responses when there is a lack of textual information. Therefore, it does rely on the quantity of interactions associated with the exercise.  In our future work, we will consider exploring the exercise relevance from both  response records and additional exercise information (e.g., text, graph, and video).\n\n**W2: The assumptions of inter-group and correlated KCs may be too strong.**\n\n**A2**: Thanks for your question. We believe that the independence among high-level concepts (coarse-grained) is higher than low-level concepts (fine-grained). This viewpoint is inspired by practical applications, and the analysis is summarized as follows. To better clarify the argument, we provide the partial knowledge concept tree of NIPS2020EC dataset in Figure 1 in our rebuttal global pdf file. Firstly, the course chapter hierarchy is a typical example of concepts with a tree structure. Intuitively, the associations among concepts at the high level of the hierarchy are weaker compared to the associations between concepts at the low level. For example, if we adopt the 2nd level concepts (i.e., algebra, data and statistics, ...) to group the last level concepts, the independence among groups is higher than the groups determined by the 3rd level concepts (i.e., inequalities, formula, data collection, ...). Moreover, there is a higher likelihood that concepts within the same chapter are simultaneously assessed in the same exercise. For instance, the concepts whose parent concept is inequalities would have a higher probability occur in the same exercise. However, the solving linear inequalities concept whose parent concept is inequalities and the tally charts concept whose parent concept is data collection would have a lower probability occur in the same exercise. \n\n### Questions\n\n**Q1: Did the author validate (a subset of) the discovered KCs with domain experts? Did the author evaluate how their model performs with small student responses to questions in V2?**\n\n**A1**:  Thank you for your question.\n\n- Validate the discovered KCs with domain experts\n\n     Yes, we conducted experiments to validate the discovered knowledge concepts with domain experts. The results are presented in Appendix A of the supplementary materials in which we compare DisenCD with pre-filling algorithm. \n\n- Evaluate performance with small student response\n \n     We agree that the performance of the proposed algorithm rely on the quantity of interactions associated with the exercise. DisenCD may not perform  well on exercises with small student responses. As you mentioned in weakness 1, we would consider incorporating multimodal information to make up for the shortcoming."}, {"review_id": "FnWTGsgYqp", "reviewer": "Reviewer_GRC1", "summary": "The dissociation-based cognitive diagnosis (DisenCD) model proposed in this paper addresses the cognitive diagnosis challenge of limited practice labels by using students' historical practice records to model their proficiency, practice difficulty, and practice label distribution. The model introduces novel modules to disentangle factors related to knowledge concepts and align them with the limited labels available. Experiments on three real datasets demonstrate the effectiveness of the model. Overall, this paper presents a promising approach to the cognitive diagnosis of intellectual education. However, in the process of specific description, the introduction of the two modules is too vague, and some experimental details are not provided in the article.", "questions": "1.Can you please provide more details on the pre-filling algorithm used for missing Q-matrix in the experiments？\n\n2. Could you explain further the two novel modules, group-based disentanglement and limited-labeled alignment, that DisenCD uses to disentangle relevant knowledge concepts and align them with the available limited labels?\n\n3.Can you elaborate on the hyperparameter settings during the experiment?", "limitations": "No, the authors did not provide any discussions for limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Practical Approach: DisenCD provides an effective approach for cognitive diagnosis under limited exercise labels, representing one of the few attempts to focus on this problem. Extensive experiments on three real-world datasets demonstrate the effectiveness of the model.\n\tGroup-based disentanglement and limited-labeled alignment modules: The DisenCD model introduces two novel modules, group-based disentanglement, and limited-labeled alignment, which disentangle factors relevant to knowledge concepts and align them with the available limited labels. These modules help to overcome the problem of limited exercise labels and improve the accuracy of cognitive diagnosis.\n\tInterpretable: The DisenCD model achieves the best interpretability in all scenarios, especially in the 10% and 20% Q-matrix (label-scarce exercises) scenarios, where it significantly outperforms interpretable baselines in terms of Degree-of-Alignment (DOA). This demonstrates the superior interpretability of the model.", "weaknesses": "\tLimited Generalizability: Although the effectiveness of the DisenCD model is demonstrated on three real-world datasets, the model's generalizability to other datasets or samples still needs to be established. Further studies are, therefore, needed to establish the generalizability of DisenCD to more extensive and diverse datasets.\n\tLack of Explicit Discussion on Limitations: Although the article describes the proposed model, its training procedures, and experimental results, the limitations of the DisenCD model should be explicitly discussed. This may limit the reader's ability to interpret the results and implications of the model. A more explicit discussion of the model's limitations would have made the study more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments, efforts, and time. We respond to each of your questions and concerns one-by-one as follows:\n\n### Weaknesses\n**W1-(1): Limited Generalizability: Although the effectiveness of the DisenCD model is demonstrated on three real-world datasets, the model's generalizability to other datasets or samples still needs to be established. Further studies are, therefore, needed to establish the generalizability of DisenCD to more extensive and diverse datasets.**\n\n**A1-(1)**:\nThank you for your suggestion. We focus on proposing a cognitive diagnosis model based on a tree structure of knowledge concept relationships. Therefore, we have adopted the most popular datasets in this field, which includes the tree structure of knowledge concept relationships. In fact, these three datasets are most commonly used and representative datasets in cognitive diagnosis, given our best efforts. In the future, we would be happy to validate the generalizability of DisenCD on more datasets if we come across them.\n\n\n**W1-(2): Lack of Explicit Discussion on Limitations: Although the article describes the proposed model, its training procedures, and experimental results, the limitations of the DisenCD model should be explicitly discussed. This may limit the reader's ability to interpret the results and implications of the model. A more explicit discussion of the model's limitations would have made the study more comprehensive.**\n\n**A1-(2)**: \nThank you. Due to the page limitation, we placed the limitations of DisenCD and an outlook on future work in the supplementary material B. We are sorry for not indicating the location of the limitation description in the manuscript. We will add it accordingly.\n\n### Questions\n\n**Q1: Can you please provide more details on the pre-filling algorithm used for missing Q-matrix in the experiments?**\n\n**A1**: \nThank you. Due to the page limitation, we placed this part in Section C.3 of the supplementary material in which we provide a detailed pseudo-code algorithm description on filling missing knowledge concept. Besides, we also compare DisenCD with pre-filling algorithm in section A.1 of the supplementary material.\n\n**Q2: Could you explain further the two novel modules, group-based disentanglement and limited-labeled alignment, that DisenCD uses to disentangle relevant knowledge concepts and align them with the available limited labels?**\n\n**A2**:\nGroup-based disentanglement module:\n\nThe fundamental assumption of disentangled representation learning is that each latent variable is independent. However, in cognitive diagnosis, each latent variable corresponds to a knowledge concept, and there are relationships between knowledge concepts. For example, certain knowledge concepts are assessed together, and one must master certain knowledge concepts before moving on to the next one. Therefore, to apply disentangled representation learning in cognitive diagnosis, we attempt to find independent relationships between knowledge concepts and group them accordingly. Knowledge concepts within a group have high correlation, while knowledge concepts between groups have low correlation, thus constraining the independence of knowledge concepts between groups. To achieve this, we incorporate knowledge concept tree structure information, where we consider higher-level knowledge concepts to have coarser granularity and higher independence among them compared to lower-level knowledge concepts.\n\nAlignment module: \n\nIn the few-labeled exercise scenario, we assume that there are a large number of unlabeled exercises and a small number of labeled exercises. For the labeled exercises, we directly align dimensions with knowledge concepts in the exercise relevance representation using the supervised information from the labeled exercises. As for the unlabeled exercises, considering the sparse nature of knowledge concept annotations in the exercises, we constrain the sparsity of the unlabeled relevance. Here, we divide the exercise relevance representation into three parts: 1) Those most likely to correspond to true knowledge concepts; 2)Those serving as candidate knowledge concepts; and 3)Those with very low probability of being knowledge concepts. We use margin loss to encourage the first part to be as close to 1 as possible, allowing the model to adaptively infer missing knowledge concepts. \n\n\n**Q3: Can you elaborate on the hyperparameter settings during the experiment?**\n\n**A3**: \nThank you. Due to the page limitation, we placed the hyperparameter settings in Section C.5 of the supplementary materials in our original submission. We will try our best to reorganize the layout to add some important information regarding the settings into the main text. Thank you again for your suggestion.\n"}]

Author rebuttal:
We sincerely appreciate all reviewers' thoughtful comments, efforts, and time. In the current global response, we have included an **attached rebuttal global file**. We will be **referencing some figures or tables when replying to each reviewer**. Thanks again to the reviewers for their efforts. 

The references occurred in the rebuttal are listed as follows:

[1] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In AAAI, pages 6153–6161, 2020.

[2] Xinping Wang, Caidie Huang, Jinfang Cai, and Liangyu Chen. Using knowledge concept aggregation towards accurate cognitive diagnosis. In CIKM, pages 2010–2019, 2021.

[3] Haiping Ma, Manwei Li, Le Wu, Haifeng Zhang, Yunbo Cao, Xingyi Zhang, and Xuemin Zhao. Knowledge-sensed cognitive diagnosis for intelligent education platforms. In CIKM, pages 1451–1460, 2022.

[4] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In SIGIR, pages 501–510, 2021.


Meta-review: The paper tackles the problem of cognitive diagnosis, i.e., measuring students' proficiency, when limited exercises are labeled with concepts. The problem is well-motivated and many real-world settings face the challenge of limited exercise labels, given the huge labeling cost involved. The proposed method, Disentanglement-based Cognitive Diagnosis (DisenCD), addresses this problem using a semi-supervised disentanglement modeling approach. Extensive experimental evaluation on three real-world benchmarks showcases the effectiveness of the proposed method. The reviewers acknowledged that the paper considers an important problem setting and that the proposed method of cognitive diagnosis with limited exercise labels is of practical interest. However, the reviewers also raised several concerns and questions in their initial reviews. We thank the authors for their detailed responses and for actively engaging with the reviewers during the discussion phase. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)