Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling
Authors: Weibo Gao
Qi Liu
Linan Yue
Fangzhou Yao
Hao Wang
Yin Gu
Zheng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Learner Modeling
Intelligent Education System
Cognitive Diagnosis
Disentangled Representation Learning
Primary area: other

Abstract: Learners sharing similar implicit cognitive states often display comparable observable problem-solving performances. Leveraging collaborative connections among such similar learners proves valuable in comprehending human learning. Motivated by the success of collaborative modeling in various domains, such as recommender systems, we aim to investigate how collaborative signals among learners contribute to the diagnosis of human cognitive states (i.e., knowledge proficiency) in the context of intelligent education.
The primary challenges lie in identifying implicit collaborative connections and disentangling the entangled cognitive factors of learners for improved explainability and controllability in learner Cognitive Diagnosis (CD). However, there has been no work on CD capable of simultaneously modeling collaborative and disentangled cognitive states. To address this gap, we present Coral, a $\underline{Co}$llabo$\underline{ra}$tive cognitive diagnosis model with disentang$\underline{l}$ed representation learning. Specifically, Coral first introduces a disentangled state encoder to achieve the initial disentanglement of learners' states.
Subsequently, a meticulously designed collaborative representation learning procedure captures collaborative signals. It dynamically constructs a collaborative graph of learners by iteratively searching for optimal neighbors in a context-aware manner. Using the constructed graph, collaborative information is extracted through node representation learning. Finally, a decoding process aligns the initial cognitive states and collaborative states, achieving co-disentanglement with practice performance reconstructions.
Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over state-of-the-art methods across several real-world datasets.
Our code is available at https://github.com/bigdata-ustc/Coral.

Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling
Weibo Gao Qi Liu 1.2 Linan Yue'
Fangzhou Yao Hao Wang'
Yin Gu' Zheng Zhang'
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center weibogao @ waillustc..du.com qiliuqi @ustc.uuuuu.... (Inyue, fangzhouyao) maiil.sttc.edu.om wanghao3 ustc.edu.cn; [gy128, 128, zhangzheng) mail.ustc. ust..edu.ccnnn
Abstract
Learners sharing similar implicit cognitive states often display comparable observ- able problem-solvee solving performances. Leveraging collaborative connections among such similar learners proves valuable in comprehending human learning. Motivated by the success of collaborative modeling in various domains, such as recommender systems, aim to investigate how collaborative signals among learners contribute to the diagnosis of human cognitive states (i.e., knowledge proficiency) in the context of intelligent education. The primary challenges lie in identifying implicit collaborative connections and disentangling the entangled cognitive factors of learn- for improved explainability and controllability in learner Cognitive Diagnosis (CD). However, there has been no work on CD capable of simultaneously modeling collaborative and disentangled cognitive states. To address this gap, we present Coral, a Collaborative cognitive diagnosis model with disentangled representa- tion learning. Specifically, Coral first introduces a disentangled state encoder achieve the initial disentanglement of learners' states. Subsequently, a meticulously designed collaborative representation learning procedure captures collaborative signals. It dynamically constructs collaborative graph of learners by iteratively searching for optimal neighbors in context-aware manner. Using the constructed graph, collaborative information is extracted through node representation learning. Finally, a decoding process aligns the initial cognitive states and collaborative states, achieving co-disentanglement with practice performance reconstructions. Exten- sive experiments demonstrate the superior performance of Coral, showcasing signif- icant improvements over state-of-the-art methods across several real-world datasets. Our code is available at phttp:: //gi thub. com/ ctw/ bigdata ustc/Coral
Introduction
Qecatoes
Kenntedyi axtapo
Beb
Nanee
A: Function
Legend
D: Cube
A
B: Derivative Cone
It is common notion that individuals with sim- ilar implicit states frequently exhibit similar ex- plicit behaviors. Therefore, establishing inter-
C: Number
92
O: Correct/ wrong logs
A.B
Prodiciency
connections among similar users crucial for understanding human behaviors. For instance, social connections play pivotal role in under- standing current consumer preferences and pre- dicting future behaviors 42). Similarly, in the
qa
Beb
Natey
Cognlttive states
Figure 1: An example of human learning, where learners individually select questions practice. Each question tests at least one knowledge concept.
Indicates Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

context of intelligent education, a better modeling of fike-minded learners with similar learning expe- riences, is essential for understanding the human learning process [I. analyzing their knowledge proficiency and facilitating personalized tutoring tailored to individual needs [55
As illustrated in Figure | we infer Nancy is likely answer the Cone-related question qs correctly according to the correct practice responses of Bob and Alice, who share similar learning behaviors with Naney. The underlying psychological assumption is that learners with similar experiences generally possess similar cognitive states - how well the learner masters each knowledge concept, influencing their subsequent responses. To gain a deeper understanding of the human learning process, it is crucial to explicitly diagnose unobservable cognitive states. Existing Cognitive Diagnosis (CD) methods seek to enhance diagnostic accuracy by fully utilizing the inner-learner information (i.... individual attributions [4]] and explicit practice records) and question-side features (c.g.. difficulty 16), textual content [26], and educational relations [12 14). However, the issue of how similar (a.k.a. collaborative) connections among inter-learners with similar states facilitate understanding of learners' knowledge proficiency remains largely unexplored.
In this study, to efficiently harness collaborative information among similar learners and thereby more accurately diagnose the cognitive states of each individual, we advocate for the incorporation of inter-learner connections into the CD process. However, designing a collaborative CD model in educational scenarios presents two distinct challenges due to the complexity of human learning.
First, acquiring explicit collaborative connections among learners proves to be formidable challenge. On the one hand, unlike many well-defined social scenarios (e.g., Twitter.com), where user preference similarities are manifested through explicit social actions such as following and liking, the directly available social behaviors among learners in learning environments (e.g.. LeetCode.com cannot be used for diagnosis modeling since these social attributes cannot reflect true cognitive-oriente connections. On the other hand, some related studies (28 13 attempt to design different similarity functions based on practice data compute cognitive similarities among learners. However, these approaches pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases. Although various methods for constructing user relationships have been proposed in other domains (22] [ these approaches do not consider the domain-specifer attributes of students in learning scenarios and cannot be directly applied in educational contexts.
Second, an ideal collaborative diagnosis procedure requires disentangling and uncovering the mixed explanatory latent factors hidden in the observed learning behaviors. The basic motivation is that learners demonstrate complex and diverse patterns driven by entangled states across both inner- and inter-learner perspectives. For instance, from an inner perspective, Nancy may not master Cone since she does not practice Cone-related questions. However, based on inter-learner data, one can infer high probability that she has mastered Cone. Most prior attempts can not fulfill this requirement since they learn representations in an entangled way. Although recent models 8 [5] achieve dimension- -evell disentanglement of cognitive states, they lack consideration of modeling the influence of collaborative connections, ignoring the complex relations between inner- and leterel connections of different individuals. Thereby, it needs to find a suitable way to achieve the co- disentanglemeet from both the inner- and inter-learner views for cognitive representations with higher interpretability and controllability.
To tackle the above challenges, we propose Coral, a Collaborative cognitive diagnosis model with disentangled representation learning, to reveal learner cognitive states while simultaneously modeling both inner- and inter-learner learning information. Specifically, our approach begins with the disentangled cognitive representation encoding to establish initial disentangled learner states through reconstructing their practice performance from the inner-learner perspective. Next, our focus shifts to effectively learning collaborative cognitive representations from the inter-learner perspective. The most significant point is to find the implicit collaborative relations between learners. To address this challenge, we present context-aware collaborative graph learning mechanism that automatically explores all K -ootiial neighbors for each learner given their basic cognitive states to facilitate the explicit modeling of collaborative connections among learners. Based on the constructed graph, collaborative information can be effectively fused into disentangled learner cognitive states through learning collaborative node representations. Finally, a decoding and reconstruction process is conducted to merge initial states and collaborative states so to achieve co-disentanglemmnt from both the inner- and inter-learner perspectives. Extensive experiments demonstrate the superior performance of Coral, showing significant improvements over SOTA methods across several datasets.

---

2
Related Work
Cognitive Diagnosis As fundamental task, cognitive diagnosis (CD) has been well-researched for decades in educational psychology [25 4 [3] It aims to profile the implicit cognitive states (i.e., the proficiency of specific knowledge concepts) of learners by exploiting observed practice records (e-g., correct or wrong). Existing research on CD assumes that learners' knowledge proficiency is proportional to their practice performance and thus can be diagnosed through predicting their practice performance [1]. Since the diagnostic results can be applied many intelligent applications, such as exercise recommendation [18] and learning path suggestions (55]. many CD models have been proposed in recent years. The early works from psychology like IRT [16] and MIRT [ focus on modeling learners' answering process by predicting the probability of learner answering a question correctly, which utilizes latent factors as the learner's ability. These methods lack interpretability, i.e., they are inability to output explicit multidimensional diagnostic results on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of questions to diagnose learners' proficiency on all knowledge concepts [38 45 46 32).
Representative NCDM [40] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works (e.g., IRT, and MIRT). In summary, existing CD studies enhance diagnostic accuracy by fully utilizing the inner-learner information (i.e., individual attributions and explicit practice records) A1 [0)and and question-side features (e.g., difficulty [16, (8). textual content [26). and educational relations [12 14, (). However, to the best of our knowledge, the problem of collaborative diagnostic modeling remains largely unexplored.
Collaborative modeling in Education Collaborative connections among learners in the education context commonly refer to learners with similar explicit practice behaviors, testing scores and However, due the complexity and implicitness of the human learning process, these relations are commonly not explicitly and directly available. Existing studies 228 in AI Education have attempted design different similarity functions based on practice data to compute cognitive similarities among learners. However, these methods pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases.
implicit knowledge proficiency [28 54 44
Disentangled Representation Learning Disentangled Representation Learning (3) which aims to produce robust, controllable, and explainable representations, has become one of the core problems in machine learning. Typical methods include variational method (20.. weakly supervised models [21], as well as the recent combination with the diffusion model (6) DRL has wide range of applications in user modeling to disentangle attributes. For example, recommendation with several aspects of users' interests [2 33 fair user representation to disentangle sensitive attributes [10]. In education, DCD attempts to disentangle learners' cognitive representations via variational framework, which motivates us to conduct further study on collaborative CD) setups.
Coral
We first introduce the problem setup, followed by details on three core components of Coral: i) Disentangled Cognitive Representation Encoding, ii) Collaborative Representation Learning and iii) Decoding and Reconstruction. Figure shows the framework. The algorithm is listed Algorithm
3.1 Problem Setup
Our setup considers the human learning dataset D including the practice records between M learners and N questions. The practice records of each learner u is denoted by x, {Zu,), where Fu,a equals or 0, representing that leamer u answered question i correctly or not, respectively. Each question is related at least one knowledge concept. The association relations between N questions and C' knowledge concepts is represented by = [ where e; E RRC and ç₁ equals or 0 denoting that question i is related to concept not. The practice records are regarded as the explicit inner-learnigg information in our context.
Besides, we consider the collaborative connections among learners with similar cognitive states, which provide the inter-learner information. We define collaborative connections as a graph structure G == (V, E') which contains a set of nodes (i.e., learners) V and a set of edges F where (u, v) E E or (u, E G indicates that the existence of collaborative connection between learner u and v (i.e., and have similar latent cognitive states). Notably, the collaborative connections in educational
3

---

step
Graph modeling
8,,
Rus
[
KOOung Se..
Aub
Yap
N00.00
Decoding and Reconstruction
Disentangled Cogritive enpaiientation Encoding
Collsborative Representation Learning
Notations:
O
Questinn practicing correctly incoreecty
Leamer under one particulor knowledge concept
Unchosen leamers
Figure 2: The overall framework of Coral.
scenarios are generally not explicitly or directly available, and it needs to design an adaptive strategy to automatically infer similar leamerr from observed learning data during the training process.
To achieve cognitive state disentanglement, we initially assign C factorized representations to each learner, i.e., 2, (2)) Zu
(2). z E RdxC' with Gaussian Mixture initialization since the Gaussian distribution has long been recognized as a proper statistic model for the cognitive states
of learners in educational psychology 4 The component ((() is expected to capture the learner u's cognitive state over knowledge concept c. We denote θ as the of trainable parameters
for the proposed model. Based on the above setups, the goal of Coral is to learn co-disentangled representations Ż = {²m) for the M learners from both the inner-learner practice perspective and inter-learner collaborative perspective.
3.2 Disentangled Cognitive Representation Encoding
The practice response x,, of each learner u provides valuable inner-learner insights regarding his/her proficiency since learners' performance on each question is assumed to be proportional to their cognitive proficiency on question-related knowledge concepts [8] Therefore, we implement an encoder for encoding the disentangled cognitive state 2, of each learner u by reconstructing their practice responses. For a learner u. we assume that his/her practice performance on candidate questions can be generated from the following distribution:
[ pe (xu Zu: po
Pe (xx.) = E
(1)
where p(C) (C) and pe (x,, Zu, C) is naturally a cognitive diagnosis procedure to predict practice performance. The key point of this task to learn an optimal encoder pe (z..) via practice records X, to encode the cognitive state variational distribution qe (z,, x.,) to approximate pe (z.). following the VAE literature [3], through maximizing lower bound of Pe pe (xx) based on the following property.
of each learner u. To optimize O we introduce
Property max log pe (x,) bounded as follows:
log pa (x,)= 2 (p(C)nsex....x... xx.) [log pe (Xa, zzn]l Ep(c)) [DKL (ge (zu x.) || pe (zw))]-
(2)
See the Appendix A for the proof.
In Property 1, the first term reconstructs the true practice performance x., of learner and the varia-
tional encoder qe (z,, xx) in the second term approximates the true encoder pe (z.) ) by minimizing
the KL divergence DKL. The variational distribution de (z., x.)) and the expectation (e. (X..)
are intractable, thus we employ the re- c-aarmmeterizztion trick 220 for the model optimization.
Furthermore, the diagnosis procedure pe (x., C) is achieved by estimating how well learner answers question from both the perspectives of cognitive states and comprehensive abilities. From the perspective of cognitive states, solving question requires learner to master all knowledge concepts related to this question. Regarding comprehensive abilities, each learner possesses latent state reflecting their overall learning ability, which is shared when addressing different questions.

---

Formally, this process can be described as:
pe (Xu Zuz C) = II
pe (Fu.) Bus C)2
Exn
pe (Frus
Zus C)
Σ
Ci de (0 2 -hi) O,, = Σ the (2(3)
where (-).) and ve(+): : IRd + R+ are two shallow neural networks, vee() estimates the compre- hensive ability of the learner and (e(+) predicts the performance of learner with given cognitive state ( and a comprchensive ability 6, over question i in terms of concept c. h; is a learnable latent representation for question Besides, to ensure psychometric interpretability of prediction, we set the weights of (-) are positive values, i.e., (-)/ > 0, assuming that the probability of correctly answering the question monotonically increases with learners' cognitive state. Please
note that we found that the mean operation here can also be replaced with a neural network (i.e.,
d's RC + R+) with positive weights, formulated as d'a (c. (0. 2! - -h:) h:)) as in 40 without affecting prediction performance. Particularly, in contrast to most methods that consider entangled cognitive factors as input, our diagnosis model can better capture learners' proficiency on each knowledge concept by disentangling cognitive states under each concept.
Furthermore, inspired by the outstanding performance of 8-TCVAE in disentanglement, we prompt statistical independence among its dimensions to obtain a better trade-off between the recon- struction accuracy and the quality disentangled representation through g(z!.) = I[ de (Fn))
where qe (2(() is the aggregated posterior Zu. i..., ge(z).) -f ge (z()))) (x.) = Pdata (x.). This setup is encouraged by the term (-I in Eq. based on Property 2.
(c)
Xx )p (x,) dx, where
Property 2. The DKL(-) ) in Eq. (2) can be rewritten as:
DRL. (ge (Z..
x.) pe (z..)) (Zur, Xu) + DKI. (ze(Zu) || pe()))
See Appendix A for the proof. On one hand, (Zu, X..) maximizes the mutual information (MI) between z, and which obtains the useful information for the diagnosis task as much possible according to the information bottleneck theory (2]. On the other hand, given a Gaussian distribu-
tion pe (e( II5 11i-Pe pe (x..)) the KL divergence term encourages independence among the
dimensions of (c)
by preventing each latent variable from deviating too far from specified priors. Compared to prior VAE-based CD models [50] Coral additionally considers ability parameters
from psychology [16] to enhance the expressive power of disentangled cognitive states.
Overall, we penalize Eq. 2 by Lagrange multiplier (3 resulting in the following objective:
log (xw) 2 E Ep(C)uolz. X.) [log (x, 2.)l 3 Ep(()) [DEL (ge (zu x.) pe (z.))]
(5)
3.3 Collaborative Representation Learning
Collaborative information among similar learners provides auxiliary inter-learner insight for cognitive representation learning. However, collaborative connections among learners with similar states are typically readily accessible. To address this challenge, we design context-aware graph construction strategy that searches similar neighbors automatically via the initial disentangled cognitive states. Based the constructed collaborative graph, can learn collaborative node representations by aggregating collaborative signals to generate collaborative cognitive states.
3.3.1 Context-aware Collaborative Graph Learning
The core goal of constructing the collaborative graph is to find K optimal neighbors for each learner node in V via their initial disentangled cognitive state [2(e) le=1- For different knowledge concepts, the cognitive connections between the same learner pair are typically different. Thereby, it needs to search C groups of similar neighbors for each learner via each disentangled component 2(e), This
means that would generate C' collaborative graphs, i.c., G (G(o)) Each collaborative
5

---

graph (c)) under concept is expected to characterize the cognitive similarities of learners regarding concept Formally, this task is defined as computing pe(G V, Z) by identifying all the K similar neighbors for each learner covering each concept Ned) denote the set of K similar neighbors for the learner u, the task can be described as:
max pe (G V. == max
po (Nrt) ae!?)
[Togpe 1
c
(6)
c
C
max Σ (11)) (Z0) +
Σ c=1
E po (z(o))
[logpe
( max
Σ
(Mic,,
c=l
where A1t2 and (0) = [Zm las are the neighbor set and feature set of all the learners regarding
(0)
knowledge concept c, respectively. The number of A1 equals the combination of arbitrary K neighbors from all M learners for each learner node under each concept i.e.,
[Nde)
M! FTM
thus Eq. (6) is computationally expensive especially for larger M and K. To facilitate computation, transform the Eq. (6) that requires global MI maximization the task of maximizing MI locally via locally available context information inspired by [22] and derive a lower bound of it as the following Property 3.
Property 3. max log Pe (G) V. V) is bounded as follows:
C M
K
ce),k where c(O). exp (fie (bL. ;ra (r())--1)
max maxloggr log pe (G| V Z)
Everto exp (Jie) (uprlohk-1
(c),k-1
See the Appendix A for the proof. The Eq. 7 iteratively searches K neighbors for the learner under each knowledge concept e from step 1 to K. Cle), the function [36. Let ru (c).x-1 denote the current context at step (I: 1) (i.e., the set of (I- 1) neighbors
InfoNCE loss
selected from step (k-
- 1)). ((),, is the affinity candidate learner in the (M k-) nonneighbor learners. Let Vy(0) denote the current set of nonneighbor learners, and we hence have (c).) E V1(0).
f(c) (b(c) (r(c))k- is a matching function measuring the similarity between of nonneighbor ((c).
and the current context ru (c)ak- where the higher the scalar score means the higher likelihood of (c).k is a new neighbor.
Furthermore, we have c(c).k Sic) (b(o)/, ; -.))).k)). Thus, given the context of (k- neighboring leamners (i.e., we have found (A: - 1) neighbors for the learner u) and matching function f(e) (-), our goal following the Property 3 is to find learner b(c).k from nonneighbor set V.(0) that can maximize the matching score f(e)
the k-th neighbor of u. In other words, p(G V,Z) can be optimized through maximizing the matching score (-))()) from k 1 to K iteratively. Thereby, at each step k we sort the scores of the nonneighbor learners and select the learner with the highest score to label
as k-th neighbor (c).k i.e., b (c).k +- arg max, (1e)(;; r( (c), ), E v(c) After obtaining the k-th
neighbor 6( (c))k the context ri (c).k-1 is updated to ru (c)., by absorbing ((().
The calculation of matching score (-))()) usually relies the node representations (i.e., learner cognitive states). However, the sub-optimal cognitive state learning during the initial training epochs probably results in the matching function exhibiting biases. To enhance the stability of model
training, instead directly aggregating node representations as the context ru (c),k=1 as many graph learning works, denote it using relative representations w.r.t. the learner [22]. Without loss of generality, we first establish relative collaborative coordinate systems with learner node u as the origin, and process relationship measurements between node and each of its neighbors as
(c) (e) (c) [|2. Then the context-aware features can be generated by aggregating each node Zu,t = l
(c) in the context rı [c).A-1 i.e.. re (c),k-1 Everroh Zu.y. Thereby, let 2 denote (c).k with feature
Zin, we have flcrmx = f(0) (bLo) ;ru (0))-11) = 2 rc ke)

---

3.3.2 Collaborative Graph Modeling
After iteratively searching K neighbors under each concept, regarding each learner, i.e., (e) ) Then, we consider collaborative modeling as a node repre- sentation learning task within each collaborative graph (e)- It relies nonlinear kernel function (-)()) to aggregate neighboring information and update each disentangled cognitive state, i.e.,
obtain C' collaborative graphs
Na (e {z! (u, v) E G( Given the disentangled learner cognitive states generated by the variational posterior distribution qe (zy Xx,) from Property (-e()) is naturally expected to contain C' channels to extract different concept features from similar learners, though projecting the representation 2u into different subspaces, i.e., 2 = <W"
(c) z!!? + b(e))/lla(( (e)
2 + b(o))l2..
where (()) E Rd and ((e) R are learnable parameters of channel and o--) is nonlinear activation function (e.g.. Sigmoid), and || is Lę normalization ensuring numerical stability. Then the collaborative learner representation modeling in terms of concept c can be described as:
((O)T
((()
fley,
(c)
Σ VENN
(c) u.v
2(0)
(c) 8,0
(8)
W
ESENT
(efT (c) Zu 2
> f(e),o
where Su,0 is the attention weight between u and v, considering both the collaborative aggregation (the first term) commonly used in graph modeling works and the corresponding context-aware attention (the second term) calculated in the iterative graph construction process in Eq. 77) When K is set large in Eq. (7). there is possibility of introducing non-collaborative noise. In such cases, (c) can assign lower values to conccoblaboreeive neighbors to mitigate the negative impact of noise, allowing for the adaptive tuning of attention in graph modeling. During training. the channels will remain changing because different subsets of the neighborhood will be searched for dynamically aggregating neighbor information in different iterations.
With Gaussian Mixture initialization from the Disentangled Cognitive Representation Encoding (section 3.2p, we derive the theorem on convergence as:
Theorem 1. The Collaborative Representation Learning (section 3.3 procedure equiva- lent to an expectation-maxiiization (EM) algorithm (35] for the mixture model. In partic- ular, it converges
point estimate of (1(()) that maximizes the marginal likelihood ( (u,v) E G(a)) (ra
l ((at
Je=1) ( where du,v equals or 0 denoting whether learner
is a collaborative neighbor of learner u regarding concept e or not. See the Appendix A for the proof.
3.4 Decoding and Reconstruction
Given the initial disentangled encoding via inner-learner information (section 3.2 and the collab- orative representation learning via inter-learner information (section 3.3 this part encourages an aligmment between the initial encode 2, and collaborative state ru, formulating a co-disentangled representation as ž, Z, + ru. This operation is inspired by the residual block (17) address the second challenge, where r, can be treated as a disentangled auxiliary information of z,, from collaborative graphs.
The decoding process predicts the practice performance of each learner u on candidate ques- tions, given her co-disentangled representation ă, (z(1) z x(()) i.e., pe (x.)
(C) ppe (X,, 2. C)), similar to the reconstruction procedure in Eq. (1. Thus, putting Eq. 5
and Eq. (7) together, we have the overall training objective:
M
arg min L
Σ
Σ
a. BCE (N pe (rru Buy C)) 3 Dk
Σ
BCE (#u pe (2u,i))
Zu. Exu
C K
clc)k
s.t.
arg max

---

Datasets
ASSIST
Junyi
NeurIPS2020FC
#students
1,256
1,400
1,000
#questions
16,818
#knowledge concepts
120
#concepts per exercise
1.21
4.02
#records
199,790
70,797
331,187
#records per student
159,07
50.67
331.19
#correct records #incorrect records
67.08%
77.20%
53.87%
Table I: The statisties of three datasets.
where BCE(:)) the binary cross entropy loss function between ground-trutt practice behaviors Xa and the reconstructed ones. a and [ are hypee-parameters.
By optimizing with minimizing the above loss function Eq. (9). the cognitive state 2., of each learner w can be jointly refined serving as the diagnostic results. During the testing phase, we evaluate the model performance by matching the difference between the predicted score pe (F and the true
score x., Specifically, when proficiency value is required instead of the vector ž we can obtain by averaging each dimension of 2!!0)
Experiments
We empirically evaluate the performances the proposed Coral model over three real-world datasets and conduct several experiments prove its effectiveness.
4.1 Experimental Setup
Datasets We conduct experiments three real-world datasets: ASSIST [II, Junyi [ and
NeurIPS2020EC 43 The statistics of datasets are listed in Table The details about datasets and preprocessing are depicted in the Appendix C
Baselines The baselines include the matrix factorization-bast model, i.e., PMF (3), the typical
latent factor models derived from educational psychology. including IRT [16), MIRT [[] and the
neural networks-based models, including NCDM [40], RCD [1), KaNCD [1 and DCD [8.
Evaluation Since cognitive states cannot be directly observed in practice, it is common to indirectly evaluate CDMs through the student performance prediction task on test datasets prediction performance, we adopt ACC and AUC and F1-score as metrics from the perspective of classification, using threshold of 0.5, and RMSE as metrics from the perspective of regression, following previous work 12 23)
To evaluate
Settings We the dimension size d as 20, the layer of graph modeling as and the mini-batch size as 512. In the training stage, we select the learning rate from (0.002, 0.005, 0.01, 0.02, 0.05]. select a from {0.05, 0.1, 0.5, 1) and / from {0.25, 0.5, 1), and select neighboring number K from 2, 4, 5, 10, 15, 20, 15, 30, 25, 40, 45, 50). All network parameters are initialized with Xavier ini- tialization [T5]. Each model is implemented by PyTorch [37] and optimized by Adam optimizer [19]. Specially, for the implementation of baselines, we set the dimensional sizes of each representation in PMF, NCDM, KaNCD, RCD and DCD as the number of knowledge concepts. All experiments are conducted on a Linux server equipped with two 3.00GHz Intel Xeon Gold 5317 CPUs and two Tesla A100 40G GPUs.
4.2 Experimental Results
Prediction Comparison We evaluate prediction performance of Coral against baselines under three setups: normal, sparse, and cold-start scenarios.
Table 2 reports the performance comparison under normal settings for all the models across three datasets on four evaluation metrics. In this setting, we split all the datasets with a 7:1:2 ratio into training sets, validation sets, and test sets. The proposed Coral model significantly outperforms most baselines. This demonstrates two key benefits of Coral. First, the iterative graph construction process effectively generates collaborative connections for modeling. Second, the co- co-disentangled representation learning successfully discovers disentangled cognitive states for each learner.
We extend our analysis to assess the performance of Coral in sparse scenarios. In order to simulate varied sparse environments, we systematically discard 80%, 60%, 40%, and 20% of the training data from the ASSIST dataset under the normal settings described above. The experimental results

---

shown in FigureBiar reveal that Coral consistently outperforms baselines across a range of sparse environments. Moreover, our model exhibits robust performance consistently, demonstrating its adaptability and effectiveness in diverse sparse scenarios.
Moreover, conduct analysis of Coral's performance in a cold-start environment. To replicate this scenario, we retain solely the cold- start response data for each learner in the test set of Junyi, corresponding to the knowledge con- cepts they had not previously practiced in the training set. The Figure (b) illustrates the ex- perimental results, highlighting the exceptional performance of the Coral model (with K = 10) in cold-start scenario.
Metric
Dataset
Method
ACC T
AUC T Fl-score RMSE
IRT
69.36
69.81
78.14
45.61
MIRT
71.26
72.59
79.80
44.50
PMF
71.34
72.27
80.68
48.67
NCDM
72.27
74.27
79.97
48.67
ASSIST
72.43
75.38
80.22
48.67
RCD
72.04
73.14
80.60
43.74
DCD
70.33
73.98
79.09
43.94
Coral
71.53
74.72
81.16
43,66
IRT
79.26
76.46
87.54
38.38
MIRT
77.74
74.46
86.0S
40.29
Collaborate Graph Learning We investigate the influence of the generated neighbor number K on diagnostic performance. Figure 3 (c) dis- plays the prediction performances for various values of K on Junyi under the normal scenario. The model performance exhibits improvement as K increases, particularly noticeable when K is small. This observation suggests that the inter- learner information automatically retrieved by Coral contributes positively to the model. How- ever, once K surpasses threshold, the perfor- mance gain becomes less pronounced. This di- minishing effect arises because users beyond the threshold (i.e., K 10 in this dataset) may lack significant collaborative relationships, thus lim- iting the useful clues they offer. We observe that when K exceeds the threshold, the model's performance remains acceptable, and even the performance improves after K exceeds 30. This indicates that Coral effectively perceives the similarity functions of the scenario and collaborative context. Consequently, it assigns lower similarity scores to non-collaborative neighbors, robustly adjusting the attention weight in graph modeling.
PMF
79.65
77.17
88.18
44.10
NCDM
79.91
78.91
87.73
38.35
Junyi
KaNCD
81.79
80.93
89.02
36.11
81.02
37.23
DCD
79.29
79.55
87.62
37.83
Coral
81.15
80.94
89.12
36.08
IRT
70.11
75.60
71.59
44.68
MIRT
69.95
75.52
71.24
45.51
PMF
69.85
75.39
72.62
48.33
NCDM
71.66
78.57
71.36
43.21
NeurlPS2020EC
KaNCD
71.28
77.60
72.50
43.71
RCD
70.43
77.25
72.64
44.01
71.53
75.63
71.13
Coral
71.72
78.88
72.82
43,20
Table 2: Performance comparison. The best per- formance is highlighted in bold. (L) means the higher (lower) score the better (worse) per- formance, the same below.
To obtain more intuitive insight into the iterative graph construction process, we randomly select two learners (called target learners) from the Junyi dataset (with the normal setup) and visualize their neighbor selection process. Initially, of each learner u, which can be obtained by aggregating each disentangled cognitive component
utilize t-SNE [39] to present the aggregated cognitive vector
learned by Coral (setting K = 30), i.e., >e x(c) The embedding of the target learner is highlighted red, while the nodes representing neighboring learners are color-coded based on the selection steps, with unselected points displayed in gray. The outcomes are illustrated in Figure (a), showcasing how Coral organizes neighbors according to cognitive states and exemplifying a compelling strategy for neighbor selection that takes into account cognitive similarity.
Disentanglement We evaluate the disentanglement level achieved by assessing independence of dimensions within Zu. The independence level [L) of each z, is quantified as IL(u) = [c)
E 2 Eigassa 12
(0)
[i]l- where 2(0) [i] represents the 7th dimension of z!!) fol-
49]. In Figure (d), we depict IL = > IL(u) and the corresponding model performances at different training epochs on ASSIST (with the normal setup). Notably, Coral (setting K = 10) gradually achieves a high degree of disentanglement during the training process, and the model performances generally exhibit positive correlation with the degree of disentanglement. This observation reveals the effectiveness of the disentanglement process.
lowing a prior methodology [48
Additionally, we visualize the disentangled cognitive component representations (i.c., {!!) of each learner u learned by Coral. We treat each knowledge component in the representation as independent points, with each component colored differently. For visual clarity, we randomly select 200 learners and knowledge concept components for display. The results in Figure (b) uses t-SNE to visualize

---

(4)
(3)
(00.50
(004
0.5
=RNND NEEM
=Coaal
BCD
#Cral
BCD
1831
0.85
MBC
Hemm
0.5
MIRT
0.55
06
MCC
Flaore
IKT
0.70
0.5
80%
60%
00%
207
0.59
3
10
2
5
6
Patlo clscardinng
AOC
Valund
Epoch
Figure 3: (a) Performance sparse scenarios. (b) Performance under cold-start scenarios. (c) Perfor- mance with different values of K. (d) Disentanglement level and its correlation with performance.
60
ASSIST
Junyi
NIPS2020
Figure 4: (a) Selected neighbors of the target learner at different steps. (b) t-SNE visualizations of learner representations colored based on knowledge concepts.
learners' cogmitive states, with each color representing distinct category of knowledge-related learner states. This illustrates Coral's ability to achieve well-separated representations.
Explainability We further investigate the interpretability of the cognitive diagnosis outputs based on Coral. We aim to explore whether Coral can provide reasonable predictions for knowledge concepts that learners have not practiced in the training set during actual inference. Firstly, we randomly select target student from the Junyi dataset and identify 5 knowledge concepts (denoted A E) that u has not learned in the training data. Subsequently, based on the refined model, we retrieve the top 4 most similar neighboring learners (i.e., U1 U4) the target student u. F the assessed knowledge concepts (A ~ E) and the corresponding mastery levels of selected neighbors using radar chart. Table 3 presents the diagnostic outputs for the proficiency of u, along with 5 questions related to knowledge A ~ E, the predicted scores answering correctly and the actual performances of u. We observe that, despite the cold-start nature of these knowledge concepts for Coral, the model effectively outputs cognitive states that align with the true performance of by considering the mastery levels of collaborative learners with similar cognitive states. This sufficiently demonstrates the interpretability of Coral's 's diagnostic results.
E
84
Question
237
213
302
577 620
Knowledge concept
A
C
D
E
D
U
C
D
22
C
Proficiency (%)
67.2
68.1
48.3 52.3 $2.6
Predicted score (%)
73.4
75.2
47.3
50.8 57.2
E
24
B
E
66
True performance
Table 3: The comparison between the diagnostic results of Coral and the true performance, where v denotes an- swering correctly and x denotes answering incorrectly.
D
M3
C
D
U4
C
Figure 5: The example of diagnosis output.
Conclusion
We are pioneering the exploration of collaborative cognitive diagnosis by disentangling the implicit cognitive representations of learners. Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over SOTA methods across several real-world datasets. We believe this endeavor marks crucial step towards collaborative modeling for "Al Education". Furthermore, this work offers valuable insights into conseious-awan learner modeling, under the assumption that human learner proficiency can be effectively represented in disentangled manner.
Acknowledgments
This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2021 YFF0901003) the Key Technologies R & Program of Anhui Province (No. 202423k09020039 and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Statistics Reference Online, 2014.
[2] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2022.
[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):
1798-1828, 2013.
[4] Haoyang Bi, Enhong Chen, Weidong He, Han Wu, Weihao Zhao, Shijin Wang, and Jinze Wu. Beta-cd: A bayesian meta-learned cognitive diagnosis framework for personalized learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5018-5026, 2023.
[5] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In EDM, pages 532-535, 2015.
[6] Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang. Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity-peeeering disentangled tuning for subject-driven text-to-image generation. 2023.
[7] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[8] Xiangzhi Chen, Le Wu, Fei Liu, Lei Chen, Kun Zhang, Richang Hong, and Meng Wang. Disentangling cognitive diagnosis with limited exercise labels. In shirty- Conference on Neural Information Processing Systems, 2023.
[9] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems, 13:19314-1932 2020.
[10] Elliot Creager, David Madras, förn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pages 1436-1445. PMLR, 2019.
[11] Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with an online system that tutors as it assesses. User modeling and user-adapted interaction, 19:243-266, 2009.
[12] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Red: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 501-510, 2021.
[13] Weibo Gao, Qi Liu, Hao Wang, Linan Yue, Haoyang Bi, Yin Gu, Fangzhou Yao, Zheng Zhangm Xin Li, and Yuanjing He. Zero-1-to-3: Domain-level zero-shot cognitive diagno- sis via one batch of early-bird students towards three diagnostic objectives. arXiv preprint arXiv:2312.1343 2023.
[14] Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang. Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 983-992, 2023.
[15] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
11

---

[16] Robert Harvey and Allen L Hammer. Item response theory. The Counseling Psychologist. 27 (3):553-383, 1999.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Yujia Huo, Derek Wong. Lionel M Ni, Lidia S Chao, and Jing Zhang. Knowledge model- ing via contextualized representations for lstm-based personalized exercise recommendation. Information Sciences, 523:266-22 278, 2020.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6900 2014.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[21] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi supeevupe learning with deep generative models. Advances in neural information processing systems, 27, 2014.
[22] Rongfan Li, Ting Zhong, Xinke Jiang, Goce Trajcevski, Jin Wu, and Fan Zhou. Mining spatio- temporal relations via self-paced graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 936-944, 2022.
[23] Rui Li, Liyang He, Qi Liu, Yuze Zhao, Zheng Zhang, Zhenya Huang, Yu Su, and Shijin Wang. Consider: Commonalities and specialties driven multilingual code retrieval framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8679-8687 2024.
[24] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoen- coders for collaborative filtering. In Proceedings of the 2018 world wide web conference, pages 689-698, 2018.
[25] Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST), 944):1226, 2018
[26] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Fxercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 13(1):100-111) 2019.
[27] Chung Kwan Lo. What is the impact of chatgpt on education? rapid review of the literature. Education Sciences, 13(4):410, 2023.
[28] Ting Long, Jiarui Qin, Jian Shen, Weinan Zhang, Wei Xia, Ruiming Tang. Xiuqiang He, and Yong Yu. Improving knowledge tracing with collaborative information. In Proceedings the fifteenth ACM international conference on web search and data mining, pages 599-607, 2022.
[29] Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18733-18741, 2024.
[30] Xinwei Long, Jiali Zeng, Fandong Meng, Jie Zhou, and Bowen Zhou. Trust in internal or external knowledge? generative multi-modal entity linking with knowledge retriever. In Findings of the Association for Computational Linguistics ACL 2024, pages 7559-7569, 2024.
[31] Alcksandra Luszezynska and Ralf Schwarzer. Social cognitive theory. Fac Health Sci Publ, pages 225-51, 2015.
[32] Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang. Enhancing cognitive diagnosis using un-interacted exercises: A collaboration-aware mixed sampling approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8877-8885, 2024.
12

---

[33] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation, Advances in neural information processing systems, 32. 2019.
[34] Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. Advances in neural information processing systems, 20, 2007.
[35] Todd K Moon. The ippectation-maximiiiii (6):47-60, 1996.
algorithm. IEEE Signal processing magazine, 13
[36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018.
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, peshoperformacc deep learning library. Advances in neural information processing systems, 32, 2019.
[38] Emiko Tsutsumi, Ryo Kinoshita, and Maomi Ueno. Deep-irt with independent student and item networks. International Educational Data Mining Society, 2021.
[39] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[40] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6153-6161, 2020.
[41] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang. Yu Yin, Shijin Wang, and Yu Su. Neuraled: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 2022.
[42] Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Gc, and Wenwu Zhu. Curriculum co- disentangled representation learning across multiple environments for social recommendation. In International Conference on Machine Learning, pages 36174-36192. PMLR, 2023.
[43] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobato Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Instructions and guide for diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007. /2061, 2020.
[44] Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable seene simulation for autonomous driving via collaborative Ilm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15077-15087, 2024.
[45] Shangshang Yang, Haoyu Wei, Haiping Ma, Ye Tian, Xingyi Zhang, Yunbo Cao, and Yaochu Jin. Cognitive diagnosis- based personalized exercise group assembly via a multi-objective evolutionary algorithm. IEEE Transactions on Emerging Topics in Computational Intelligence, 733):829-844, 2023.
[46] Fangzhou Yao, Qi Liu, Min Hou, Shiwei Tong, Zhenya Huang, Enhong Chen, Jing Sha, and Shijin Wang. Exploiting non-interactive exercises cognitive diagnosis. Interaction, 100(200): 300, 2023.
[47] Fangzhou Yao, Qi Liu, Linan Yue, Weibo Gao, Jiatong Li, Xin Li, and Yuanjing He. Adard: An adaptive response denoising framework for robust learner modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3886-3895, 2024.
[48] Linan Yue, Qi Liu, Yichao Du, Yanqing An, Li Wang, and Enhong Chen. Dare: disentanglement- augmented rationale extraction. Advances in Neural Information Processing Systems, 35: 26603-26617, 2022.
13

---

[49] Linan Yue, Qi Liu, Ye Liu, Weibo Gao, Fangzhou Yao, and Wenfeng Li. Cooperative classifica- tion and rationalization for graph generalization. Proceedings of the ACM Web Conference 2024, pages 344-352, 2024.
[50] Yunfei Zhang, Chuan Qin, Dazhong Shen, Haiping Ma, Le Zhang, Xingyi Zhang, and Hengshu 7hu. Relied: A reliable cognitive diagnosis framework with confidence awareness. In 2023 IEFE International Conference on Data Mining (ICDM), pages 858-865 IEEE, 2023.
[51] Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang, Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. Fairlisa: Fair user modeling with limited sensitive attributes information. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[52] Zheng Zhang, Qi Liu, Zirui Hu, Yi Zhan, Zhenya Huang, Weibo Gao, and Qingyang Mao. Enhancing fairness in meta-learned user modeling via adaptive sampling. In Proceedings of the ACM on Web Conference 2024, pages 3241-3252 2024.
[53) Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. Understanding and improving fairness in cognitive diagnosis. Science China Information Sciences, 67(5):152106, 2024.
[54] Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, and Jing Wang. Lane: Logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. arXiv preprint MXXv:22407002833 2024.
[55] Yan Zhuang, Qi Liu, GuanHao Zhao, Zhenya Huang, Weizhe Huang, Zachary Pardos, Enhong Chen, Jinze Wu, and Xin Li. A bounded ability estimation for computerized adaptive testing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
A Proofs
Property 1. max maglooge pe (xn) bounded as follows:
log (xw)= 2 Bp(C)as(aa)xx)) [log (x, 2.)] - Ep(C) [DLL (ge (zn xx) pe (z,))) (10)
The proof is as follows.
Proof.
log pe (x.) log Ep((C) (pes (Xu. Zu: C) pe (zn))
(C) [log Pe (xx,) de (z [x,)
Ep(Cooo xx.)
[log pe (xx))
log Pe (X., Znn)
(x..)
Pe (Zu Xx)
log Pe (x., Z.y)
9e (2. x.)
pe
Zuu Xx)
qe ZZu
xx)
log / (z.,
x.)
log pe (Xu. Znd
(p(C)/e(e.
Ep(())a(2z) (x.)
pe
(Z,
x,)
go (Zr xxx) xx.)) + (2) log pe (Xn,
Zu pe (2n)
EPpcy [De.. (ge (zu
x.)
pe (z.,
90 (Zu Xx)
Ep(c) [DLL (ge (zu x.) pe (zm,
xx)) DKL (ge (z,, xx) pe (zw)))
+ C)p (X..)) [log pe (Xa zw)]
Ep(C) [E (2,, (X.) log pe (x,, z..) DKL (ge (z,, xx)) pe (zw))]
(11)
which completes the proof.
14

---

Property 2. The DRL(-) in Eq. ②2 can be rewritten as:
DKL (ge (Zm xx) |l pe (zw))
(12)
(Zus Xx) + DRL. (ze((n) pea (zu))-
The proof is as follows.
Proof. Given that pe (xu) Pdata (xx) and go (zu) = qa (zu xxx) pe (xx)), we then have
DRL. (ge (zu
x,) pe (zw))
log de (Zus xx)) qe (zu)
=
Exq (xx)
Pe (Zn) qe (Zm)
log yo (Zu xx))
qo (zw)
Egg Egn (xul)x)) (x.))
+ Ege (eu lew)
9a (Z.u)
pe (2m)
(13)
log 9o (Zu Xxx)
qe (zu)
= -Ega (z. xx Paace (Xu)
+ Eeg (an ben, Paara
(Zn)
pe (zu)
Egn (x.) log go (Z xx.)
ge (zu)
+ 18 (z.)
4e (zw)
pe (Zu)
=1 (Zui x.) + DKL (ge (zw) pe (zw)).
where I(A:B) calculates mutual information (MI) between and B. i.c. I(A:B)
Ep(a, by [log p(alb) M(a)
Therefore, the proof is completed.
Property 3. max Pe V,Z) is bounded follows:
K ΣΣΣ
max le V.Z)
where C()). exp ((o) (bLO) ((()))()))1)))
(14)
Enerde exp ((o)
(c),k
The proof is as follows:
Proof. Given the following inequality,
[logr] pe (N(+) [2(
max (G V.Z)= max
E Pe (NS2),
(Aro)) z(0)
logre re (z(z))
B Be (z)e)
(15)
(Nit) Z(0)
max
we further derive the term / (A/1)) (0) that search the combination of K similar neighbors from the permutation perspective as follows:
1 (
[Log pe (N 12!6)) + H (NTce
E
pe (A 2!"))
(16)
[log pe (R(() { ]] + H (N%c)
E (R) 2001)
r
where H (A) - Ep(a) (og)(a). R!(e) denotes the permutation of neighbors representing routes
Ade)) in given orders, where R((0) = (M-K)!- However, the search space of R((() is huge and
prohibitive. Inspired by the equivalent task [22]. next present the Eq. [6 in heuristic
15

---

style by maximizing the MI between the context of selected similar learner neighbors and the next neighbor iteratively.
Specifically, the core goal of Eq. 15) to find all K neighbors (dd for each learner u under a specific concept the data once, from the perspective of maximizing MI globally. However, this task challenging due to the especially large search spaces of N1 and R( Thereby, we decompose the globally optimal task into equivalent task in iterative local optimal process. Concretely, assume that we have found (kg- 1) optimal neighbors for the learner u formulating a route ra (c),8o-1 in specific order from learner node to (ko- - 1), we then search the ko-th neighbor by ((o).ko equally from the rest (M ko + 1) learners for the learner u. i..., p (b)o)
M--k+TT
Given arbitrary ko optimal neighbors for learner with specific sub-route /()), derive pe Ru 28 in Eq. (16) as follows:
K II
pe
(((0) 2(()
pe
(r(c).b))
pe
(lo)s r((hh-1)
(17)
pes
We can derive the log term log pe
R1(0)
z!!
in Eq. 16 as follows:
K
log pe (R(
Blo). rra (x(o).-1)
pe (2.
log
pe
(c),)
(c),b)
K
pe
pes (R!
log
(18)
pe
pe
(c).k
(o),k-1
K log pe (b(O).)
log
pe (((o)
bn
(c),K)
With Eq. (17) and
18). we can derive the first term in Eq. (16) as follows:
log ped (R!
2 )
E
Pe (R(2) (x("))
pe (c),k (c),k-)
M
K
E
Pe (2)
pe (c))).k))
II pe b(e), (c),,)11
log
pe (((),
==
i=
k=
M
K II pe B(o), | (
K
pe (r(e (o))).k))
log pe (blo).t)
pe (R(SS)
==1
K
E
(())
II pe
r
<MMMM K),
R((0)
i=k
(19)
where c(M, 2 0 is a constant term regarding M and K. Inspired by 36 we have the lower bound of ((0), ; :Tu 10)1-1 follows:
(B00) : ((c))-)))) log M - C(o).
;ru Lle), exp (f() (r))),;,())) (()))-1))
(20)
where
EvEr exp (fe) uyrth (()))-1))
The Eq. 20 iteratively searches K neighbors for the learner u under each knowledge concept c from step à 1 to K. k(c),k is the well-known InfoNCE loss function (36). Let ru (c),k-1 denote the
16

---

current context at step (k- 1) (i.e., the set of (A: - 1) neighbors selected from step 1 to (k (k-1)) 1)). by (( the affinity candidate learner in the (M - k) nonneighbor learners, Let Vy(e) denote the current
set of nonneighbor learners, and we hence have ((.)), V!!.). ((te (bLe).k (r (()))-)) is matching
function measuring the similarity between of nonneighbor b(.)*a and the current context ra (o) where the higher the scalar score means the higher likelihood (c), is a new neighbor.
Furthermore, we have k fic) ((())(;; ;ra (-)))-1) Thus, given the context of (k- neighboring learners (i.c., we have found (A: 1) neighbors for the learner u) and matching function (-), our goal following the Eq. 220 is to find a learner (())* from nonneighbor set Vz(() that can maximize the matching score f(e) (*) the k-th neighbor of u. In other words, p(G V, Z)Z) can be optimized through maximizing the matching score (-) from k = to K iteratively. Thereby, at each step k, we sort the scores of the nonneighbor learners and select the learner with the highest score to label ), (c).b-1 is updated to ra (c).k by absorbing (()).k
as k-th neighbor (. i.e.,
[(c), < arg max, (u; (1;r(())-1)) E V(le. After obtaining the k-th neighbor bu (c).k the context Tu
Based on the Eq. 15 Eq. 16 and Eq. (20). we have
M
K
maxlog MarlloggGGiI V
clo..
(21)
c=1u=lkm
which completes the proof.
Theorem 1. With Gaussian Mixture initialization from the Disentangled Cognitive Representa- tion Encoding (section 3.2), the Collaborative Representation Learning (section (.3) procedure equivalent to an expectation-maxiiizaaoo (EM) algorithm
(35) for the mixture model. In
particular, it converges to point estimate of (r( Je= that maximizes the marginal likelihood 1 (fal)) ab. (u.v) E e Go) c=l
fone
where az, equals or denoting whether learner is
collaborative neighbor of learner regarding concept not.
The proof is as follows.
Proof. The collaborative modeling process can be approximatively equivalent to an expectation- maximization (EM) algorithm for the mixture model. Let A {o.
{A(a)) )= where A(e)
{al):(uu : (u, E G where an, equals or O denoting whether learner v is a collaborative neighbor of learner regarding concept
not, which is type unknown factor in EM algo-
rithm. Given R {r(e) and Z = [
1=11 the EM algorithm maximizes the likelihood (A; = Σx4 1 (A, Z; R). Let q (A) is the distribution over A, we then have
log / (A; R) Σ (A) (Z,R)
(((A): I(A,Z, Z;
1(A) Z;R)
(22)
(A)) (A,,; (A)
(A)
+ +
I(A Z;
Let L (A)) denote (A) [(A.ZR)) ((A)
rewrite Eq. 22 as
log/ (Z L (R, (A)) DKL (g (A) (A Z: Z:R))
(23)
< L (R, (()),
where T. (R. (A)) is lower bound of (A; R) since the KI. divergence from [(A Z; R.) towards ((A) is non-negative.
17

---

At every E-step, the EM algorithm aims to search the optimal g(A) that tightens the lower bound, which ((A) is set to (A Z; R) since the KL divergence will become zero. Given that
(A Z; R) p(G V, V,Z),
(24)
and Property 3, the optimal (AA) that tightens the lower bound can be achieved through iteratively maximizing the following objective as
;ru ;r (110)A-1) Everoo exp ((o) (e (e: (2)))))111)
exp (Jre)
max ΣΣΣ
(25)
which performs the E-step.
After the E-step, an M-step performed maximize the lower bound ((A) w.r.. R with (AA) fixed found in the E-step. Given that
(c)
Σ
2!00
(26)
we optimize the (()) upon
(L().q(A)) to zero, which is actually performing the M-step. (rl))
Let ((A)* and re be the result of the k-th E-step and the k-th M-step, respectively, we then have
(A (R* - ((A)*)) + DKL (A())* || (A* Z; R R*-)))
= L (R* 10.4-.99/ -a(A)*)
< T. ((A)))))))
(27)
< L (((A)) + DKL (A 1 Z:R*))
R*)
Hence, this can prove that the likelihood will increase monotonically and be upper-bounded by zero the same time. Therefore, the algorithm converges, which completes the proof.
B Algorithm
To offer a more comprehensive description of Coral's structure, we outline the algorithm (see Algorithm D for the entire model, including four functions and main function.
C Dataset Description and Preprocessing
We conduct experiments on three real-world datasets, i.e., ASSIST [1], Junyi S and
NeurIPS2020EC 43 The statistics of these datasets summarized in Table 1 For all datasets,
we preserve the first-time axercise-answeriie cognitive diagnosis aligning with common settings used in previous related studies 40 The detailed information on datasets and preprocessing method are depicted as follows:
record for the same learner-question pairs to support
ASSIST (ASSISTments 2009-2010 "skill builder") This dataset is open dataset collected by the ASSISTmente online tutoring system? which has become one of the popular benchmark datasetss for cognitive diagnosis. We preserve learners with more than 30 practice records for ASSIST to guarantee that each learner has enough data for diagnosis.
Junyi (5) This dataset contains learner online learning logs collected from a Chinese online educational platform called Junyi Academy
3 Nowadays, Junyi is widely used in the evaluation online education tasks (47 8]. We randomly select 1,400 learners with more than 15 practice records from Junyi to guarantee that each learner has enough data for diagnosis.
httpa://sites http::///iieee.gggle..m www//stteaaeeeeecccmcccccoooeeee www.//itte/eiiittenssda.a.tt
attps://wwe http:://www.uuyiaccaacee.. junyiacademy,.o org/
18

---

Algorithm Coral Model
1: Input: {c) practice logs (Xu
2: function DISENTANGLED COGNITIVE, BAA.:AAAAAC..CTTTTAEEEEEEETTETTC
x.))
3: I/ Gaussian Mixture initialization of learner w
4: Zu 6- [z z
5: // Calculate ability of learner
6:
0y f (z. c 1,
7: // Reconsttuutt practice performance of learner
8: pe( 1.) 4 Ci (0 zu ) C 1,2, ..,, C
9: return BC P 1)), DKL Zu
end function
// Search neighbors for learner from all the learners WWA.E_CCLLAOORATVVECGAALLLI for 1,2, ..,, C do
12: function
Zu
14:
1/ Let be the removing operation
Vi0) <- V/
15:
16:
// Initial neighbor set of u, i.e., Ru,
kc).k + (u), where k = 0
17:
18:
Iteratively calculating the cognitive similarity scores between u and each learner in V(e) the initial step corresponds to k =
19:
for = =I...., K do for E Vi(c) do
20:
Scoreu, +- (1 (l0.)) // end for
21:
// Select k-th neighbor for u, denoted b (c).k (c). + arg max Score v vi(e)
23:
25:
// Update neighbors and nen-neighbor learners
ru + ru (0),b-1 (b(c),b)
le).k
Vi() t Vife) (.(o)k
27:
end for 29: end for
28:
return (G
31:
end function
32: function COLLABORATIVE_ GRAPH BRAPHGGMDELLLNNGEEEEE MODELING((GG))) z.)
for c 1,2, 34: Fı + (Z,
do
35: end for
36: return ru
37:
end function
38:
function DECODING_AND DECCONNGLNDDAEECOSSTUUTTOOOO
RECONSTRUCTION (c) I Zuo x.)
39:
Calculate ž, return Ex,,,
40:
41:
end function
42:
BEGIN MAIN FUNCTION:
43: Initialize z., learning rate ir, B, epoch <- 0, Tolal Epoch.
44: repeat
45:
for u 1,2, M do
BCE(s.,, ( 1)), D Zn <-
Coogniiive_Reppreeettttiiii
11 denotes omitting parameters
47:
(G Context-aware. Collaborative Beraph_Lerrnngt.... ..)
48:
BCE Pe(u.)))
Decoding Benpadiinnnnn_Reeecseeeeeeeeeeee Reconstruction((..)
49:
Calculate arg min L
50:
@ arg maxe L by Ir. VeL
51:
epoch + epoch + end for
19
53:
until epoch equals T T'otalEpoor

---

NeurIPS2020EC 43
This dataset is originated from NeurIPS 2020 Education Challenge, which provides learners' practice logs on mathematical questions from Eed We randomly select 1,000 learners with more than 15 practice records from NeurIPS2020EC to guarantee that each learner has enough data for diagnosis.
D Additional Experimental Results
Metric
Method
ACC
AUC
FI-score t
RMSE +
w/o KI.
0.693156
0.659914
0.803728
0.452067
w/o collar
0.667321
0.606090
0.786824
0.465231
w/ knn
0.708520
0.721339
0.794198
0.440430
Coral
0.709710
0.721823
0.810755
0.437818
Table 4: Ablation study of Coral on ASSIST.
D.1 Ablation Study
We additionally perform ablation studies to assess the impact of key components within Coral. The results in 4 the performances of Coral (setting = 5) under various conditions: without the KL term for encoding (w/o KI.,, without the collaborative aggregation during decoding (w/o collar), and replacing the collaborative graph construction procedure using a knn-based methods (w/ knn) used in [13 on the ASSIST dataset. These findings show the effectiveness of each key component in enhancing the overall performance of Coral.
D.2 Efficiency Improvement
We additionally implement three efficiency optimization strategics to further reduce the complexity of Coral. These strategies cannot theoretically guarantee optimal performance, but they can enhance applicability and scalability of Coral through empirical balancing of efficiency and accuracy. We refer them as Coral with n-sample, Coral with m-selections, and Coral with full-kit, as follows:
Coral with samaeeee: During the iterations of searching for neighbors, randomly sam- ple n subsets from all M learners to replace V in the original approach. This reduces computational efficiency from x to n x K, where n < M.
Coral with m-selections: Based the basic Coral, replace selecting neighbor per iteration with selecting m neighbors. This decreases computational efficiency from M x K MxK where m K. m
Coral with full-kit: A combination of Coral with n-sample and Coral with m-selections, further reducing computational efficiency from x K to ax where a & and m < K.
Following the three strategies outlined above, we conduct several experiments on the Junyi dataset, with K 40, to assess prediction performance. The results are summarized in the Figure 6 These experimental results demonstrate Coral's potential improve computational efficiency while maintaining acceptable performance, as evidenced by the varying levels of accuracy achieved with different optimization strategies.
E
Broader Impact and Limitation
This research delves into modeling human cognitive states within the realm of intelligent education. The proposed Coral model significantly enhances the diagnostic accuracy implicit learners' knowl- edge states. This improvement not only provides effective insights for online personalized tutoring services, such as question recommendations but also lays the foundation for further research this area. Moreover, the automatic construction strategy for collaborative connections among learners
https://eedi.com/
20

---

0.88
0,800
0,880
0.88
Fl score
0.700
0.877
0.875
0.87
0.86
0.870
0.850
500
772 20
10
1811
10008 8008:10 5008:20 200 20 & m
71
Figure 6: The prediction performance of the improved model is illustrated, with the orange bar representing the performance of the original Coral, which achieves the highest score.
offers valuable insights that contribute subsequent investigations in this field. Lastly, anticipate that the proposed techniques can be extended to other domains, including but not limited to interest modeling and social network modeling. Although method is effective both theoret- ically and empirically, suffers from computational inefficiencies. We have explored preliminary optimization strategies in the Appendix D.2 and will focus on improving computational efficiency in future research. In addition, future research plans to consider issues of fairness [51] [2] and explore the integration of large language models and multi-modal knowledge enhance interpretabil- ity [27 [29 (3). In essence, our work is dedicated to advancing intelligent education and deepening the understanding human cognitive proficiency. cannot cause negative effects. We anticipate its crucial role in fostering progress in both pertinent technologies and societal advancements.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction clearly claim our task (scope), contributions and
solutions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We provide all the proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
We provide a github repository to publish our work (ttts::/ggihuub.oom/bbggzz
uste/Coral).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/geee guwde//ddee/// Cote/ooodeeeeeseeeeeeee wwwprbmmsssioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In the experimental setup (Section 4.1) we provide all the setups.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: our paper, we provide the average scores. Given the impending submission deadline, we will provide error bars during the rebuttal phase by updating the README file in the anonymous GitHub repository.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in the implementation details (Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
These information is discussed in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cite the original papers or website links about the dataset and open-source codes.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "4eDEmw2q38", "reviewer": "Reviewer_vXdf", "summary": "The paper presents a novel model called Coral, aimed at enhancing cognitive diagnosis in educational contexts by leveraging collaborative signals among learners. The authors argue that learners with similar cognitive states often exhibit comparable problem-solving performances, and thus, understanding these collaborative connections can significantly improve the diagnosis of individual cognitive states.\nKey Contributions - \n1. Disentangled Representation Learning: Coral introduces a disentangled state encoder to effectively separate the cognitive states of learners. This approach allows for clearer interpretations of individual knowledge proficiencies by disentangling the various cognitive factors that influence learning performance.\n2. Collaborative Graph Construction: The model incorporates a context-aware collaborative representation learning mechanism that dynamically constructs a graph of learners. This graph is built by identifying optimal neighbors based on their cognitive states, facilitating the extraction of collaborative information.\n3. Co-disentanglement Process: Coral achieves co-disentanglement by aligning the initial cognitive states with collaborative states through a decoding process. This integration enhances the model's ability to reconstruct practice performance while providing a more nuanced understanding of learners' cognitive states.\n4. Empirical Validation: The paper reports extensive experiments demonstrating that Coral outperforms state-of-the-art methods across several real-world datasets, highlighting its effectiveness in accurately diagnosing cognitive states through collaborative learning.\nOverall, this work addresses significant gaps in existing cognitive diagnosis methods by simultaneously modeling both individual and collaborative learning information, paving the way for more personalized educational interventions.", "questions": "Dataset Diversity: What criteria did you use to select the datasets for your experiments? How do you ensure that the datasets adequately represent the diversity of learning environments and learner behaviors?\n\nBaseline Comparisons: Can you provide more details on the baseline models you compared Coral against? Specifically, how do these models differ in their approach to cognitive diagnosis, and why were they chosen as benchmarks?\n\nHyper-parameter Sensitivity: Did you conduct a sensitivity analysis on the hyper-parameters of the Coral model? If so, what were the findings, and how do hyper-parameter choices impact the model's performance?\n\nInterpretability of Results: How can educators and practitioners interpret the outputs of Coral in practical settings? Are there specific examples or case studies that illustrate how the model's insights can be applied in educational contexts?\n\nOverfitting Concerns: What measures did you take to prevent overfitting in your model? Can you provide validation metrics or results from cross-validation to support the generalization capabilities of Coral? In your \"data_loader.py\" script you have provided a check for validation set on line 60 (\"val_set.json\") and also in the paper on line 279 under the Experimental setup --> Settings section you claim \"split all the datasets with a 7:1:2 ratio into training sets, validation sets, and testing sets\". However, no validation metrics or validation sets seem to be used.\n\nEthical Considerations: Have you considered the ethical implications of using collaborative cognitive diagnosis models in educational settings? How do you address potential concerns regarding data privacy and algorithmic bias?", "limitations": "The authors have not addressed the limitations and potential negative societal impact of their work, but there are still areas that could be improved. They could further elaborate on the generalizability of their findings beyond the specific educational contexts covered in the experiments. The authors do not explicitly discuss the potential negative societal impact of their work. This is an important aspect that should be addressed, as cognitive diagnosis models can have significant implications for learners, educators, and educational institutions. An in depth explanation should be provided pertinent to limitations on dimensions like : Data Privacy, Algorithmic Bias, Misuse of Diagnostic Information etc.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\nThe paper introduces a unique approach to cognitive diagnosis by integrating collaborative signals among learners, which has not been extensively explored in existing literature. The concept of using a disentangled representation to model cognitive states while simultaneously capturing collaborative learning dynamics is innovative. Prior models either focused solely on individual cognitive factors or did not adequately account for collaborative interactions. I think it reflects a creative synthesis of ideas from domains of educational psychology and collaborative filtering.\n\nQuality\nThe authors provide a comprehensive framework that includes a disentangled state encoder, a context-aware collaborative graph learning mechanism, and a co-disentanglement process. The empirical validation through extensive experiments on real-world datasets demonstrates the robustness of the model. The results indicate significant improvements over state-of-the-art methods, showcasing the effectiveness of the proposed approach. The paper also includes detailed mathematical formulations and algorithms, which enhance the reproducibility of the research.\n\nClarity\nThe paper is well-structured, with a logical progression from the introduction of the problem to the presentation of the model and its components. The use of clear diagrams and figures aids in the understanding of complex concepts, such as the collaborative graph construction and the disentanglement process. The performance comparison table is extremely effective in showcasing how the contribution is significantly better than the state-of-the-art methods on multiple datasets.\n\nSignificance\nThe work definitely shows potential impact on the field of intelligent education. By improving the accuracy of cognitive diagnosis, the Coral model can facilitate more personalized learning experiences. The integration of collaborative learning dynamics addresses a critical gap in existing models and could influence the design of educational technologies and interventions, making this research valuable not only academically but also practically in educational settings.", "weaknesses": "1. Limited Generalizability of Experiments\nWhile the paper demonstrates the effectiveness of the Coral model through extensive experiments on several real-world datasets, the choice of datasets may limit the generalizability of the findings. The experiments primarily focus on specific educational contexts, which may not capture the diverse learning environments and learner behaviors present in broader educational settings.\nActionable Insight: Future work should consider testing the model on a wider variety of datasets, including those from different educational domains (e.g., K-12, higher education, online learning platforms) and varying learner demographics. This would help validate the robustness of the model and its applicability across different contexts.\n2. Lack of Comparison with Baseline Models\nThe paper claims significant improvements over state-of-the-art methods, but it does not provide a comprehensive comparison with a more wider range of baseline models. Especially collaborative filtering methods and other machine learning approaches that could serve as alternatives.\nActionable Insight: The authors should include a more diverse set of baseline models in their experiments to provide a clearer picture of Coral's performance. Additionally, discussing the strengths and weaknesses of these models in comparison to Coral would enhance the reader's understanding of the model's contributions.\n3. Insufficient Exploration of Hyper-parameter Sensitivity\nThe model's performance could be sensitive to various hyper-parameters, such as the number of neighbors in the collaborative graph or the dimensions of the disentangled representations. However, the paper does not provide a thorough analysis of how these hyper-parameters impact the model's performance.\nActionable Insight: Conducting a sensitivity analysis on key hyper-parameters would help identify optimal settings and provide insights into the model's robustness. This could involve systematic experiments varying one parameter at a time while keeping others constant, followed by a discussion of the results.\n4. Explainability and Interpretability\nWhile the paper emphasizes the importance of disentangled representations for explainability, it lacks a detailed exploration of how the model's outputs can be interpreted in practical educational contexts.\nActionable Insight: The authors should include case studies or examples demonstrating how the model’s outputs can be interpreted and applied in real educational settings. This could involve visualizing the relationships between cognitive states and learner performance or providing guidelines for educators on how to utilize the model's insights.\n5. Potential Overfitting Concerns\nThe complexity of the Coral model, with its multiple components and parameters, raises concerns about potential overfitting, especially if the training datasets are not sufficiently large or diverse. The paper does not address how the model's complexity is managed or validated against overfitting.\nActionable Insight: Providing validation metrics would strengthen the paper's claims regarding the model's generalization capabilities. \n6. Limited Discussion on Ethical Considerations\nThe paper does not address potential ethical implications of using collaborative cognitive diagnosis models in educational settings, such as data privacy concerns or the impact on learner autonomy and agency.\nActionable Insight: Including a section on ethical considerations would enhance the paper's comprehensiveness and relevance. This could involve discussing how to ensure data privacy, the implications of algorithmic bias, and strategies for maintaining learner agency in personalized educational interventions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive remarks and insightful feedback! We answer your comments and questions below. Please let us know if you have additional questions.\n> **Weakness 1:** Concerns on data diversity.\n\nThank you for your concerns about the datasets.\n\nWe would like to emphasize that our datasets are diverse, covering both K-12 scenarios (ASSIST) and online learning scenarios (Junyi and NIPS2020EC). These datasets are also representative, as they are benchmark datasets in CD research. Nearly all top-tier work (e.g., [1][2][3]) uses these datasets for experiments.\n\nFollowing your suggestion, we further add a new dataset for experiments. **The experimental results are detailed in the Public Response** (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), supporting our model's effectiveness across diverse domains.\n> **Weakness 2:** Add baselines.\n\nThank you for your feedback.\n\nWe add four related baselines and compare them across three datasets. **The experiments settings, results and model comparison analysis are listed in Public Response** （https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), demonstrating the superiority of Coral over these baselines. We will add complete results and detailed model comparison analysis in a revised version.\n\n> **Weakness 3 and Question 3:** Concerns on hyper-parameter sensitivity experiments.\n\nThank you for your concern on hyper-parameter sensitivity experiment.  We would like to emphasize that our paper already includes hyper-parameter experiments (Fig.3(c)) on the number of neighbors $K$.\n\nAdditionally, we conduct experiments on different embedding dimensions $d$, with values of 124, 200, 256, 521, and 1024. **The experimental results of Coral on ACC are listed in Public Response**（https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), demonstrating that within an appropriate range, the dimension $d$ does not significantly affect Coral's performance. We will include the complete results in the revised version.\n> **Weakness 4 and Question 4:** Interpretability concerns.\n\nWe have already provided an interpretable example in Appendix E. This example demonstrates how Coral reasonably infers each learner's future performance by referencing the cognitive states of similar learners, thereby providing educators with a basis for their inferences.\n\nAdditionally, we would like to clarify that Coral's interpretability comes from its ability to use cognitive states of similar learners to predict how learners will perform on unfamiliar problems. Disentanglement is a technique to enhance the modeling of collaborative relations, not the source of interpretability. We infer that you might be misled by \"co-disentanglement\" in line 77, which means aligning learner representations from two perspectives to optimize collaborative relationship modeling, thereby enhancing interpretability. We will make this insight clearer in a revised paper.\n> **Weakness 5 and Question 5:** Overfitting issue.\n\nThank you for the insightful feedback.\n\nWe use an early stopping strategy to mitigate overfitting, stopping training when model's ACC (validation metric) on validation set stabilizes. \nEach model is trained 5 times with a repartition of data, and the average score is reported, similar to 5-fold cross-validation. We provide Coral's ACC error bars for 5 runnings on ASSIST, Junyi, and NIPS2020EC, $\\pm 0.00023$, $\\pm 0.00017$, and $\\pm 0.00031$, ensuing robustness.\n\nIn fact, our paper verified Coral's robustness in both sparse and diverse scenarios:\nFig. 3(a) shows experiments on data sparsity, showing Coral's robustness in handling overfitting with insufficient data. Fig. 3(b) shows cold start experiments where new knowledge is introduced to the test data, increasing test scenario diversity, proving the model's robustness even when the training data lacks diversity.\n\n> **Weakness 6 and Question 6:** Ethical concerns.\n\nThank you for raising these important ethical concerns.\nTo address data privacy, federated learning or cross-domain modeling can be integrated to enhance Coral's privacy-preserving capabilities. For fairness, we will ensure equitable outputs by identifying and mitigating the impact of sensitive attributes on model predictions. Additionally, LLM-based agents are expected to maintain learner autonomy and agency. We will include a discussion of these ethical considerations in the revised version of the paper.\n\n> **Question 1:** Dataset selection criteria.\n\nThe data selection criterias include:\n1. Authenticity: We use 3 real-world, open-source datasets, as detailed in Appendix D. These datasets reflect real learner interactions.\n2. Diversity: Our datasets span different educational domains. The ASSISTments dataset targets K-12 education, while the Junyi Academy and NIPS datasets focus on online learning environments with global participants. This ensures coverage of varied learning contexts.\n3. Representativeness: These datasets are established benchmarks in CD, widely used in current research for consistent evaluation.\n\nObviously, our datasets adequately represent diversity of learning environments and learner behaviors.\n\n> **Question 2:** Baseline descriptions.\n\nWe selected baselines from three aspects:\n\n1. Educational Psychology Methods: IRT and MIRT, foundational in cognitive diagnosis,diagnose learners' cognitive states using psychology priors.\n\n2. Machine Learning Approaches: PMF, NCDM, KaNCD, and RCD represent key methods using machine learning techniques to model student-item interactions.\n\n3. Variational or Disentangled Methods: DCD and ReliCD, which are related to our solution.\n\nWe will add detailed descriptions of each baseline in the Appendix of the revised version.\n\n---\n\n[1] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.\n\n[2] Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework. SIGKDD'22.\n\n[3] RCD: Relation map driven cognitive diagnosis for intelligent education systems. SIGIR'21.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YiraVnLIuL", "reviewer": "Reviewer_t9qg", "summary": "This paper introduces Coral, a novel model for collaborative cognitive diagnosis with disentangled representation learning, that can help in developing intelligent education systems. The key contributions are incorporating collaborative information among learners to improve cognitive state diagnosis by using disentangled representation learning to model cognitive states, and then using context-aware graph construction method to identify implicit collaborative connections. Coral demonstrates superior performance over state-of-the-art methods across several real-world datasets, showing particular strength in sparse and cold-start scenarios. The approach offers improved interpretability and adaptability in modeling learners' cognitive states, potentially advancing personalized tutoring and understanding of human learning processes.", "questions": "* How sensitive is the model performance to changes in key hyperparameters such as the number of neighbors (K) and the beta value? Adding a sensitivity analysis for these hyperparameters in the appendix can help assess the model's robustness and guide practitioners in tuning it for different datasets.\n\n* Can you provide a breakdown of Coral's performance as the percentage of questions answered by new learners varies (e.g., 0%, 5%, 10%, 20%)? This would give a clearer picture of the model's efficacy in true cold-start scenarios.\n\n* Is it possible to show how disentangled factors correspond to specific knowledge areas (since mapping to interpretable cognitive factors are not easy) across different datasets? This would strengthen the claim of improved interpretability over existing methods.", "limitations": "* The authors briefly mention computational inefficiencies as a limitation in Section 5. They also provide some preliminary optimization strategies in Appendix F.2, which is positive. However, it would be valuable to also expand it to other concerns like any potential bias in the modeling process, limitations in the datasets used or generalizability to other educational contexts, any limitations in the disentanglement approach to capture all relevant cognitive factors, etc\n\n* Also the authors do not fully address potential ethical concerns and mitigation strategies related to privacy and fairness in cognitive modeling. E.g., privacy concerns related to modeling learner cognitive states, potential bias in the model that could disadvantage certain groups of learners, etc", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper introduces an intriguing dual-perspective approach to cognitive diagnosis. By examining both inner-learner and inter-learner aspects, it offers a more comprehensive view of cognitive states. The integration of context-aware collaborative graph learning with disentangled representation learning is particularly noteworthy, as it potentially allows for more nuanced insights into learner interactions.\n\n* The empirical validation using three diverse datasets (ASSIST, Junyi, NIPS2020EC) lends credibility to the model's effectiveness. Its superior performance across various metrics compared to existing models is encouraging. Moreover, the model's resilience in scenarios with sparse data and cold start environments, suggests practical applicability in real-world educational settings.\n\n* From a structural standpoint, the paper presents its ideas coherently. The authors have taken care to provide detailed explanations and mathematical formulations for each component of their model. The inclusion of key equations and visual aids, such as t-SNE plots, effectively illustrates the model's inner workings, particularly in terms of disentanglement and neighbor selection processes.\n\n* This research makes a valuable contribution to the field of cognitive diagnosis. By enhancing both the accuracy and interpretability of learner cognitive state assessments, it opens up new avenues for personalized education. The model's broad applicability across various educational datasets, coupled with its novel methodologies like context-aware graph learning and co-disentanglement, provides a solid foundation for future research in educational data mining and related fields.", "weaknesses": "* The model suffers from computational inefficiencies, which can limit its practical applicability, especially in large-scale educational settings. This issue is transparently acknowledged by the authors too, which is positive. \n\n* While the paper aims to achieve disentangled representations, the evaluation of disentanglement quality remains simple and limited. It will be valuable to add more ablation studies related to disentanglement quality and run qualitative analysis of how individual latent dimensions correspond to interpretable knowledge concepts across different datasets.\n\n* The paper doesn't adequately address potential limitations of the collaborative approach, such as negative transfer or performance in edge cases. It will be valuable to discuss scenarios where the collaborative approach might not be beneficial or could lead to negative transfer. E.g., a highly advanced learner who struggles in one specific area may be grouped with generally low-performing learners", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive remarks and insightful feedback! We address your comments and questions below. Please let us know if you have additional questions.\n\n> **Weakness 1:** Concerns on computational efficiency.\n\nThank you for acknowledging our discussion of the model's efficiency limitation.\n\nIn fact, we also provide alternative efficient solutions in Appendix F.2 to address this limitation, where we design three practical strategies to improve Coral's computational efficiency. Although these solutions may not guarantee theoretical optimality, the experimental results demonstrate their effectiveness.\n\n> **Weakness 2:** Disentanglement evaluations.\n\nThank you for your suggestion on evaluating disentanglement quality. \n\nOur disentanglement involves decoupling each learner into $C$ vectors (as vector-level disentanglement) and using KL divergence at the element level to encourage independence between each dimension of each vector based on Eq.5 (as dimension-level disentanglement). The current paper assesses dimension-level disentanglement quality by: (1) quantitative correlations between dimension independence level (IL) and prediction performance (Fig.3(a)), (2) qualitative embedding visualizations (Fig.4(b)), and (3) an ablation study of dimension-level disentanglement by discarding the KL term (\"w/o KL\" in Tab.2).\n\nTo further address your concerns, we add two additional studies:\n\n1. We set different values for $\\beta$ to explore the effect of varying disentanglement strengths (KL term in Eq. 5) on IL (i.e., dimensions' independence level of disentangled vectors) and performance, inspired by [1]. The experimental results on ASSIST are as follows:\n\n|Value of $\\beta$|0.25|0.5|0.75|1|\n|-|-|-|-|-|\n|ACC|0.71164|0.72533|0.72130|0.72017|\n|AUC|0.74318|0.77312|0.77151|0.76817|\n|IL|0.82173|0.85031|0.87263|0.87342|\n\nThe results indicate that as the disentanglement strength increases, the IL metric gradually increases, demonstrating that each dimension of learner representations becomes more independent. The model's predictive performance initially improves and then slightly decreases after reaching a threshold ($\\beta = 0.5$). This suggests that both too low and too high disentanglement strengths can affect the model's performance.\n\n2. We conduct ablation studies to remove the vector-level disentanglement by representing each learner with a single $C$-dimensional vector instead of the original $C$ d-dimensional vectors, denoted as \"w/o VecDis\". The experimental results on NIPS2020EC are as follows:\n|Model|ACC|AUC|RMSE|\n|-|-|-|-|\n|w/o KL|0.71026|0.78990|0.44382|\n|w/o VecDis|0.71523|0.78331|0.43412|\n|**Coral**|**0.71622**|**0.79103**|**0.43200**|\n\nThese results demonstrate the effectiveness of vector-level disentanglement. We will report the full results in a revised paper.\n\n> **Weakness 3:** How to address potential negative transfer impact in the collaborative graph.\n\nOur model addresses potential negative transfer through attention weights (see Eq.8 for $s_{u,v}$), as discussed on lines 230-233. When computing attention between learners $u$ and $v$, both cognitive similarity and contextual similarity are considered. This approach ensures that attention is low for non-similar and non-collaborative neighbors, thus mitigating negative transfer impacts.\n\nWe conduct additional ablation experiments to further support this clarity:\n\n|ASSIST|ACC|AUC|RMSE|junyi|ACC|AUC|RMSE|NIPS2020EC|ACC|AUC|RMSE|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|w/o attention|0.71996|0.75023|0.42533|w/o attention|0.82172|0.82969|0.37112|w/o attention|0.71173|0.78334|0.43430|\n|**Coral**|**0.72533**|**0.77312**|**0.42034**|**Coral**|**0.82534**|**0.83503**|**0.36403**|**Coral**|**0.71622**|**0.79103**|**0.43200**|\n\nThe model without attention (w/o attention) performs worse compared to Coral, demonstrating the effectiveness of attention in managing negative transfer. We will include the full results in the revised paper and emphasize this technical advantage.\n\n> **Question 1:** Concerns on hyper-parameter sensitivity experiments.\n\nThank you for your feedback. \n\nCurrent paper already included hyper-parameter experimental results (Fig.3) and findings  (lines 323-328) on the number of neighbors $K$.\n\nAdditionally, **we conduct experiments on different embedding dimensions $d$, as detailed in the Public Response** (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=4eDEmw2q38).\n\n\n> **Question 2:** Data sparsity impacts.\n\nCurrent paper provides experimental results of Coral with 20%, 40%, 60%, and 80% sparsity (Fig.3(a)). To fully address your concerns, we further conduct experiments with 5% and 10% sparsity as follows:\n|ACC|IRT|MIRT|NCDM|KaNCD|RCD|**Coral**|\n|-|-|-|-|-|-|-|\n|5%|0.57073|0.59274|0.609715|0.60332|0.64113|**0.64933**|\n|10%|0.64231|0.66774|0.67752|0.67321|0.66432|**0.67933**|\n\nThe experiments show the robustness of Coral. For 0% sparsity, current CD research that rely on practice data cannot effectively handle this scenario due to the need for initial data from each learner.\n> **Question 3:** Interpretability.\n\nEach learner is disentangled into $C$ vectors, so that the disentanglement representation can naturally correspond to a specific knowledge, e.g., $z_{u}^{(c)}$ is $u$'s cognitive state of knowledge $c$.\n\nAn additional interpretability case study is given in Appendix E.\n\n> **Limitations:**\n\nThank you for your feedback. We supplement with a PISA 2015 data to expand dataset diversity (**see the Public Response**). To address data privacy, federated learning or cross-domain modeling can be integrated to enhance Coral's privacy-preserving capabilities. For fairness, we will ensure equitable outputs by identifying and mitigating the impact of sensitive attributes on model predictions. In the future, we will also design more powerful disentanglement algorithms.\n\nWe will include a discussion on these technical limitations and ethical considerations.\n\n---\n[1] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bPezibX2d7", "reviewer": "Reviewer_bUz3", "summary": "The authors propose Coral, a Collaborative Cognitive Diagnosis model with Disentangled Representation Learning, aimed at improving our understanding of human learning by leveraging collaborative signals among learners. By disentangling cognitive states and dynamically constructing a collaborative graph, Coral captures implicit connections between learners, leading to more accurate cognitive diagnoses. The model's innovative approach involves an iterative process of encoding, collaborative representation learning, and decoding, which significantly enhances its performance over existing methods. Extensive experiments demonstrate Coral's superior effectiveness in real-world datasets, making it a promising tool for intelligent education systems.", "questions": "See weaknesses.", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Coral’s use of disentangled representation learning and dynamic graph construction is a novel approach in the field. This allows the model to capture implicit collaborative signals among learners, leading to more accurate and insightful cognitive diagnoses.\n\n2. The extensive experiments and superior performance across multiple real-world datasets highlight Coral’s practical utility.\nBy disentangling cognitive states and integrating collaborative information, Coral offers greater explainability and controllability in cognitive diagnosis. This may make it easier for educators and researchers to understand and interpret the learning patterns and cognitive states of students.", "weaknesses": "1. Coral's effectiveness relies heavily on the quality and quantity of learner data available. In scenarios where such data is sparse or noisy, the model's performance may not be as robust.\n\n2. While the model shows great promise, its computational requirements for dynamically constructing collaborative graphs and learning representations might pose scalability issues, especially in very large educational environments.\n\n3. There may lack some highly related baselines/papers for the cognitive diagnosis. \n\n[1] https://arxiv.org/pdf/2002.00276\n[2] https://ieeexplore.ieee.org/document/10027634\n[3] https://arxiv.org/pdf/2005.13107", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and constructive feedback! We answer your comments and questions below. Please let us know if you have additional questions.\n\n> **Weakness 1:** Robustness under data sparsity or noise scenarios.\n\nThank you for highlighting the robustness of CD in data sparsity or noise scenarios.\n\n**In fact, we have presented experiments under data sparsity and cold start scenarios in Fig3(a) and (b),** which demonstrate that Coral exhibits better robustness compared to the baseline models.\n\n**Additional experiments with noisy data:**  \nDue to the lack of noisy labels, current CD models often struggle with data noise issue. Educational data noise is typically caused by implicit learner errors or guesses, which are difficult to quantify. Therefore, existing methods often treat noise as latent trainable parameters (e.g., [1][2]), but they fail to validate the model's ability to handle noise since the extent of noise in the dataset is unknown. To address your concerns about noisy data, we explicitly simulate noisy training scenarios by randomly flipping the performance labels of 5% of learners in each training dataset (0 -> 1, 1 -> 0). The test data remains unchanged and is considered noise-free, allowing us to assess whether models trained on noisy data maintain robustness on a noise-free dataset. Below are some of the results:\n\n|ASSIST with 5% Noise|ACC|AUC|RMSE|junyi with 5% Noise|ACC|AUC|RMSE|NIPS2020EC with 5% Noise|ACC|AUC|RMSE|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|IRT|0.69577|0.70034|0.45321|IRT|0.79632|0.79314|0.37674|IRT|0.69221|0.74732|0.45621|\n|MIRT|0.71233|0.73674|0.43551|MIRT|0.79314|0.79667|0.38117|MIRT|0.70443|0.71074|0.44152|\n|PMF|0.70544|0.74327|0.45217|PMF|0.78512|0.78617|0.38211|PMF|0.703774|0.77662|0.44513|\n|NCDM|0.70522|0.73316|0.44021|NCDM|0.78334|0.77513|0.40207|NCDM|0.70217|0.77441|0.45122|\n|KaNCD|0.7127|0.75672|0.43641|KaNCD|0.80332|0.80017|0.37724|KaNCD|0.70107|0.77347|0.44102|\n|RCD|0.70032|0.73147|0.44263|RCD|0.81163|0.81522|0.37233|RCD|0.70732|**0.78533**|0.44324|\n|**Our Coral**|**0.72174**|**0.75807**|**0.43217**|**Our Coral**|**0.81873**|**0.81751**|**0.37027**|**Our Coral**|**0.70965**|**0.78533**|**0.44132**|\n\nThese results clearly show that Coral exhibits greater robustness compared to baselines in noisy scenarios.\nWe will add all experimental results and discuss the impact of noisy data in the limitations section of the revised paper.\n\n> **Weakness 2:** Limited computational efficiency.\n\nThank you for your feedback on computational efficiency.\n\nOur paper has presented three effective solutions to improve the efficiency of Coral in Appendix F.2. These strategies significantly enhance Coral's computational efficiency. While they may not guarantee theoretical optimality, the experimental results demonstrate substantial improvements, which prove Coral can be used to some large-scale scenarios. For real-time response, co-disentangled cognitive state (line 249) can be precomputed and stored, with only retrieval needed during online computation. We will continue to optimize the model efficiency in future studies.\n\n> **Weakness 3:** Add three related baselines.\n\nThank you for pointing out this issue and giving us the opportunity to supplement baselines.\n\nWe conduct experiments for the three baselines you suggested on three datasets. **The experimental setups, results and analysis are available in the Public Response** (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66).\nExperimental results demonstrate that Coral consistently outperforms these baselines across all datasets.\nWe will add the complete experimental results and introduce these significant related works in the revised version.\n\n---\n\nWe hope these responses comprehensively address your points and enhance your understanding of our work. We greatly appreciate your constructive feedback and remain open to any further inquiries or suggestions for improvement. Thank you for your support and valuable contribution to improving our paper.\n\n---\n\n[1] Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Statistics Reference Online, 2014.\n\n[2] https://ieeexplore.ieee.org/document/10027634", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSMRU72QNX", "reviewer": "Reviewer_UPTU", "summary": "This paper explores the Coral model, which combines disentangled representation learning with collaborative signals to enhance cognitive diagnosis in intelligent education. It focuses on identifying implicit collaborative links among learners to improve understanding of their cognitive states. The model integrates a context-aware collaborative graph learning mechanism to simulate the collaborative relationships between learners based on their cognitive states. By incorporating collaborative information into the disentangled cognitive states of learners, the model achieves collaborative disentanglement. Experiments on real datasets demonstrate Coral's performance compared to existing methods.", "questions": "1. The motivation of this paper is to consider collaborative information among learners. What are the specific differences between the collaborative information proposed in this paper and the collaborative information in recommendation systems?\n2. Do the ASSIST and NIPS2020EC datasets have similarities and prerequisite relationships among the knowledge concepts that support RCD?\n3. In the example of question recommendation in Appendix E, are these five questions answered by learners in comparable situations (i.e., without acquiring new knowledge during the answering process)?", "limitations": "The authors discuss the limitations of the computational efficiency of the paper. Meanwhile, they also give 2 simple yet effective solutions to address current limitations in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper will undoubtedly attract attention and excitement from researchers in various fields, e.g., personalized learning, cognitive diagnosis, knowledge tracing, and performance prediction. It represents an exploration of modeling user collaboration in educational settings. While the modeling of collaborative relationships among users is well-established in other fields (e.g., social networks, recommendations), the technological development in online education has lagged. The research line on modeling learner relationships is almost non-existent. I hope this work will advance the intelligent education community.\n2. Compared to previous cognitive diagnosis research, Coral introduces a new method that combines disentangled representation learning with collaborative signals, holding great potential for enhancing cognitive diagnosis.\n3. Theoretical Contribution: The authors have meticulously derived the optimization form of the lower bound of cognitive representation encoders, the approximate optimization objective of collaborative relationship modeling, and the model convergence. The theoretical proofs are reasonable and solid.\n4. The experimental results on real datasets in normal, data-sparse, and cold-start scenarios indicate that Coral outperforms existing methods, demonstrating its effectiveness in improving diagnostic accuracy.\n5. The writing is good. The proposed problem and model are described clearly.", "weaknesses": "I expect that this paper will comfortably clear the bar for acceptance. However, I believe two main issues should be addressed first. Due to these reasons, I have set my score at a relatively borderline level but anticipate increasing it after the authors' rebuttal.\n\n1. If two students have fewer interaction records, leading to increased similarity in their interaction patterns, they might exhibit different cognitive levels. Using the method outlined in the paper to learn their representations may yield similar results, but this might not accurately reflect the true situation.\n2. The paper employs GNN to learn network representations, but the graph-based baseline comparison only includes RCD and does not include some recent graph-based cognitive diagnosis methods, such as [1].\n\nOther/Minor issues (Optional):\n\n1. In the second paragraph, the authors describe an example: experienced teachers can summarize students' similarities. Similarly, could large language models be used instead of real human teachers to construct collaborative graphs? I look forward to the authors discussing the integration with LLMs. Although this does not affect the readability and acceptance of the paper, I believe readers will be very interested.\n2. In Table 6, \"w/ full-kit (m,m)\" should be \"w/ full-kit (n,m).\"\n3. In Table 1, why are there two results for DCD on the NIPS2020EC dataset? It seems like results from two different runs.\n\n```\n[1] Wang, Shanshan, et al. \"Self-supervised graph learning for long-tailed cognitive diagnosis.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and constructive feedback! We answer your comments and questions below. Please let us know if you have additional questions.\n\nFor the weaknesses:\n---\n> **Weakness 1:** How to mitigate bias issues in establishing learner collaboration relationships with insufficient interaction data.\n\nThank you for raising this issue. Our variational encoder can mitigate this problem by magnifying subtle differences in the latent feature space. In fact, existing CD models that rely solely on student interaction data typically face this challenge[1].\n\n> **Weakness 2:** Add a graph-based baseline.\n\nThank you for pointing out the absence of a graph-based baseline, i.e., SCD. **We have conducted learner performance prediction tasks using SCD on three datasets, as shown in our Public Response** (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66).\nExperimental results prove the consistent superiority of Coral compared with SCD. We will report the complete results in the revised version.\n\nFor the concerns:\n---\n> **Concern 1:** How to use LLMs for constructing the collaborative graph?\n\nThank you for your forward-thinking perspective. We believe that LLMs have significant potential in constructing educational collaboration graphs by designing agents that can identify collaborative learners based on educational prior knowledge and psychological assumptions. Existing studies [2,3,4] have demonstrated the use of LLMs to predict and analyze graphs in text, which can provide valuable insights for constructing collaboration graphs among learners in educational settings. We consider this a valuable research direction and plan to explore it in future studies.\n\n> **Concern 2:** Some typos.\n\nThank you for pointing out this issue. We will correct the symbols in the revised paper.\n \n> **Concern 3:** Confusion on baselines with duplicate names.\n\nWe apologize for the confusion. The first line of the DCD results does not include the tree structure, aligning with the settings for the ASSIST and Junyi datasets, which do not provide tree structures (as described in the \"Settings\" section). The second line of DCD results includes the tree structure. We will correct this in the revised version and label the second set as DCD_tree.\n\nFor the questions:\n---\n> **Question 1:** Explain the difference of \"collaboration\" between education and recommendation.\n\nThe core idea in both contexts is to model similarities between entities to aid predictions. In the educational scenario, \"collaboration\" considers cognitive similarities between learners, aiming to diagnose the learner's cognitive state rather than predict the final outcome of answering questions. In a recommendation system, \"collaboration\" models the similarity of preferences among users, aiming to predict similar users' decisions on the same item through similarity.\n\n> **Question 2:** How to construct conceptual relations from ASSIST and NIPS2020EC datasets for running RCD?\n\nThe two datasets do not provide the concept maps required by RCD. Instead, we constructed the conceptual relations from exercise data using the statistical tool (https://github.com/bigdata-ustc/RCD) proposed by RCD.\n\n> **Question 3:** Details about the case study.\n\nThe example questions in the case study are randomly selected without a specific recommendation strategy, aiming to showcase the model's performance and interpretability in cold-start scenarios. Regarding \"comparable situations\", note that during inference, all model parameters remain fixed, meaning learners do not gain new knowledge. We will provide more details in the revised version.\n\n---\n\n[1] Towards the Identifiability and Explainability for Personalized Learner Modeling: An Inductive Paradigm[C]//Proceedings of the ACM on Web Conference 2024. 2024: 3420-3431.\n\n[2]  Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education[J]. arXiv preprint arXiv:2402.14293, 2024.\n\n[3] Large Language Model (LLM)-enabled Graphs in Dynamic Networking[J]. arXiv preprint arXiv:2407.20840, 2024.\n\n[4] Canonicalize: An LLM-based Framework for Knowledge Graph Construction[J]. arXiv preprint arXiv:2404.03868, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lP7fkD6Qot", "reviewer": "Reviewer_k7ko", "summary": "This paper presents a Collaborative Cognitive Diagnosis model, named Coral, which incorporates a disentangled state encoder, collaborative graphs, and a state decoder. The goal of Coral is to model collaborative and disentangled cognitive states using representation learning techniques.", "questions": "1. What is the performance impact if the concepts are not disentangled? Currently, there is insufficient motivation for integrating collaborative signals into the disentangled representations.", "limitations": "There is no significant negative societal impact identified in this work. The authors acknowledge an inefficiency issue within the model.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of collaborative connections to cognitive diagnosis is well-motivated and logical.\n\n2. The paper is well-structured, and the methodology is explained in detail.\n\n3. The release of the code enhances the reproducibility of the research.", "weaknesses": "1. While the authors identify the challenge of obtaining explicit collaborative connections, they do not provide clear solutions to address this issue.\n\n2. The heavy use of theoretical notations makes the paper less readable.\n\n3. The detailed presentation of the methodology makes the experimental section appear crowded.\n\n4. The performance improvements demonstrated by Coral are moderate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful reading and detailed discussion of our paper. We address your concerns below and please let us know if there are remaining questions or unclear points.\n\nFor the weaknesses:\n---\n> **Weakness 1:** Explain the solution of collaborative relation construction in detail.\n\nThank you for your feedback and for giving us the opportunity to clarify our solution.\n\nConstructing collaborative connections aims to find $K$ collaborative neighbors for each learner $u \\in V$ and build the collaborative graph $G$ (see lines 11-31 in Algorithm 1). This process is formulated as $ \\max P(G|V,Z) $, where $V$ is the set of learners and $Z$ is the learner state vector. To achieve this, we iteratively find the $K$ optimal neighbors for each learner $u$ with the following initialization: $u$'s neighbor set initially includes only $u$, i.e., $ r_{u}^{0} = {u} $, and the non-neighbor set consists of all other learners, i.e., $ V_u = V \\setminus {u} $.\nThe iteration starts and repeats $K$ times until $u$ has $K$ neighbors. Each iteration includes the following steps:\n\n**Iteration (for $ k = 1, 2, \\ldots, K $):**\n   - Compute context vector of $u$ using $(k-1)$ neighbors: $ \\text{Context}(u) = \\sum_{v \\in r_{u}^{k-1}} Z_v $\n   - Calculate similarity scores between $u$'s context and each non-neighbor learner $v$: $ \\text{score}_{u,v} $ for each $ v \\in V_u $\n   - Select the most similar learner as the $k$-th neighbor: $b_{u}^k = \\arg\\max_{v \\in V_u} \\text{score}_{u,v} $\n   - Update neighbor set: $ r_{u}^{k} = r_{u}^{k-1} \\cup \\{b_{u}^k\\} $\n   - Update non-neighbor set: $ V_u = V_u \\setminus \\{b_{u}^k\\} $\n\nWe sincerely hope our explanation resolves your concerns. We will include more examples in the revised version for clarity.\n\n> **Weakness 2 and  Weakness 3:** Issue of balance between theoretical details and experiments.\n\nWe apologize for any confusion. To enhance readability, we included detailed proofs and algorithms in the appendix while summarizing key conclusions in the main text of the paper. For instance, the core construction method of the collaborative graph is summarized in lines 202-207 of the main text, with detailed derivations in Appendix B and algorithms in Appendix C. This structure ensures that our key conclusions are understandable without requiring a deep dive into the theoretical derivations, as acknowledged by reviewers vXdf, t9qg, bUz3, and UPTU.\n\nWe will simplify the theoretical symbols in the revised version and include more examples to improve clarity. Additionally, we will increase the size and spacing of Fig.3 and 4 in experimental section.\n\n> **Weakness 4:** Concerns on performance improvement.\n\nCoral's performance improvements are validated through experiments across three different scenarios (line 296), **not solely in Table 1**. We sincerely encourage you to consider the model's performance across all scenarios to appreciate Coral's comprehensive enhancement.\n\nSpecifically, Table 1 lists results for the normal scenario (20% test data), where Coral demonstrates significant improvements on Junyi and ASSIST. The improvement on the NIPS202EC dataset is relatively modest, primarily because the NIPS202EC dataset is very dense (as shown in Table 3, with 367.5 problem-solving records per learner), leading to smaller differences between models. Additionally, Figures 2(a) and (b) illustrate significant performance improvements in scenarios with data sparsity and cold start, which are commonly encountered in real platforms, showing that Coral significantly outperforms the baselines.\n\nFor the questions:\n---\n> **Question 1:** Concept disentanglement impacts and insufficient motivation for collaboration modeling.\n\n- **Concept disentanglement impacts:**\nTo answer your first question, we conduct experiments removing the disentanglement on concepts by representing each learner with a single $C$-dimensional vector instead of the original $C$ $d$-dimensional vectors, denoted as w/o VecDis. The following experimental results on NIPS2020EC prove the effectiveness of disentanglement.\n|Model|ACC|AUC|F1|RMSE|\n|-|-|-|-|-|\n|w/o VecDis|0.71523|0.78331|0.72667|0.43412|\n|**Coral**|**0.71622**|**0.79103**|**0.72890**|**0.43200**|\n- **Clarify the motivation:**\n\nThank you for your attention to our motivation and for giving us the opportunity to clarify. In fact, our core motivation is to integrate collaborative signals into CD. Based on this motivation, disentangled representations are introduced to manage the increased complexity of modeling both individual and collaborative representations. Detailed explanations are as follows:\n\n1. Motivation 1 (collaborative Modeling): Introducing collaborative signals between learners in CD models enhances both performance and interpretability. It is sufficiently supported by social learning theories [1], data analysis (see experiments 1 and 2 in the response to Weakness 2 of Reviewer Ty5J), and empirical effectiveness on learner sequences [2] and group modeling [3], demonstrating the benefits of incorporating collaborative signals.\n2. Motivation 2 (disentanglement): To model both individual and collaborative learner representations following motivation 1, complexity increases. Traditional single-vector representations may not adequately capture this complexity. Thus, disentanglement is naturally introduced to convert single vector into multiple vectors, reducing dimensional correlation and enhancing modeling capability, inspired by [4].\n\nThank you again, and we sincerely hope our explanation resolves your concerns. We will make this insight clearer in a revised paper.\n\n---\n[1] Albert Bandura and Richard H Walters. Social learning theory. 1977.\n\n[2] Improving knowledge tracing with collaborative information. WSDM'22.\n\n[3] Homogeneous Cohort-Aware Group Cognitive Diagnosis: A Multi-grained Modeling Perspective. CIKM'23.\n\n[4] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eobhzt1rMT", "reviewer": "Reviewer_Ty5J", "summary": "This paper introduces Coral, a model that enhances cognitive diagnosis by integrating collaborative signals among learners with disentangled representation learning. Coral addresses the importance of leveraging collaborative connections among learners to better understand human learning processes. Traditional cognitive diagnosis methods focus on individual attributions and explicit practice records, but they often overlook the potential of collaborative signals. The proposed Coral model aims to fill this gap by integrating collaborative and disentangled cognitive states. The disentangled state encoder separates the cognitive states of learners and reconstructs their practice performance from an inner-learner perspective. The collaborative representation learning dynamically constructs a collaborative graph by iteratively searching for optimal neighbors in a context-aware manner, effectively capturing collaborative signals. The model is evaluated on three real-world datasets: ASSIST, Junyi, and NIPS2020EC. The results show that Coral outperforms existing methods in terms of ACC, AUC, F1-score, and RMSE. Coral also performs well in sparse and cold-start scenarios, demonstrating its robustness and adaptability.", "questions": "Please see the weaknesses", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Coral introduces an integration of collaborative signals with disentangled representation learning, addressing a gap in existing cognitive diagnosis methods.\n2. Experimental results demonstrate performance improvements over state-of-the-art methods across various datasets and scenarios.\n3. The disentangled representation provides better explainability and controllability of cognitive states, which is crucial for personalized education.", "weaknesses": "1. The approach in this paper is not very innovative, and the motivation is not very convincing.\n2. Although visualizing the embeddings provides some level of interpretability, it still does not yield any explicit conclusions or observable phenomena. Rather than visualizing the embeddings (t-SNE has a certain degree of randomness), I would prefer to see some interesting conclusions based on the dataset.\n3. The paper mentions computational inefficiencies, which could limit the model's applicability in large-scale or real-time settings.\n4. The model requires careful tuning of hyperparameters, which may not be straightforward for all datasets and might require extensive experimentation.\n5. The paper could benefit from a more extensive comparison with a broader range of alternative approaches to strengthen its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful reading and detailed feedback on our paper. We address your concerns below and please let us know if there are remaining questions or unclear points.\n> **Weakness 1:** Unconvincing motivation and limited methodological novelty.\n\nThank you for highlighting these concerns and for giving us the opportunity to provide a clearer explanation.\n- **Our motivation is well-founded** and supported by data analysis and relevant literature, involving two aspects:\n1. Motivation 1 (collaborative modeling): Introducing collaborative signals between learners in CD can enhance performance and interpretability. This is supported by social learning theories[1], data analysis (see experiments 1 and 2 in the next response to Weakness 2), and empirical effectiveness on learner sequences[2] and group modeling[3], which demonstrate the benefits of incorporating collaborative signals. However, most CD models focus on individual modeling and often neglect collaborative aspects. Thus, we incorporate collaborative signals in to CD is both convincing and meaningful.\n2. Motivation 2 (disentanglement): Modeling both individual and collaborative learner representations following Motivation 1 increases complexity, which traditional single-vector representations may not adequately handle. Thus, disentanglement is naturally introduced to convert single-vector representations into multiple vectors, reducing dimensional correlation and enhancing modeling ability, inspired by [4].\n\n- **Mthodological novelty** on key modules:\n1. Disentangled Encoder (inner-learner view): Inspired by DCD[3], our encoder considers both vector-level and dimensional-level disentanglement, unlike DCD, which only considers the latter. This approach's effectiveness is empirically demonstrated through ablation studies.\n2. Collaborative Modeling (inter-learner view): This design is unique as relevant work in CD is almost blank. We provide theoretical derivations to support the model's rationale.\n3. Co-Disentanglement Decoder:  Inspired by residual connections, our decoder effectively and simply aligns inner- and inter-learner perspectives. This alignment represents a novel exploration in this direction.\n\nOur model's innovation is supported by both empirical and theoretical evidence.\n\n> **Weakness 2:** Concerns on interpretability.\n\nThank you for your attention to interpretability.\n\nWe would like to clarify that our interpretability analysis is detailed in a case study in Appendix E, rather than through visualizations. This example demonstrates how Coral infers each learner's future performance by referencing the cognitive states of similar learners, thereby providing educators with a basis for their inferences.\n\nTo provide more data-based conclusions, we conduct additional experiments on ASSIST:\n1. We represent $M$ learners $U$’ performance on $N$ questions with a matrix $X = \\\\{x\\_{u,i}\\\\}\\_{M \\times N}$, where $x\\_{u,i}$ indicates whether a learner $u$ answered problem $i$ correctly (1), incorrectly (0), or did not attempt it (-1). We split this matrix into $X\\_{\\text{train}} = \\\\{x\\_{u,i} \\mid u\\in U, i \\in [1, N/2]\\\\}$ and $X\\_{\\text{test}} = \\\\{x\\_{u,i} \\mid u\\in U,i \\in [N/2+1, N]\\\\}$. For each learner $u$ in $X\\_{\\text{train}}$, we find the most similar learner $v$ by maximizing $\\text{cosine}(X\\_{u}, X\\_{v})$, and then compute the absolute difference $|\\Delta(u,v)|$ in their passing rates on $X\\_{\\text{test}}$.\n2. Based on a trained Coral, we find the most similar learner $v$ for each learner $u$ via learned embedding cosine similarity. We then calculate the absolute difference $|\\Delta(u,v)|$ in the accuracy of problem-solving between $u$ and $v$ on the test data.\n\nMean and variance of all $|\\Delta(u,v)|$ are listed as:\n| Experiment | Mean| Variance|\n|-|-|-|\n|1| 0.19250| 0.00185|\n|2| 0.23114| 0.00497|\n\nExperiment 1 shows that learners with similar training data experiences tend to have similar performance on future practice, supporting our collaborative modeling motivation from the perspective of dataset statistic. Experiment 2 proves the effectiveness and interpretability of learner embeddings learned by Coral. Both experiments reinforce that similarity of learner embeddings by Coral can align with statistical data. We will make this insight clearer in the revised paper.\n> **Weakness 3:** Computational inefficiency.\n\nOur paper provides 3 alternative efficient solutions in Appendix F.2 to improve Coral's computational efficiency. Although these solutions may not guarantee theoretical optimality, the experimental results demonstrate their effectiveness, which prove Coral can be used to some large-scale scenarios. For real-time response, co-disentangled cognitive state (line 249) can be precomputed and stored, with only retrieval needed during online computation. We will continue to optimize the model efficiency in future studies.\n> **Weakness 4:** Concerns on hyperparameter tuning complexity.\n\nCoral requires minimal hyperparameter tuning. We adjust only two parameters, $\\beta$ and $K$, with other parameters fixed based on prior experience. Experimental results in Fig3 (c) show $K$ stabilizes after reaching a threshold due to attention control. Thus, when tuning, we can start with a larger $K$ and gradually decrease it. For $\\beta$, we recommend values of 0.25, 0.5, and 0.75. Typically, 3 to 4 iterations are sufficient to find satisfactory parameters.\n> **Weakness 5:** Add more baselines.\n\nWe conduct additional experiments with four new baselines. **The experimental setups, results and analysis are given in Public Response** (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66).\n\n---\n[1] Albert Bandura and Richard H Walters. Social learning theory. 1977.\n\n[2] Improving knowledge tracing with collaborative information. WSDM'22.\n\n[3] Homogeneous Cohort-Aware Group Cognitive Diagnosis: A Multi-grained Modeling Perspective. CIKM'23.\n\n[4] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Public Response to All Reviewers**
---

---

We would like to express our thanks to the reviewers for their thorough reading of the paper and insightful comments. We first add some common experiments as suggested by the reviewers.

> **1. Additional baselines**

We conduct experiments on the four suggested baselines: VIBO[1], AGCDM[2], VarFA[3], and SCD[4], using three datasets. Each model is implemented based on their open-source code with minor modifications. For VIBO[1] and VarFA[3], we use 2PL (multi-dimensional) IRT as the backbone. AGCDM[2] requires modifications to handle our datasets, as its original code cannot distinguish between incorrect answers (negative samples) and unanswered questions. We mark unattempted records as -1, correct answers as 1, and incorrect answers as 0, and filter out unanswered data by adding the line of code “*data = np.array([entry for entry in data if entry['score'] != -1])*” in line 63 of AGCDM code's run.py file. For SCD[4], we use an open-source tool to construct the concept map and set feature vector dimensions for students and questions to match Coral. The results are as follows:

|ASSIST|ACC|AUC|RMSE|junyi|ACC|AUC|RMSE|NIPS2020EC|ACC|AUC|RMSE|
|-|-|-|-|-|-|-|-|-|-|-|-|
|VIBO|0.70265|0.70229|0.45342|VIBO|0.81265|0.79592|0.36928|VIBO|0.69667|0.73669|0.45723|
|AGCDM|0.70935|0.67705|0.45120|AGCDM|0.79310|0.66405|0.40817|AGCDM|0.70266|0.70318|0.45332|
|VarFA|0.71034|0.74630|0.44312|VarFA|0.81708|0.80347|0.37102|VarFA|0.71221|0.75224|0.44774|
|SCD|0.72003|0.75630|0.43102|SCD|0.81172|0.82037|0.37015|SCD|0.71013|0.78556|0.43617|
|**Coral**|**0.72533**|**0.77312**|**0.42034**|**Coral**|**0.82534**|**0.83503**|**0.36403**|**Coral**|**0.71622**|**0.79103**|**0.43200**|

The results demonstrate that Coral consistently outperforms the baselines across all datasets. Among the baselines, AGCDM's performance is impacted by the imbalanced distribution of positive and negative samples in the ASSIST and Junyi datasets (Please refer to Table 3 of our paper for specific statistics), resulting in high ACC but low AUC. In contrast, VIBO, VarFA, and SCD mitigate the impacts of imbalanced data through their use of variational inference and graph modeling techniques.

> **2. Hyper-parameter sensitivity experiments on embedding  dimensions**

We conduct additional hyper-parameter sensitivity experiments on different embedding dimensions $d$, with values of 124, 200, 256, 521, and 1024. Coral's performance on ACC is shown below:

|Value of $d$|124|200|256|512|1024|
|-|-|-|-|-|-|
|ASSIST|0.72667|0.72533|0.72411|0.71882|0.70611|
|junyi|0.82344|0.82134|0.81920|0.81474|0.82256|
|NIPS2020EC|0.72083|0.71622|0.72072|0.71334|0.71227|

We find that within an appropriate range, the dimension $d$ does not significantly affect the Coral's performance.

> **3. Add a new dataset (Optional suggestion)**

Before introducing a new dataset, we would like to first emphasize that our existing datasets are diverse, covering both K-12 scenarios (ASSIST) and online learning scenarios (Junyi and NIPS2020EC). These datasets are also representative, as they are benchmark datasets in cognitive diagnosis (CD) research. Nearly all top-tier work (e.g., [4][6][7]) uses these datasets for experiments.

To further demonstrate Coral's performance in different scenarios, we supplement our experiments with the PISA 2015[5] dataset. This dataset includes students' practice data from various countries and regions. We extract records of 1,000 learners from the Asian region, focusing on 200 questions. We filter out students with fewer than 20 response records, resulting in 21,814 practice records. The performance of Coral and representative baselines on this new dataset is shown below:

|Model|ACC|AUC|RMSE|
|-|-|-|-|
|IRT|0.67527|0.73416|0.46031|
|MIRT|0.62210|0.66973|0.48431|
|NCDM|0.65822|0.71403|0.49013|
|KANCD|0.67373|0.74271|0.45364|
|RCD|0.67234|0.72673|0.46253|
|**Our Coral**|**0.68312**|**0.74807**|**0.45217**|

The experimental results demonstrate that Coral maintains its performance advantage.

---

[1] https://arxiv.org/pdf/2002.00276

[2] https://ieeexplore.ieee.org/document/10027634

[3] https://arxiv.org/pdf/2005.13107

[4] Self-supervised graph learning for long-tailed cognitive diagnosis. AAAI'23.

[5] https://github.com/bigdata-ustc/EduData

[6] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.

[7] RCD: Relation map driven cognitive diagnosis for intelligent education systems. SIGIR'21.

Meta-review: This paper introduces Coral, a Collaborative Cognitive Diagnosis model utilizing Disentangled Representation Learning. It enhances cognitive diagnosis by dynamically capturing collaborative signals among learners, leading to improved performance over state-of-the-art methods in several educational datasets. The proposed methodology shows promise in advancing personalized education through more nuanced insights into learners’ cognitive states.

**Strengths**
- Novelty: The paper introduces a unique approach by integrating collaborative signals among learners using disentangled representation learning. This allows for a better understanding of learners’ cognitive states and could enhance personalized educational interventions (Reviewers vXdf, bUz3).
- Strong Empirical Results: The model demonstrates superior performance across various real-world datasets, outperforming state-of-the-art methods in several metrics, including ACC, AUC, F1-score, and RMSE, highlighting its practical utility in both cold-start and sparse data scenarios (Reviewers Ty5J, t9qg).
- Explainability: By disentangling cognitive states, the Coral model provides greater explainability and controllability in diagnosing students’ cognitive states, which can be valuable for educators in understanding student learning patterns (Reviewers Ty5J, vXdf, UPTU).
- Collaborative Graph Learning: The model’s iterative graph construction process allows it to dynamically search for optimal neighbors, enhancing the model’s capability to leverage collaborative signals (Reviewers Ty5J, vXdf).

**Areas for Improvement (Weaknesses)**
- Computational Inefficiency: The model suffers from high computational costs, which may hinder its scalability and practical application in large-scale educational settings (Reviewers Ty5J, UPTU, bUz3).
- Limited Generalizability: The choice of datasets may limit the model’s generalizability to other educational contexts, as the current experiments focus on specific scenarios (Reviewers bUz3, Ty5J, t9qg).
- Weak Interpretability Evaluation: Although the model claims improved explainability, the visualizations (such as t-SNE) may not be sufficient to provide meaningful interpretability. More rigorous qualitative analysis and ablation studies are needed to evaluate the disentanglement quality (Reviewers Ty5J, bUz3).
- Sensitivity to Hyperparameters: The model’s performance is highly sensitive to hyperparameters like the number of neighbors (K) and beta value. This reliance could make it difficult for practitioners to fine-tune the model for different datasets (Reviewer Ty5J).

The Authors' actively and extensively participated in the rebuttal and Author-Reviewer discussion phase. They acknolwedged and addressed most Reviewer and AC concerns and adhered to most of the reasonable and valuable suggestions provided by the Reviewers and AC. This strengthens their work significantly from its initial state.

An overview of key concerns categorized by the degree to which they were addressed is listed below:

.1. Addressed Concerns:
- Data diversity and additional baselines: The Authors added a new dataset and ran additional experiments on suggested baselines like VIBO, AGCDM, VarFA, and SCD, with Coral outperforming them consistently. This strengthened the original claims about Coral’s superior performance.
- Hyper-parameter sensitivity: They conducted additional sensitivity experiments for different embedding dimensions, which showed Coral’s performance was stable within a reasonable range.
- Interpretability: The Authors clarified that disentanglement improved interpretability by aligning cognitive states and collaborative states and offered clearer explanations of how Coral’s predictions could be interpreted by educators.
2. Partially Addressed Concerns:
- Overfitting: While the Authors explained that they used early stopping and conducted experiments on data sparsity and cold-start scenarios, concerns about overfitting were acknowledged but not fully explored with extensive validation metrics.
- Explainability: The Authors provided examples of explainability in Appendix E, and additional conclusions were offered to support their claim. However, Reviewer concerns about limited exploration of how individual cognitive states relate to knowledge concepts were only partially addressed.
3. Negatively Addressed or Unaddressed Concerns:
- Computational efficiency: The Authors admitted the model’s inefficiency but offered practical solutions without significant exploration of how these might affect large-scale applications. This limitation may affect the practical deployment of Coral in real-world, large-scale settings.
- Ethical considerations: The Author response included a discussion of potential privacy and fairness concerns but lacked a detailed ethical framework or proposed safeguards for applying the model in real-world education systems.

Overall, the Author rebuttals generally improved the submission by addressing key concerns raised in the reviews, although some areas, particularly around computational efficiency and ethical considerations, remained only partially addressed.

Key considerations for my recommendation include (1-3 are positive, 4 is a remaining weakness, but has been acknowledged by the Authors):

1. Author Rebuttal Quality: The Authors have thoroughly addressed most of the Reviewers’ concerns. For instance, they provided new baseline comparisons, further hyper-parameter sensitivity analyses, and new datasets, including the PISA 2015 dataset. These additions strengthened the evidence for the model’s robustness, handling of cold-start scenarios, and disentanglement quality.
2. Reviewer Feedback: The Reviewers generally upgraded their ratings after the Author rebuttal phase, reflecting a consensus toward acceptance. Reviewer Ty5J increased their score after the rebuttal, acknowledging improvements in the paper.
3. Strengths Reinforced: The novelty and robustness of the Collaborative Cognitive Diagnosis model with Disentangled Representation Learning (Coral) was strongly supported by the empirical evidence provided by the Authors. The Authors successfully demonstrated the model’s generalizability, performance on diverse datasets, and its advantage over existing state-of-the-art methods.
4. Remaining Weaknesses: While computational inefficiencies and concerns about scalability remain, the Authors have proactively addressed these limitations with proposed solutions, and these do not appear to detract significantly from the overall contribution of the work.

The Author rebuttal phase was constructive, with several key concerns being effectively addressed. However, to further strengthen the submission, the Authors should address the identified weaknesses, particularly in terms of computational efficiency, ethical considerations, and further hyperparameter sensitivity analyses. 

The paper makes positive contributions to collaborative* cognitive diagnosis with its novel approach, technical rigor, and strong empirical results. Overall, Coral’s impact on the field of educational data mining and cognitive diagnosis is promising. Regardless of whether the paper is accepted, it is strongly suggested that the Authors implement the changes that they agreed to for their paper. If the paper is accepted, then the Authors are expected to do so.
Decision: Accept (poster)