Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling

Authors: weibo  gao
qi  liu
linan  yue
fangzhou  yao
hao  wang
yin  gu
zheng  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:40:15

Abstract: Learners sharing similar implicit cognitive states often display comparable observable problem-solving performances. Leveraging collaborative connections among such similar learners proves valuable in comprehending human learning. Motivated by the success of collaborative modeling in various domains, such as recommender systems, we aim to investigate how collaborative signals among learners contribute to the diagnosis of human cognitive states (i.e., knowledge proficiency) in the context of intelligent education.
The primary challenges lie in identifying implicit collaborative connections and disentangling the entangled cognitive factors of learners for improved explainability and controllability in learner Cognitive Diagnosis (CD). However, there has been no work on CD capable of simultaneously modeling collaborative and disentangled cognitive states. To address this gap, we present Coral, a $\underline{Co}$llabo$\underline{ra}$tive cognitive diagnosis model with disentang$\underline{l}$ed representation learning. Specifically, Coral first introduces a disentangled state encoder to achieve the initial disentanglement of learners' states.
Subsequently, a meticulously designed collaborative representation learning procedure captures collaborative signals. It dynamically constructs a collaborative graph of learners by iteratively searching for optimal neighbors in a context-aware manner. Using the constructed graph, collaborative information is extracted through node representation learning. Finally, a decoding process aligns the initial cognitive states and collaborative states, achieving co-disentanglement with practice performance reconstructions.
Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over state-of-the-art methods across several real-world datasets.
Our code is available at https://github.com/bigdata-ustc/Coral.

Main paper: 
 introduction: 
 It is a common notion that individuals with similar implicit states frequently exhibit similar explicit behaviors. Therefore, establishing interconnections among similar users is crucial for understanding human behaviors. For instance, social connections play a pivotal role in understanding current consumer preferences and predicting future behaviors [42] . Similarly, in the context of intelligent education, a better modeling of like-minded learners with similar learning experiences, is essential for understanding the human learning process [31] , analyzing their knowledge proficiency and facilitating personalized tutoring tailored to individual needs [55] .As illustrated in Figure 1 , we can infer Nancy is likely to answer the Cone-related question q 5 correctly according to the correct practice responses of Bob and Alice, who share similar learning behaviors with Nancy. The underlying psychological assumption is that learners with similar experiences generally possess similar cognitive states -how well the learner masters each knowledge concept, influencing their subsequent responses. To gain a deeper understanding of the human learning process, it is crucial to explicitly diagnose unobservable cognitive states. Existing Cognitive Diagnosis (CD) methods seek to enhance diagnostic accuracy by fully utilizing the inner-learner information (i.e., individual attributions [41] and explicit practice records) and question-side features (e.g., difficulty [16] , textual content [26] , and educational relations [12, 14] ). However, the issue of how similar (a.k.a. collaborative) connections among inter-learners with similar states facilitate understanding of learners' knowledge proficiency remains largely unexplored.In this study, to efficiently harness collaborative information among similar learners and thereby more accurately diagnose the cognitive states of each individual, we advocate for the incorporation of inter-learner connections into the CD process. However, designing a collaborative CD model in educational scenarios presents two distinct challenges due to the complexity of human learning.• First, acquiring explicit collaborative connections among learners proves to be a formidable challenge. On the one hand, unlike many well-defined social scenarios (e.g., Twitter.com), where user preference similarities are manifested through explicit social actions such as following and liking, the directly available social behaviors among learners in learning environments (e.g., LeetCode.com) cannot be used for diagnosis modeling since these social attributes cannot reflect true cognitive-oriented connections. On the other hand, some related studies [28, 13] attempt to design different similarity functions based on practice data to compute cognitive similarities among learners. However, these approaches pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases. Although various methods for constructing user relationships have been proposed in other domains [22, 9] , these approaches do not consider the domain-specific attributes of students in learning scenarios and cannot be directly applied in educational contexts. • Second, an ideal collaborative diagnosis procedure requires disentangling and uncovering the mixed explanatory latent factors hidden in the observed learning behaviors. The basic motivation is that learners demonstrate complex and diverse patterns driven by entangled states across both innerand inter-learner perspectives. For instance, from an inner perspective, Nancy may not master Cone since she does not practice Cone-related questions. However, based on inter-learner data, one can infer a high probability that she has mastered Cone. Most prior attempts can not fulfill this requirement since they learn representations in an entangled way. Although recent models [8, 50] achieve a dimension-level disentanglement of cognitive states, they lack consideration of modeling the influence of collaborative connections, ignoring the complex relations between inner-and inter-learner connections of different individuals. Thereby, it needs to find a suitable way to achieve the co-disentanglement from both the inner-and inter-learner views for cognitive representations with higher interpretability and controllability.To tackle the above challenges, we propose Coral, a Collaborative cognitive diagnosis model with disentangled representation learning, to reveal learner cognitive states while simultaneously modeling both inner-and inter-learner learning information. Specifically, our approach begins with the disentangled cognitive representation encoding to establish initial disentangled learner states through reconstructing their practice performance from the inner-learner perspective. Next, our focus shifts to effectively learning collaborative cognitive representations from the inter-learner perspective. The most significant point is to find the implicit collaborative relations between learners. To address this challenge, we present a context-aware collaborative graph learning mechanism that automatically explores all K-optimal neighbors for each learner given their basic cognitive states to facilitate the explicit modeling of collaborative connections among learners. Based on the constructed graph, collaborative information can be effectively fused into disentangled learner cognitive states through learning collaborative node representations. Finally, a decoding and reconstruction process is conducted to merge initial states and collaborative states so as to achieve co-disentanglement from both the inner-and inter-learner perspectives. Extensive experiments demonstrate the superior performance of Coral, showing significant improvements over SOTA methods across several datasets. 
related work: 
 Cognitive Diagnosis As a fundamental task, cognitive diagnosis (CD) has been well-researched for decades in educational psychology [25, 4, 53] . It aims to profile the implicit cognitive states (i.e., the proficiency of specific knowledge concepts) of learners by exploiting observed practice records (e.g., correct or wrong). Existing research on CD assumes that learners' knowledge proficiency is proportional to their practice performance and thus can be diagnosed through predicting their practice performance [12] . Since the diagnostic results can be applied to many intelligent applications, such as exercise recommendation [18] and learning path suggestions [55] , many CD models have been proposed in recent years. The early works from psychology like IRT [16] and MIRT [1] focus on modeling learners' answering process by predicting the probability of a learner answering a question correctly, which utilizes latent factors as the learner's ability. These methods lack interpretability, i.e., they are inability to output explicit multidimensional diagnostic results on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of questions to diagnose learners' proficiency on all knowledge concepts [38, 45, 46, 32] .Representative NCDM [40] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works (e.g., IRT, and MIRT). In summary, existing CD studies enhance diagnostic accuracy by fully utilizing the inner-learner information (i.e., individual attributions and explicit practice records) [41, 50] and question-side features (e.g., difficulty [16, 38] , textual content [26] , and educational relations [12, 14, 8] ). However, to the best of our knowledge, the problem of collaborative diagnostic modeling remains largely unexplored.Collaborative connections among learners in the education context commonly refer to learners with similar explicit practice behaviors, testing scores and implicit knowledge proficiency [28, 54, 44] . However, due to the complexity and implicitness of the human learning process, these relations are commonly not explicitly and directly available.Existing studies [28, 13] in AI Education have attempted to design different similarity functions based on practice data to compute cognitive similarities among learners. However, these methods pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases.Disentangled Representation Learning Disentangled Representation Learning (DRL) [3] , which aims to produce robust, controllable, and explainable representations, has become one of the core problems in machine learning. Typical methods include variational method [20] , weakly supervised models [21] , as well as the recent combination with the diffusion model [6] . DRL has a wide range of applications in user modeling to disentangle attributes. For example, recommendation with several aspects of users' interests [24, 33] , fair user representation to disentangle sensitive attributes [10] .In education, DCD [8] attempts to disentangle learners' cognitive representations via variational framework, which motivates us to conduct a further study on collaborative CD setups. 
coral: 
 We first introduce the problem setup, followed by details on three core components of Coral: i) Disentangled Cognitive Representation Encoding, ii) Collaborative Representation Learning and iii) Decoding and Reconstruction. Figure 2 shows the framework. The algorithm is listed in Algorithm 1.Our setup considers the human learning dataset D including the practice records between M learners and N questions. The practice records of each learner u is denoted by x u = {x u,i }, where x u,i equals 1 or 0, representing that learner u answered question i correctly or not, respectively. Each question is related to at least one knowledge concept. The association relations between N questions and C knowledge concepts is represented byC = {c i } N i=1, where c i ∈ R C and c i,c equals 1 or 0 denoting that question i is related to concept c or not. The practice records are regarded as the explicit inner-learning information in our context.Besides, we consider the collaborative connections among learners with similar cognitive states, which provide the inter-learner information. We define collaborative connections as a graph structure G = (V, E) which contains a set of nodes (i.e., learners) V and a set of edges E where (u, v) ∈ E or (u, v) ∈ G indicates that the existence of a collaborative connection between learner u and v (i.e., u and v have similar latent cognitive states). Notably, the collaborative connections in educational 𝒩(𝝁 1 , 𝝈 (1) ) 𝒩(𝟎, 𝜎 0 2 𝐈) 𝒩(𝝁 1 , 𝝈 (1) ) 𝒩(𝟎, 𝜎 0 2 𝐈) 𝒩(𝝁 1 , 𝝈 (1) )𝒩(𝟎, 𝜎 0 2 𝐈) 𝑢 𝑢 𝑢 x 𝑢,1x 𝑢,2x 𝑢,3x 𝑢,5x 𝑢,4x 𝑢,9x 𝑢,8x x 𝑢,1x 𝑢,2x 𝑢,3x 𝑢,5x 𝑢,4x 𝑢,9x 𝑢,8x scenarios are generally not explicitly or directly available, and it needs to design an adaptive strategy to automatically infer similar learners from observed learning data during the training process.To achieve cognitive state disentanglement, we initially assign C factorized representations to each learner, i.e., z u = [zu ; zu ; . . . ; z(C)u ] ∈ R d×C with Gaussian Mixture initialization since the Gaussian distribution has long been recognized as a proper statistic model for the cognitive states of learners in educational psychology [4] . The component z (c) u is expected to capture the learner u's cognitive state over knowledge concept c. We denote Θ as the set of trainable parameters for the proposed model. Based on the above setups, the goal of Coral is to learn co-disentangled representations Z = {z u } M u=1 for the M learners from both the inner-learner practice perspective and inter-learner collaborative perspective.The practice response x u of each learner u provides valuable inner-learner insights regarding his/her proficiency since learners' performance on each question is assumed to be proportional to their cognitive proficiency on question-related knowledge concepts [8] . Therefore, we implement an encoder for encoding the disentangled cognitive state z u of each learner u by reconstructing their practice responses. For a learner u, we assume that his/her practice performance on candidate questions can be generated from the following distribution:EQUATIONwhere p(C) = p D (C) and p Θ (x u | z u , C) is naturally a cognitive diagnosis procedure to predict practice performance. The key point of this task is to learn an optimal encoder p Θ (z u ) via practice records x u to encode the cognitive state z u of each learner u. To optimize Θ, we introduce a variational distribution q Θ (z u | x u ) to approximate p Θ (z u ), following the VAE literature [3] , through maximizing a lower bound of log p Θ (x u ) based on the following property.Property 1. max log p Θ (x u ) is bounded as follows:EQUATIONSee the Appendix A for the proof.In Property 1, the first term reconstructs the true practice performance x u of learner u and the variational encoder q Θ (z u | x u ) in the second term approximates the true encoder p Θ (z u ) by minimizing the KL divergence D KL . The variational distribution q Θ (z u | x u ) and the expectation E qΘ(zu|Xu) are intractable, thus we employ the re-parameterization trick [20] for the model optimization.Furthermore, the diagnosis procedure p Θ (x u | z u , C) is achieved by estimating how well a learner u answers question i from both the perspectives of cognitive states and comprehensive abilities. From the perspective of cognitive states, solving question i requires learner u to master all knowledge concepts related to this question. Regarding comprehensive abilities, each learner possesses a latent state reflecting their overall learning ability, which is shared when addressing different questions.Formally, this process can be described as:EQUATIONwhere ϕ Θ (•) and ψ Θ (•) : R d → R + are two shallow neural networks. ψ Θ (•) estimates the comprehensive ability of the learner and ϕ Θ (•) predicts the performance of a learner with a given cognitive state z(c)u and a comprehensive ability θ u over question i in terms of concept c. h i is a learnable latent representation for question i. Besides, to ensure psychometric interpretability of prediction, we set the weights of ϕ Θ (•) are positive values, i.e., ∂ϕ Θ (•)/∂z u > 0, assuming that the probability of correctly answering the question monotonically increases with learners' cognitive state. Please note that we found that the mean operation here can also be replaced with a neural network (i.e., ϕ ′ Θ : R C → R + ) with positive weights, formulated asϕ ′ Θ c i • θ u • z (c)u -h i , as in [40] , without affecting prediction performance. Particularly, in contrast to most methods that consider entangled cognitive factors as input, our diagnosis model can better capture learners' proficiency on each knowledge concept by disentangling cognitive states under each concept. Furthermore, inspired by the outstanding performance of β-TCVAE [7] in disentanglement, we prompt statistical independence among its dimensions to obtain a better trade-off between the reconstruction accuracy and the quality of disentangled representation through q(z(c) u ) = d j=1 q Θ z (c) u,jwhere q Θ (z(c) u ) is the aggregated posterior of z u , i.e., q Θ (z (c) u ) = q Θ (z (c) u | x u )p (x u ) dx u where p (x u ) = p data (x u ).This setup is encouraged by the term D KL (•) in Eq. ( 2) based on Property 2.Property 2. The D KL (•) in Eq. (2) can be rewritten as:EQUATIONSee Appendix A for the proof. On one hand, I (z u , x u ) maximizes the mutual information (MI) between z u and x u which obtains the useful information for the diagnosis task as much as possible according to the information bottleneck theory [2] . On the other hand, given a Gaussian distribution p Θ (z(c) u ) = d j=1 p Θ z (c)u,j , the KL divergence term encourages independence among the dimensions of z (c) u by preventing each latent variable from deviating too far from specified priors. Compared to prior VAE-based CD models [50, 8] , Coral additionally considers ability parameters from psychology [16] to enhance the expressive power of disentangled cognitive states.Overall, we penalize Eq. ( 2) by a Lagrange multiplier β resulting in the following objective:log p Θ (x u ) ≥ E p(C)qΘ(zu|Xu) [log p Θ (x u | z u )] -β • E p(C) [D KL (q Θ (z u | x u ) ∥ p Θ (z u ))] .(5)Collaborative information among similar learners provides an auxiliary inter-learner insight for cognitive representation learning. However, collaborative connections among learners with similar states are typically not readily accessible. To address this challenge, we design a context-aware graph construction strategy that searches similar neighbors automatically via the initial disentangled cognitive states. Based on the constructed collaborative graph, we can learn collaborative node representations by aggregating collaborative signals to generate collaborative cognitive states.The core goal of constructing the collaborative graph is to find K optimal neighbors for each learner node in V via their initial disentangled cognitive state {z (c) } C c=1 . For different knowledge concepts, the cognitive connections between the same learner pair are typically different. Thereby, it needs to search C groups of similar neighbors for each learner via each disentangled component z (c) . This means that we would generate C collaborative graphs, i.e., G = {G (c) } C c=1 . Each collaborative graph G (c) under concept c is expected to characterize the cognitive similarities of learners regarding concept c. Formally, this task is defined as computing p Θ (G | V, Z) by identifying all the K similar neighbors for each learner covering each concept c. Let N (c) u denote the set of K similar neighbors for the learner u, the task can be described as:EQUATIONwhere N (c) andZ (c) = {z (c)u } M u=1 are the neighbor set and feature set of all the learners regarding knowledge concept c, respectively. The number of N (c) u equals the combination of arbitrary K neighbors from all M learners for each learner node u under each concept c, i.e., N 6) is computationally expensive especially for larger M and K. To facilitate computation, we transform the Eq. ( 6) that requires global MI maximization to the task of maximizing MI locally via locally available context information inspired by [22] and derive a lower bound of it as the following Property 3.(c) u = M ! K!(M -K)! , thus Eq. (EQUATIONSee the Appendix A for the proof. The Eq. ( 7) iteratively searches K neighbors for the learner u under each knowledge concept c from step k = 1 to K. L (c),k u is the well-known InfoNCE loss function [36] . Let r (c),k-1 u denote the current context at step (k -1) (i.e., the set of (k -1) neighbors selected from step 1 to (k -1)). b , where the higher the scalar score means the higher likelihood of b(c),k u is a new neighbor. Furthermore, we have L (c),k u ∝ f (c) b (c),k u ; r (c),k-1 u. Thus, given the context of (k -1) neighboring learners (i.e., we have found (k -1) neighbors for the learner u) and matching function f (c) (•), our goal following the Property 3 is to find a learner b(c),k u from nonneighbor set V (c)u that can maximize the matching score f (c) (•) as the k-th neighbor of u. In other words, p(G | V, Z) can be optimized through maximizing the matching score f (c) (•) from k = 1 to K iteratively. Thereby, at each step k, we sort the scores of the nonneighbor learners and select the learner with the highest score to label as k-th neighbor b(c),k u , i.e., b (c),k u ← arg max v f (c) (v; r (c),k-1 u ), v ∈ V (c) u . After obtaining the k-th neighbor b (c),k u , the context r (c),k-1 u is updated to r (c),k u by absorbing b (c),k u .The calculation of matching score f (c) (•) usually relies on the node representations (i.e., learner cognitive states). However, the sub-optimal cognitive state learning during the initial training epochs probably results in the matching function exhibiting biases. To enhance the stability of model training, instead of directly aggregating node representations as the context r (c),k-1 u as many graph learning works, we denote it using relative representations w.r.t. the learner u [22] . Without loss of generality, we first establish relative collaborative coordinate systems with learner node u as the origin, and process relationship measurements between node u and each of its neighbors v as z(c) u,v = ∥z (c) u -z (c) v ∥ 2 .Then the context-aware features can be generated by aggregating each node in the context r(c),k-1 u , i.e., rc (c),k-1 u = v∈r (c),k-1 u z (c) u,v . Thereby, let v k denote b (c),k u with feature z (c) v k , we have f (c),v k = f (c) b (c),k u ; r (c),k-1 u = z (c) v k T • rc (c),k-1 u .After iteratively searching K neighbors under each concept, we can obtain C collaborative graphs regarding each learner, i.e., {G (c) } C c=1 . Then, we consider collaborative modeling as a node representation learning task within each collaborative graph G (c) . It relies on a nonlinear kernel function φ Θ (•) to aggregate neighboring information and update each disentangled cognitive state, i.e., r(c) u = φ Θ (z (c) u , {z (c) v : (u, v) ∈ G (c) }).Given the disentangled learner cognitive states generated by the variational posterior distribution q Θ (z u |x u ) from Property 1, φ Θ (•) is naturally expected to contain C channels to extract different concept features from similar learners, though projecting the representation z u into different subspaces, i.e., ẑ(c)u = σ(W T (c) z (c) u + b (c) )/∥σ(W T (c) z (c) u + b (c) )∥ 2 , where W (c) ∈ R d and b (c) ∈ R d arelearnable parameters of channel c and σ(•) is a nonlinear activation function (e.g., Sigmoid), and ∥ • ∥ 2 is L 2 normalization ensuring numerical stability. Then the collaborative learner representation modeling in terms of concept c can be described as:EQUATIONwhere s(c)u,v is the attention weight between u and v, considering both the collaborative aggregation (the first term) commonly used in graph modeling works and the corresponding context-aware attention (the second term) calculated in the iterative graph construction process in Eq. ( 7). When K is set large in Eq. ( 7), there is a possibility of introducing non-collaborative noise. In such cases, s (c) u,v can assign lower values to non-collaborative neighbors to mitigate the negative impact of noise, allowing for the adaptive tuning of attention in graph modeling. During training, the channels will remain changing because different subsets of the neighborhood will be searched for dynamically aggregating neighbor information in different iterations.With Gaussian Mixture initialization from the Disentangled Cognitive Representation Encoding (section 3.2), we derive the theorem on convergence as: Theorem 1. The Collaborative Representation Learning (section 3.3) procedure is equivalent to an expectation-maximization (EM) algorithm [35] for the mixture model. In particular, it converges to a point estimate of {r u,v equals 1 or 0 denoting whether learner v is a collaborative neighbor of learner u regarding concept c or not. See the Appendix A for the proof.Given the initial disentangled encoding via inner-learner information (section 3.2) and the collaborative representation learning via inter-learner information (section 3.3), this part encourages an alignment between the initial encode z u and collaborative state r u , formulating a co-disentangled representation as zu = z u + r u . This operation is inspired by the residual block [17] to address the second challenge, where r u can be treated as a disentangled auxiliary information of z u from collaborative graphs.The decoding process predicts the practice performance of each learner u on candidate questions, given her co-disentangled representation zu = z(1)u , z(2) u , . . . , z(C) u , i.e., p Θ (x u ) = E pΘ(C) [p Θ (x u | zu , C)],similar to the reconstruction procedure in Eq. ( 1). Thus, putting Eq. ( 5) and Eq. ( 7) together, we have the overall training objective: where BCE(•, •) is the binary cross entropy loss function between ground-truth practice behaviors x u and the reconstructed ones. α and β are hyper-parameters.EQUATIONBy optimizing with minimizing the above loss function Eq. ( 9), the cognitive state zu of each learner u can be jointly refined serving as the diagnostic results. During the testing phase, we evaluate the model performance by matching the difference between the predicted score p Θ (x u,i ) and the true score x u . Specifically, when a proficiency value is required instead of the vector z(c) u , we can obtain it by averaging each dimension of z(c) u .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We provide these details in the implementation details (Section 4.1).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 We empirically evaluate the performances of the proposed Coral model over three real-world datasets and conduct several experiments to prove its effectiveness.We conduct experiments on three real-world datasets: ASSIST [11] , Junyi [5] and NeurIPS2020EC [43] . The statistics of datasets are listed in Table 1 . The details about datasets and preprocessing are depicted in the Appendix C.Baselines The baselines include the matrix factorization-based model, i.e., PMF [34] , the typical latent factor models derived from educational psychology, including IRT [16] , MIRT [1] , and the neural networks-based models, including NCDM [40] , RCD [12] , KaNCD [41] and DCD [8] .Evaluation Since cognitive states cannot be directly observed in practice, it is common to indirectly evaluate CDMs through the student performance prediction task on test datasets [4] . To evaluate prediction performance, we adopt ACC and AUC and F1-score as metrics from the perspective of classification, using a threshold of 0.5, and RMSE as metrics from the perspective of regression, following previous work [12, 23] .Settings We set the dimension size d as 20, the layer of graph modeling as 2, and the mini-batch size as 512. In the training stage, we select the learning rate from {0.002, 0.005, 0.01, 0.02, 0.05}, select α from {0.05, 0.1, 0.5, 1} and β from {0.25, 0.5, 1}, and select neighboring number K from {1, 2, 3, 4, 5, 10, 15, 20, 15, 30, 25, 40, 45, 50}. All network parameters are initialized with Xavier initialization [15] . Each model is implemented by PyTorch [37] and optimized by Adam optimizer [19] . Specially, for the implementation of baselines, we set the dimensional sizes of each representation in PMF, NCDM, KaNCD, RCD and DCD as the number of knowledge concepts. All experiments are conducted on a Linux server equipped with two 3.00GHz Intel Xeon Gold 5317 CPUs and two Tesla A100 40G GPUs.Prediction Comparison We evaluate prediction performance of Coral against baselines under three setups: normal, sparse, and cold-start scenarios.Table 2 reports the performance comparison under normal settings for all the models across three datasets on four evaluation metrics. In this setting, we split all the datasets with a 7:1:2 ratio into training sets, validation sets, and test sets. The proposed Coral model significantly outperforms most baselines. This demonstrates two key benefits of Coral. First, the iterative graph construction process effectively generates collaborative connections for modeling. Second, the co-disentangled representation learning successfully discovers disentangled cognitive states for each learner.We extend our analysis to assess the performance of Coral in sparse scenarios. In order to simulate varied sparse environments, we systematically discard 80%, 60%, 40%, and 20% of the training data from the ASSIST dataset under the normal settings described above. The experimental resultsshown in Figure 3 (a) reveal that Coral consistently outperforms baselines across a range of sparse environments. Moreover, our model exhibits robust performance consistently, demonstrating its adaptability and effectiveness in diverse sparse scenarios. The model performance exhibits improvement as K increases, particularly noticeable when K is small. This observation suggests that the interlearner information automatically retrieved by Coral contributes positively to the model. However, once K surpasses a threshold, the performance gain becomes less pronounced. This diminishing effect arises because users beyond the threshold (i.e., K = 10 in this dataset) may lack significant collaborative relationships, thus limiting the useful clues they can offer. We observe that when K exceeds the threshold, the model's performance remains acceptable, and even the performance improves after K exceeds 30. This indicates that Coral effectively perceives the similarity functions of the scenario and collaborative context. Consequently, it assigns lower similarity scores to non-collaborative neighbors, robustly adjusting the attention weight in graph modeling.To obtain a more intuitive insight into the iterative graph construction process, we randomly select two learners (called target learners) from the Junyi dataset (with the normal setup) and visualize their neighbor selection process. Initially, we utilize t-SNE [39] to present the aggregated cognitive vector of each learner u, which can be obtained by aggregating each disentangled cognitive component learned by Coral (setting K = 30), i.e.,z(c) u . The embedding of the target learner is highlighted in red, while the nodes representing neighboring learners are color-coded based on the selection steps, with unselected points displayed in gray. The outcomes are illustrated in Figure 4 (a) , showcasing how Coral organizes neighbors according to cognitive states and exemplifying a compelling strategy for neighbor selection that takes into account cognitive similarity.We evaluate the disentanglement level achieved by assessing independence of dimensions within z u . The independence level IL(u) of each z u is quantified asIL(u) = 1 C C c=1 2 d(d-1) 1≤i,j≤d |z (c) u [i] -z (c) u [j]|, where z (c) u [i] represents the i th dimension of z (c)u , following a prior methodology [48, 42, 49] . In Figure 3 Explainability We further investigate the interpretability of the cognitive diagnosis outputs based on Coral. We aim to explore whether Coral can provide reasonable predictions for knowledge concepts that learners have not practiced in the training set during actual inference. Firstly, we randomly select a target student u from the Junyi dataset and identify 5 knowledge concepts (denoted as A ∼ E) that u has not learned in the training data. Subsequently, based on the refined model, we retrieve the top 4 most similar neighboring learners (i.e., u 1 ∼ u 4 ) to the target student u. Figure 5 depicts the assessed knowledge concepts (A ∼ E) and the corresponding mastery levels of selected neighbors using a radar chart. Table 3 presents the diagnostic outputs for the proficiency of u, along with 5 questions related to knowledge A ∼ E, the predicted scores answering correctly and the actual performances of u. We observe that, despite the cold-start nature of these knowledge concepts for Coral, the model effectively outputs cognitive states that align with the true performance of u by considering the mastery levels of collaborative learners with similar cognitive states. This sufficiently demonstrates the interpretability of Coral's diagnostic results.  
conclusion: 
 We are pioneering the exploration of collaborative cognitive diagnosis by disentangling the implicit cognitive representations of learners. Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over SOTA methods across several real-world datasets. We believe this endeavor marks a crucial step towards collaborative modeling for "AI Education". Furthermore, this work offers valuable insights into conscious-aware learner modeling, under the assumption that human learner proficiency can be effectively represented in a disentangled manner.A Proofs Property 1. max log p Θ (x u ) is bounded as follows:EQUATION)The proof is as follows.Proof.EQUATIONwhich completes the proof.Property 2. The D KL (•) in Eq. (2) can be rewritten as:EQUATION)The proof is as follows.Proof. Given that p Θ (x u ) = p data (x u ) and q Θ (z u , x u ) = q Θ (z u | x u ) p Θ (x u ), we then haveEQUATIONwhere I(A; B) calculates mutual information (MI) between A and B, i.e. I(A; B) =E p(a,b) [log p(a|b) p(a)]. Therefore, the proof is completed.• NeurIPS2020EC [43] This dataset is originated from NeurIPS 2020 Education Challenge, which provides learners' practice logs on mathematical questions from Eedi4 . We randomly select 1,000 learners with more than 15 practice records from NeurIPS2020EC to guarantee that each learner has enough data for diagnosis.  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: These information is discussed in Appendix E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Not Applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers or website links about the dataset and open-source codes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We additionally perform ablation studies to assess the impact of key components within Coral. The results in Table 4 depict the performances of Coral (setting K = 5) under various conditions: without the KL term for encoding (w/o KL), without the collaborative aggregation during decoding (w/o collar), and replacing the collaborative graph construction procedure using a knn-based methods (w/ knn) used in [13] on the ASSIST dataset. These findings show the effectiveness of each key component in enhancing the overall performance of Coral.We additionally implement three efficiency optimization strategies to further reduce the complexity of Coral. These strategies cannot theoretically guarantee optimal performance, but they can enhance applicability and scalability of Coral through empirical balancing of efficiency and accuracy. We refer them as Coral with n-sample, Coral with m-selections, and Coral with full-kit, as follows:• Coral with n-sample: During the K iterations of searching for neighbors, randomly sample n subsets from all M learners to replace V in the original approach. This reduces computational efficiency from M × K to n × K, where n ≪ M . • Coral with m-selections: Based on the basic Coral, replace selecting one neighbor per iteration with selecting m neighbors. This decreases computational efficiency from M × K to M ×K m , where m < K. • Coral with full-kit: A combination of Coral with n-sample and Coral with m-selections, further reducing computational efficiency from M × K to n×K m , where n ≪ M and m < K.Following the three strategies outlined above, we conduct several experiments on the Junyi dataset, with K = 40, to assess prediction performance. The results are summarized in the Figure 6 . These experimental results demonstrate Coral's potential to improve computational efficiency while maintaining acceptable performance, as evidenced by the varying levels of accuracy achieved with different optimization strategies.This research delves into modeling human cognitive states within the realm of intelligent education. The proposed Coral model significantly enhances the diagnostic accuracy of implicit learners' knowledge states. This improvement not only provides effective insights for online personalized tutoring services, such as question recommendations but also lays the foundation for further research in this area. Moreover, the automatic construction strategy for collaborative connections among learners offers valuable insights that can contribute to subsequent investigations in this field. Lastly, we anticipate that the proposed techniques can be extended to other domains, including but not limited to user interest modeling and social network modeling. Although our method is effective both theoretically and empirically, it suffers from computational inefficiencies. We have explored preliminary optimization strategies in the Appendix D.2 and will focus on improving computational efficiency in future research. In addition, future research plans to consider issues of fairness [51, 52] and explore the integration of large language models and multi-modal knowledge to enhance interpretability [27, 29, 30] . In essence, our work is dedicated to advancing intelligent education and deepening the understanding of human cognitive proficiency. It cannot cause negative effects. We anticipate its crucial role in fostering progress in both pertinent technologies and societal advancements.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://sites.google.com/site/assistmentsdata/https://www.junyiacademy.org/https://eedi.com/ 


Decision: This paper introduces Coral, a Collaborative Cognitive Diagnosis model utilizing Disentangled Representation Learning. It enhances cognitive diagnosis by dynamically capturing collaborative signals among learners, leading to improved performance over state-of-the-art methods in several educational datasets. The proposed methodology shows promise in advancing personalized education through more nuanced insights into learners’ cognitive states.

**Strengths**
- Novelty: The paper introduces a unique approach by integrating collaborative signals among learners using disentangled representation learning. This allows for a better understanding of learners’ cognitive states and could enhance personalized educational interventions (Reviewers vXdf, bUz3).
- Strong Empirical Results: The model demonstrates superior performance across various real-world datasets, outperforming state-of-the-art methods in several metrics, including ACC, AUC, F1-score, and RMSE, highlighting its practical utility in both cold-start and sparse data scenarios (Reviewers Ty5J, t9qg).
- Explainability: By disentangling cognitive states, the Coral model provides greater explainability and controllability in diagnosing students’ cognitive states, which can be valuable for educators in understanding student learning patterns (Reviewers Ty5J, vXdf, UPTU).
- Collaborative Graph Learning: The model’s iterative graph construction process allows it to dynamically search for optimal neighbors, enhancing the model’s capability to leverage collaborative signals (Reviewers Ty5J, vXdf).

**Areas for Improvement (Weaknesses)**
- Computational Inefficiency: The model suffers from high computational costs, which may hinder its scalability and practical application in large-scale educational settings (Reviewers Ty5J, UPTU, bUz3).
- Limited Generalizability: The choice of datasets may limit the model’s generalizability to other educational contexts, as the current experiments focus on specific scenarios (Reviewers bUz3, Ty5J, t9qg).
- Weak Interpretability Evaluation: Although the model claims improved explainability, the visualizations (such as t-SNE) may not be sufficient to provide meaningful interpretability. More rigorous qualitative analysis and ablation studies are needed to evaluate the disentanglement quality (Reviewers Ty5J, bUz3).
- Sensitivity to Hyperparameters: The model’s performance is highly sensitive to hyperparameters like the number of neighbors (K) and beta value. This reliance could make it difficult for practitioners to fine-tune the model for different datasets (Reviewer Ty5J).

The Authors' actively and extensively participated in the rebuttal and Author-Reviewer discussion phase. They acknolwedged and addressed most Reviewer and AC concerns and adhered to most of the reasonable and valuable suggestions provided by the Reviewers and AC. This strengthens their work significantly from its initial state.

An overview of key concerns categorized by the degree to which they were addressed is listed below:

.1. Addressed Concerns:
- Data diversity and additional baselines: The Authors added a new dataset and ran additional experiments on suggested baselines like VIBO, AGCDM, VarFA, and SCD, with Coral outperforming them consistently. This strengthened the original claims about Coral’s superior performance.
- Hyper-parameter sensitivity: They conducted additional sensitivity experiments for different embedding dimensions, which showed Coral’s performance was stable within a reasonable range.
- Interpretability: The Authors clarified that disentanglement improved interpretability by aligning cognitive states and collaborative states and offered clearer explanations of how Coral’s predictions could be interpreted by educators.
2. Partially Addressed Concerns:
- Overfitting: While the Authors explained that they used early stopping and conducted experiments on data sparsity and cold-start scenarios, concerns about overfitting were acknowledged but not fully explored with extensive validation metrics.
- Explainability: The Authors provided examples of explainability in Appendix E, and additional conclusions were offered to support their claim. However, Reviewer concerns about limited exploration of how individual cognitive states relate to knowledge concepts were only partially addressed.
3. Negatively Addressed or Unaddressed Concerns:
- Computational efficiency: The Authors admitted the model’s inefficiency but offered practical solutions without significant exploration of how these might affect large-scale applications. This limitation may affect the practical deployment of Coral in real-world, large-scale settings.
- Ethical considerations: The Author response included a discussion of potential privacy and fairness concerns but lacked a detailed ethical framework or proposed safeguards for applying the model in real-world education systems.

Overall, the Author rebuttals generally improved the submission by addressing key concerns raised in the reviews, although some areas, particularly around computational efficiency and ethical considerations, remained only partially addressed.

Key considerations for my recommendation include (1-3 are positive, 4 is a remaining weakness, but has been acknowledged by the Authors):

1. Author Rebuttal Quality: The Authors have thoroughly addressed most of the Reviewers’ concerns. For instance, they provided new baseline comparisons, further hyper-parameter sensitivity analyses, and new datasets, including the PISA 2015 dataset. These additions strengthened the evidence for the model’s robustness, handling of cold-start scenarios, and disentanglement quality.
2. Reviewer Feedback: The Reviewers generally upgraded their ratings after the Author rebuttal phase, reflecting a consensus toward acceptance. Reviewer Ty5J increased their score after the rebuttal, acknowledging improvements in the paper.
3. Strengths Reinforced: The novelty and robustness of the Collaborative Cognitive Diagnosis model with Disentangled Representation Learning (Coral) was strongly supported by the empirical evidence provided by the Authors. The Authors successfully demonstrated the model’s generalizability, performance on diverse datasets, and its advantage over existing state-of-the-art methods.
4. Remaining Weaknesses: While computational inefficiencies and concerns about scalability remain, the Authors have proactively addressed these limitations with proposed solutions, and these do not appear to detract significantly from the overall contribution of the work.

The Author rebuttal phase was constructive, with several key concerns being effectively addressed. However, to further strengthen the submission, the Authors should address the identified weaknesses, particularly in terms of computational efficiency, ethical considerations, and further hyperparameter sensitivity analyses. 

The paper makes positive contributions to collaborative* cognitive diagnosis with its novel approach, technical rigor, and strong empirical results. Overall, Coral’s impact on the field of educational data mining and cognitive diagnosis is promising. Regardless of whether the paper is accepted, it is strongly suggested that the Authors implement the changes that they agreed to for their paper. If the paper is accepted, then the Authors are expected to do so.

Reviews: review: paper_summary: paper_summary: The paper presents a novel model called Coral, aimed at enhancing cognitive diagnosis in educational contexts by leveraging collaborative signals among learners. The authors argue that learners with similar cognitive states often exhibit comparable problem-solving performances, and thus, understanding these collaborative connections can significantly improve the diagnosis of individual cognitive states.
Key Contributions - 
1. Disentangled Representation Learning: Coral introduces a disentangled state encoder to effectively separate the cognitive states of learners. This approach allows for clearer interpretations of individual knowledge proficiencies by disentangling the various cognitive factors that influence learning performance.
2. Collaborative Graph Construction: The model incorporates a context-aware collaborative representation learning mechanism that dynamically constructs a graph of learners. This graph is built by identifying optimal neighbors based on their cognitive states, facilitating the extraction of collaborative information.
3. Co-disentanglement Process: Coral achieves co-disentanglement by aligning the initial cognitive states with collaborative states through a decoding process. This integration enhances the model's ability to reconstruct practice performance while providing a more nuanced understanding of learners' cognitive states.
4. Empirical Validation: The paper reports extensive experiments demonstrating that Coral outperforms state-of-the-art methods across several real-world datasets, highlighting its effectiveness in accurately diagnosing cognitive states through collaborative learning.
Overall, this work addresses significant gaps in existing cognitive diagnosis methods by simultaneously modeling both individual and collaborative learning information, paving the way for more personalized educational interventions.
strength_weakness: strength_weakness: 1. Limited Generalizability of Experiments
While the paper demonstrates the effectiveness of the Coral model through extensive experiments on several real-world datasets, the choice of datasets may limit the generalizability of the findings. The experiments primarily focus on specific educational contexts, which may not capture the diverse learning environments and learner behaviors present in broader educational settings.
Actionable Insight: Future work should consider testing the model on a wider variety of datasets, including those from different educational domains (e.g., K-12, higher education, online learning platforms) and varying learner demographics. This would help validate the robustness of the model and its applicability across different contexts.
2. Lack of Comparison with Baseline Models
The paper claims significant improvements over state-of-the-art methods, but it does not provide a comprehensive comparison with a more wider range of baseline models. Especially collaborative filtering methods and other machine learning approaches that could serve as alternatives.
Actionable Insight: The authors should include a more diverse set of baseline models in their experiments to provide a clearer picture of Coral's performance. Additionally, discussing the strengths and weaknesses of these models in comparison to Coral would enhance the reader's understanding of the model's contributions.
3. Insufficient Exploration of Hyper-parameter Sensitivity
The model's performance could be sensitive to various hyper-parameters, such as the number of neighbors in the collaborative graph or the dimensions of the disentangled representations. However, the paper does not provide a thorough analysis of how these hyper-parameters impact the model's performance.
Actionable Insight: Conducting a sensitivity analysis on key hyper-parameters would help identify optimal settings and provide insights into the model's robustness. This could involve systematic experiments varying one parameter at a time while keeping others constant, followed by a discussion of the results.
4. Explainability and Interpretability
While the paper emphasizes the importance of disentangled representations for explainability, it lacks a detailed exploration of how the model's outputs can be interpreted in practical educational contexts.
Actionable Insight: The authors should include case studies or examples demonstrating how the model’s outputs can be interpreted and applied in real educational settings. This could involve visualizing the relationships between cognitive states and learner performance or providing guidelines for educators on how to utilize the model's insights.
5. Potential Overfitting Concerns
The complexity of the Coral model, with its multiple components and parameters, raises concerns about potential overfitting, especially if the training datasets are not sufficiently large or diverse. The paper does not address how the model's complexity is managed or validated against overfitting.
Actionable Insight: Providing validation metrics would strengthen the paper's claims regarding the model's generalization capabilities. 
6. Limited Discussion on Ethical Considerations
The paper does not address potential ethical implications of using collaborative cognitive diagnosis models in educational settings, such as data privacy concerns or the impact on learner autonomy and agency.
Actionable Insight: Including a section on ethical considerations would enhance the paper's comprehensiveness and relevance. This could involve discussing how to ensure data privacy, the implications of algorithmic bias, and strategies for maintaining learner agency in personalized educational interventions.
questions: questions: Dataset Diversity: What criteria did you use to select the datasets for your experiments? How do you ensure that the datasets adequately represent the diversity of learning environments and learner behaviors?

Baseline Comparisons: Can you provide more details on the baseline models you compared Coral against? Specifically, how do these models differ in their approach to cognitive diagnosis, and why were they chosen as benchmarks?

Hyper-parameter Sensitivity: Did you conduct a sensitivity analysis on the hyper-parameters of the Coral model? If so, what were the findings, and how do hyper-parameter choices impact the model's performance?

Interpretability of Results: How can educators and practitioners interpret the outputs of Coral in practical settings? Are there specific examples or case studies that illustrate how the model's insights can be applied in educational contexts?

Overfitting Concerns: What measures did you take to prevent overfitting in your model? Can you provide validation metrics or results from cross-validation to support the generalization capabilities of Coral? In your "data_loader.py" script you have provided a check for validation set on line 60 ("val_set.json") and also in the paper on line 279 under the Experimental setup --> Settings section you claim "split all the datasets with a 7:1:2 ratio into training sets, validation sets, and testing sets". However, no validation metrics or validation sets seem to be used.

Ethical Considerations: Have you considered the ethical implications of using collaborative cognitive diagnosis models in educational settings? How do you address potential concerns regarding data privacy and algorithmic bias?
limitations: limitations: The authors have not addressed the limitations and potential negative societal impact of their work, but there are still areas that could be improved. They could further elaborate on the generalizability of their findings beyond the specific educational contexts covered in the experiments. The authors do not explicitly discuss the potential negative societal impact of their work. This is an important aspect that should be addressed, as cognitive diagnosis models can have significant implications for learners, educators, and educational institutions. An in depth explanation should be provided pertinent to limitations on dimensions like : Data Privacy, Algorithmic Bias, Misuse of Diagnostic Information etc.
paper_summary: paper_summary: This paper introduces Coral, a novel model for collaborative cognitive diagnosis with disentangled representation learning, that can help in developing intelligent education systems. The key contributions are incorporating collaborative information among learners to improve cognitive state diagnosis by using disentangled representation learning to model cognitive states, and then using context-aware graph construction method to identify implicit collaborative connections. Coral demonstrates superior performance over state-of-the-art methods across several real-world datasets, showing particular strength in sparse and cold-start scenarios. The approach offers improved interpretability and adaptability in modeling learners' cognitive states, potentially advancing personalized tutoring and understanding of human learning processes.
strength_weakness: strength_weakness: * The model suffers from computational inefficiencies, which can limit its practical applicability, especially in large-scale educational settings. This issue is transparently acknowledged by the authors too, which is positive. 

* While the paper aims to achieve disentangled representations, the evaluation of disentanglement quality remains simple and limited. It will be valuable to add more ablation studies related to disentanglement quality and run qualitative analysis of how individual latent dimensions correspond to interpretable knowledge concepts across different datasets.

* The paper doesn't adequately address potential limitations of the collaborative approach, such as negative transfer or performance in edge cases. It will be valuable to discuss scenarios where the collaborative approach might not be beneficial or could lead to negative transfer. E.g., a highly advanced learner who struggles in one specific area may be grouped with generally low-performing learners
questions: questions: * How sensitive is the model performance to changes in key hyperparameters such as the number of neighbors (K) and the beta value? Adding a sensitivity analysis for these hyperparameters in the appendix can help assess the model's robustness and guide practitioners in tuning it for different datasets.

* Can you provide a breakdown of Coral's performance as the percentage of questions answered by new learners varies (e.g., 0%, 5%, 10%, 20%)? This would give a clearer picture of the model's efficacy in true cold-start scenarios.

* Is it possible to show how disentangled factors correspond to specific knowledge areas (since mapping to interpretable cognitive factors are not easy) across different datasets? This would strengthen the claim of improved interpretability over existing methods.
limitations: limitations: * The authors briefly mention computational inefficiencies as a limitation in Section 5. They also provide some preliminary optimization strategies in Appendix F.2, which is positive. However, it would be valuable to also expand it to other concerns like any potential bias in the modeling process, limitations in the datasets used or generalizability to other educational contexts, any limitations in the disentanglement approach to capture all relevant cognitive factors, etc

* Also the authors do not fully address potential ethical concerns and mitigation strategies related to privacy and fairness in cognitive modeling. E.g., privacy concerns related to modeling learner cognitive states, potential bias in the model that could disadvantage certain groups of learners, etc
paper_summary: paper_summary: The authors propose Coral, a Collaborative Cognitive Diagnosis model with Disentangled Representation Learning, aimed at improving our understanding of human learning by leveraging collaborative signals among learners. By disentangling cognitive states and dynamically constructing a collaborative graph, Coral captures implicit connections between learners, leading to more accurate cognitive diagnoses. The model's innovative approach involves an iterative process of encoding, collaborative representation learning, and decoding, which significantly enhances its performance over existing methods. Extensive experiments demonstrate Coral's superior effectiveness in real-world datasets, making it a promising tool for intelligent education systems.
strength_weakness: strength_weakness: 1. Coral's effectiveness relies heavily on the quality and quantity of learner data available. In scenarios where such data is sparse or noisy, the model's performance may not be as robust.

2. While the model shows great promise, its computational requirements for dynamically constructing collaborative graphs and learning representations might pose scalability issues, especially in very large educational environments.

3. There may lack some highly related baselines/papers for the cognitive diagnosis. 

[1] https://arxiv.org/pdf/2002.00276
[2] https://ieeexplore.ieee.org/document/10027634
[3] https://arxiv.org/pdf/2005.13107
questions: questions: See weaknesses.
limitations: limitations: No
paper_summary: paper_summary: This paper explores the Coral model, which combines disentangled representation learning with collaborative signals to enhance cognitive diagnosis in intelligent education. It focuses on identifying implicit collaborative links among learners to improve understanding of their cognitive states. The model integrates a context-aware collaborative graph learning mechanism to simulate the collaborative relationships between learners based on their cognitive states. By incorporating collaborative information into the disentangled cognitive states of learners, the model achieves collaborative disentanglement. Experiments on real datasets demonstrate Coral's performance compared to existing methods.
strength_weakness: strength_weakness: I expect that this paper will comfortably clear the bar for acceptance. However, I believe two main issues should be addressed first. Due to these reasons, I have set my score at a relatively borderline level but anticipate increasing it after the authors' rebuttal.

1. If two students have fewer interaction records, leading to increased similarity in their interaction patterns, they might exhibit different cognitive levels. Using the method outlined in the paper to learn their representations may yield similar results, but this might not accurately reflect the true situation.
2. The paper employs GNN to learn network representations, but the graph-based baseline comparison only includes RCD and does not include some recent graph-based cognitive diagnosis methods, such as [1].

Other/Minor issues (Optional):

1. In the second paragraph, the authors describe an example: experienced teachers can summarize students' similarities. Similarly, could large language models be used instead of real human teachers to construct collaborative graphs? I look forward to the authors discussing the integration with LLMs. Although this does not affect the readability and acceptance of the paper, I believe readers will be very interested.
2. In Table 6, "w/ full-kit (m,m)" should be "w/ full-kit (n,m)."
3. In Table 1, why are there two results for DCD on the NIPS2020EC dataset? It seems like results from two different runs.

```
[1] Wang, Shanshan, et al. "Self-supervised graph learning for long-tailed cognitive diagnosis." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.
```
questions: questions: 1. The motivation of this paper is to consider collaborative information among learners. What are the specific differences between the collaborative information proposed in this paper and the collaborative information in recommendation systems?
2. Do the ASSIST and NIPS2020EC datasets have similarities and prerequisite relationships among the knowledge concepts that support RCD?
3. In the example of question recommendation in Appendix E, are these five questions answered by learners in comparable situations (i.e., without acquiring new knowledge during the answering process)?
limitations: limitations: The authors discuss the limitations of the computational efficiency of the paper. Meanwhile, they also give 2 simple yet effective solutions to address current limitations in the appendix.
paper_summary: paper_summary: This paper presents a Collaborative Cognitive Diagnosis model, named Coral, which incorporates a disentangled state encoder, collaborative graphs, and a state decoder. The goal of Coral is to model collaborative and disentangled cognitive states using representation learning techniques.
strength_weakness: strength_weakness: 1. While the authors identify the challenge of obtaining explicit collaborative connections, they do not provide clear solutions to address this issue.

2. The heavy use of theoretical notations makes the paper less readable.

3. The detailed presentation of the methodology makes the experimental section appear crowded.

4. The performance improvements demonstrated by Coral are moderate.
questions: questions: 1. What is the performance impact if the concepts are not disentangled? Currently, there is insufficient motivation for integrating collaborative signals into the disentangled representations.
limitations: limitations: There is no significant negative societal impact identified in this work. The authors acknowledge an inefficiency issue within the model.
paper_summary: paper_summary: This paper introduces Coral, a model that enhances cognitive diagnosis by integrating collaborative signals among learners with disentangled representation learning. Coral addresses the importance of leveraging collaborative connections among learners to better understand human learning processes. Traditional cognitive diagnosis methods focus on individual attributions and explicit practice records, but they often overlook the potential of collaborative signals. The proposed Coral model aims to fill this gap by integrating collaborative and disentangled cognitive states. The disentangled state encoder separates the cognitive states of learners and reconstructs their practice performance from an inner-learner perspective. The collaborative representation learning dynamically constructs a collaborative graph by iteratively searching for optimal neighbors in a context-aware manner, effectively capturing collaborative signals. The model is evaluated on three real-world datasets: ASSIST, Junyi, and NIPS2020EC. The results show that Coral outperforms existing methods in terms of ACC, AUC, F1-score, and RMSE. Coral also performs well in sparse and cold-start scenarios, demonstrating its robustness and adaptability.
strength_weakness: strength_weakness: 1. The approach in this paper is not very innovative, and the motivation is not very convincing.
2. Although visualizing the embeddings provides some level of interpretability, it still does not yield any explicit conclusions or observable phenomena. Rather than visualizing the embeddings (t-SNE has a certain degree of randomness), I would prefer to see some interesting conclusions based on the dataset.
3. The paper mentions computational inefficiencies, which could limit the model's applicability in large-scale or real-time settings.
4. The model requires careful tuning of hyperparameters, which may not be straightforward for all datasets and might require extensive experimentation.
5. The paper could benefit from a more extensive comparison with a broader range of alternative approaches to strengthen its contributions.
questions: questions: Please see the weaknesses
limitations: limitations: n/a

Comments: title: Further Response to Question 2
Official Comment by Authors
comment: Dear reviewers, 

Thank you for recognizing our work as well as our rebuttal. We are pleased to have addressed your concerns and will revise the paper according to your suggestions.

We are very grateful to you for your suggestions on interpretability experiments from the data statistic perspective to help us improve the quality of our papers!

Thank you!

Best,

Authors
Thank you for the detailed responses and extra efforts. I have increased my score.
Thank you for your valuable comments and for recognizing the value of our work!

We are pleased to have addressed your concerns and will revise the paper according to your suggestions. If you have any other questions, please feel free to ask. We will do our best to address any concerns you may have.

Once again, we sincerely appreciate your encouragement and support.
Thank you for your response. The additional steps 1,2 and 3 in the public response provided by the authors definitely increase the paper's overall quality a bit, which I am reflecting in my slight upgrade in ratings. Thanks.
Dear Reviewer Ty5J,

We appreciate your careful reading and detailed feedback of our paper, and giving us the opportunity to answer your questions.

During the rebuttal phase, we have carefully addressed each of your concerns and are eager to receive your feedback for further discussion. We understand and appreciate the effort you put into reviewing the paper, so we have summarized our response for your quick reading:

1. Our motivation is well-supported by psychological theories, existing literature, and data analysis. Our model is the first exploration in the disentangled collaborative CD direction and is innovative. Please refer to the rebuttal for details.
2. There is a misunderstanding concerning our interpretability experiment, which is thoroughly addressed in the case study in Appendix E. Additionally, we further provide dataset-based conclusions in the rebuttal that answer your questions.
3. Thank you for your concerns regarding the computational efficiency limitation. We agree that this is a main bottleneck for our Coral model. To address this, our paper presents three preliminary strategies to improve efficiency, as detailed in Appendix F.2, and will focus on improving computational efficiency of Coral in future research. Additionally, we proactively acknowledge the computational efficiency limitations in the limitations section. According to the NeurIPS 2024 checklist guidelines, this proactive acknowledgment is considered a positive practice and should not be penalized.
4. Our experiments mainly require the tuning of two hyper-parameters ($\beta$ and $K$) with a fixed learning rate, which is not complex. The hyper-parameter tuning complexity is similar to DCD and RCD in baseline.
5. We add four additional baselines in the rebuttal.

Thank you again for your time and effort, and we will revise the paper following each of your suggestions. If you have any further questions or need clarification on any point in our response, please let us know. We will respond promptly.  

Best,

Authors.
Thank you for recognizing the value of our work! We are pleased to have addressed most of your concerns and will incorporate the suggested modifications into the revised version.

We apologize for the misunderstanding regarding Question 2. Your further explanation helped us better understand this concern.
Regarding **Question 2**, we would like to first clarify that such 5% and 10% cold-start phenomena are rare in real-world platforms unless it is a completely new platform. Current CD research focuses on learner-side cold-start scenarios where:
1. The learning platform has already accumulated rich learner data, and only a portion of learners are new [1].
2. A new business domain lacks learner data, but there is existing learner data from other related businesses [2].

In fact, **our paper's Figure 3(b) has already validated the model performance under the second cold-start setup**, where the training data is the mature domain and the test scenario is the cold-start domain.

Based on your detailed feedback, we further conduct the suggested **cold-start experiments**: selecting the earliest 5% and 20% of exercise data from learners in chronological order to train the model and then validating performance.

|ACC|IRT|NCDM|KaNCD|**Coral**|
|-|-|-|-|-|
|**New** learners with 5% data|0.56932|0.60573|0.60154|**0.64667**|
|**New** learners with 10% data |0.64228|0.67761|0.67296|**0.67993**|

The results remain consistent with the 5% and 10% sparsity settings. We infer that there are three main reasons for these experimental outcomes:
1. In online learning, most learners have very few records. Thus, low sparsity rates like 5% and 10% result in almost no data, with an average of only 3 and 6 records per learner in our ASSIST dataset. This sparsity situation closely resembles that of new learners with minimal data.
2. CD assumes a learner’s cognitive state remains constant, meaning the order in which questions are answered does not affect performance [3]. This means that for CD models, there is little difference between using the earliest 5% and 10% of data (cold-start) based on time and using a randomly selected 5% and 10% of data (data sparsity).
3. Online education allows learners to practice autonomously, so their early data typically does not suffer from focusing on only a small subset of questions. This enables us to train parameters for most questions even with a limited amount of data.

Thank you for your feedback and for the further explanation, which helped me improve our paper and better understand your concerns. We sincerely hope that our further clarification addresses your concerns.

---

[1] BETA-CD: A Bayesian meta-learned cognitive diagnosis framework for personalized learning. AAAI'23.

[2] Zero-1-to-3: Domain-Level Zero-Shot Cognitive Diagnosis via One Batch of Early-Bird Students towards Three Diagnostic Objectives. AAAI'24.

[3] Towards a New Generation of Cognitive Diagnosis. IJCAI'21.
For question 2, authors discuss the performance of Coral in sparse scenario, whereas my question was around performance in cold start scenario. This would involve testing the model with new learners who have answered 5%, 10%, 20% of questions and comparing the results to understand how well the model handles true cold-start scenarios

Saying that, I would like to thank the authors for performing additional experiments and providing other clarifications. After reviewing the results, I would like to maintain my previous scores
Thank you for your prompt feedback and for recognizing the value of our work!

We are pleased to have addressed your concerns and will revise the paper according to your suggestions. If you have any further questions, please feel free to bring them up for discussion. We will address your concerns as quickly as possible!

Once again, thank you for your support and constructive feedback.
Thank you for your prompt feedback and for recognizing the value of our work!

We are pleased to have addressed your concerns and will revise the paper according to your suggestions. If you have any further questions, please feel free to bring them up for discussion. We will address your concerns as quickly as possible!

Once again, thank you for your support and constructive feedback.
I appreciate the authors’ response and the additional experiments. I have revised my review score to Accept.
Your additional experiments under noisy data scenarios are valuable. The explicit simulation of noisy training conditions and the clear presentation of results demonstrate Coral's robustness effectively.  

While it might be beneficial to also discuss potential future work on handling different types of noise, such as systematic biases or varying noise levels, we understand that this may be beyond the scope of the current study focused on cognitive tasks (does not require additional designed experiments). 

Addtionally, the inclusion of experiments with the suggested baselines is a significant improvement.  

Once again, thank you for your hard work and the thoughtful rebuttal. Your contributions are valuable, and I would like to improve my score.