Title: Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Learner Modeling

Intelligent Education System

Cognitive Diagnosis

Disentangled Representation Learning
Abstract: Learners sharing similar implicit cognitive states often display comparable observable problem-solving performances. Leveraging collaborative connections among such similar learners proves valuable in comprehending human learning. Motivated by the success of collaborative modeling in various domains, such as recommender systems, we aim to investigate how collaborative signals among learners contribute to the diagnosis of human cognitive states (i.e., knowledge proficiency) in the context of intelligent education.
The primary challenges lie in identifying implicit collaborative connections and disentangling the entangled cognitive factors of learners for improved explainability and controllability in learner Cognitive Diagnosis (CD). However, there has been no work on CD capable of simultaneously modeling collaborative and disentangled cognitive states. To address this gap, we present Coral, a $\underline{Co}$llabo$\underline{ra}$tive cognitive diagnosis model with disentang$\underline{l}$ed representation learning. Specifically, Coral first introduces a disentangled state encoder to achieve the initial disentanglement of learners' states.
Subsequently, a meticulously designed collaborative representation learning procedure captures collaborative signals. It dynamically constructs a collaborative graph of learners by iteratively searching for optimal neighbors in a context-aware manner. Using the constructed graph, collaborative information is extracted through node representation learning. Finally, a decoding process aligns the initial cognitive states and collaborative states, achieving co-disentanglement with practice performance reconstructions.
Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over state-of-the-art methods across several real-world datasets.
Our code is available at https://github.com/bigdata-ustc/Coral.

Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling

Abstract

Learners sharing similar implicit cognitive states often display comparable observable problem-solving performances. Leveraging collaborative connections among such similar learners proves valuable in comprehending human learning. Motivated by the success of collaborative modeling in various domains, such as recommender systems, we aim to investigate how collaborative signals among learners contribute to the diagnosis of human cognitive states (i.e., knowledge proficiency) in the context of intelligent education. The primary challenges lie in identifying implicit collaborative connections and disentangling the entangled cognitive factors of learners for improved explainability and controllability in learner Cognitive Diagnosis (CD). However, there has been no work on CD capable of simultaneously modeling collaborative and disentangled cognitive states. To address this gap, we present Coral, a Collaborative cognitive diagnosis model with disentangled representation learning. Specifically, Coral first introduces a disentangled state encoder to achieve the initial disentanglement of learners' states. Subsequently, a meticulously designed collaborative representation learning procedure captures collaborative signals. It dynamically constructs a collaborative graph of learners by iteratively searching for optimal neighbors in a context-aware manner. Using the constructed graph, collaborative information is extracted through node representation learning. Finally, a decoding process aligns the initial cognitive states and collaborative states, achieving co-disentanglement with practice performance reconstructions. Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over state-of-the-art methods across several real-world datasets. Our code is available at https://github.com/bigdata-ustc/Coral .

1 Introduction

It is a common notion that individuals with similar implicit states frequently exhibit similar explicit behaviors. Therefore, establishing interconnections among similar users is crucial for understanding human behaviors. For instance, social connections play a pivotal role in understanding current consumer preferences and predicting future behaviors [42]. Similarly, in the

𝑞1

𝑞2

𝑞3

𝑞4

𝑞5

?

?

A

B

A, B

C, D

C, E

Knowledge

concepts

Questions

Legend

A: Function

B: Derivative

C: Number

D: Cube

E: Cone

/     : Correct / wrong logs

A

B

C

D

E

Cognitive states

Practice records

Proficiency

of concepts

context of intelligent education, a better modeling of like-minded learners with similar learning experiences, is essential for understanding the human learning process [31], analyzing their knowledge proficiency and facilitating personalized tutoring tailored to individual needs [55].

As illustrated in Figure 1, we can infer Nancy is likely to answer the Cone -related question q 5 correctly according to the correct practice responses of Bob and Alice, who share similar learning behaviors with Nancy. The underlying psychological assumption is that learners with similar experiences generally possess similar cognitive states - how well the learner masters each knowledge concept, influencing their subsequent responses. To gain a deeper understanding of the human learning process, it is crucial to explicitly diagnose unobservable cognitive states. Existing Cognitive Diagnosis (CD) methods seek to enhance diagnostic accuracy by fully utilizing the inner-learner information (i.e., individual attributions [41] and explicit practice records) and question-side features (e.g., difficulty [16], textual content [26], and educational relations [12, 14]). However, the issue of how similar (a.k.a. collaborative) connections among inter-learners with similar states facilitate understanding of learners' knowledge proficiency remains largely unexplored.

In this study, to efficiently harness collaborative information among similar learners and thereby more accurately diagnose the cognitive states of each individual, we advocate for the incorporation of inter-learner connections into the CD process. However, designing a collaborative CD model in educational scenarios presents two distinct challenges due to the complexity of human learning.

First, acquiring explicit collaborative connections among learners proves to be a formidable challenge. On the one hand, unlike many well-defined social scenarios (e.g., Twitter.com ), where user preference similarities are manifested through explicit social actions such as following and liking, the directly available social behaviors among learners in learning environments (e.g., LeetCode.com ) cannot be used for diagnosis modeling since these social attributes cannot reflect true cognitive-oriented connections. On the other hand, some related studies [28, 13] attempt to design different similarity functions based on practice data to compute cognitive similarities among learners. However, these approaches pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases. Although various methods for constructing user relationships have been proposed in other domains [22, 9], these approaches do not consider the domain-specific attributes of students in learning scenarios and cannot be directly applied in educational contexts.

Second, an ideal collaborative diagnosis procedure requires disentangling and uncovering the mixed explanatory latent factors hidden in the observed learning behaviors. The basic motivation is that learners demonstrate complex and diverse patterns driven by entangled states across both innerand inter-learner perspectives. For instance, from an inner perspective, Nancy may not master Cone since she does not practice Cone -related questions. However, based on inter-learner data, one can infer a high probability that she has mastered Cone . Most prior attempts can not fulfill this requirement since they learn representations in an entangled way. Although recent models [8, 50] achieve a dimension-level disentanglement of cognitive states, they lack consideration of modeling the influence of collaborative connections, ignoring the complex relations between inner- and inter-learner connections of different individuals. Thereby, it needs to find a suitable way to achieve the co-disentanglement from both the inner- and inter-learner views for cognitive representations with higher interpretability and controllability.

To tackle the above challenges, we propose Coral, a Collaborative cognitive diagnosis model with disentangled representation learning, to reveal learner cognitive states while simultaneously modeling both inner- and inter-learner learning information. Specifically, our approach begins with the disentangled cognitive representation encoding to establish initial disentangled learner states through reconstructing their practice performance from the inner-learner perspective. Next, our focus shifts to effectively learning collaborative cognitive representations from the inter-learner perspective. The most significant point is to find the implicit collaborative relations between learners. To address this challenge, we present a context-aware collaborative graph learning mechanism that automatically explores all K -optimal neighbors for each learner given their basic cognitive states to facilitate the explicit modeling of collaborative connections among learners. Based on the constructed graph, collaborative information can be effectively fused into disentangled learner cognitive states through learning collaborative node representations. Finally, a decoding and reconstruction process is conducted to merge initial states and collaborative states so as to achieve co-disentanglement from both the inner- and inter-learner perspectives. Extensive experiments demonstrate the superior performance of Coral, showing significant improvements over SOTA methods across several datasets.

2 Related Work

Cognitive Diagnosis As a fundamental task, cognitive diagnosis (CD) has been well-researched for decades in educational psychology [25, 4, 53]. It aims to profile the implicit cognitive states (i.e., the proficiency of specific knowledge concepts) of learners by exploiting observed practice records (e.g., correct or wrong). Existing research on CD assumes that learners' knowledge proficiency is proportional to their practice performance and thus can be diagnosed through predicting their practice performance [12]. Since the diagnostic results can be applied to many intelligent applications, such as exercise recommendation [18] and learning path suggestions [55], many CD models have been proposed in recent years. The early works from psychology like IRT [16] and MIRT [1] focus on modeling learners' answering process by predicting the probability of a learner answering a question correctly, which utilizes latent factors as the learner's ability. These methods lack interpretability, i.e., they are inability to output explicit multidimensional diagnostic results on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of questions to diagnose learners' proficiency on all knowledge concepts [38, 45, 46, 32]. Representative NCDM [40] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works (e.g., IRT, and MIRT). In summary, existing CD studies enhance diagnostic accuracy by fully utilizing the inner-learner information (i.e., individual attributions and explicit practice records) [41, 50]and question-side features (e.g., difficulty [16, 38], textual content [26], and educational relations [12, 14, 8]). However, to the best of our knowledge, the problem of collaborative diagnostic modeling remains largely unexplored.

Collaborative modeling in Education Collaborative connections among learners in the education context commonly refer to learners with similar explicit practice behaviors, testing scores and implicit knowledge proficiency [28, 54, 44]. However, due to the complexity and implicitness of the human learning process, these relations are commonly not explicitly and directly available. Existing studies [28, 13] in AI Education have attempted to design different similarity functions based on practice data to compute cognitive similarities among learners. However, these methods pose a significant challenge of manually selecting appropriate metrics and corresponding thresholds, introducing additional inductive biases.

Disentangled Representation Learning Disentangled Representation Learning (DRL)[3], which aims to produce robust, controllable, and explainable representations, has become one of the core problems in machine learning. Typical methods include variational method [20], weakly supervised models [21], as well as the recent combination with the diffusion model [6]. DRL has a wide range of applications in user modeling to disentangle attributes. For example, recommendation with several aspects of users' interests [24, 33], fair user representation to disentangle sensitive attributes [10]. In education, DCD [8] attempts to disentangle learners' cognitive representations via variational framework, which motivates us to conduct a further study on collaborative CD setups.

3 Coral

We first introduce the problem setup, followed by details on three core components of Coral: i) Disentangled Cognitive Representation Encoding, ii) Collaborative Representation Learning and iii) Decoding and Reconstruction. Figure 2 shows the framework. The algorithm is listed in Algorithm 1.

3.1 Problem Setup

Our setup considers the human learning dataset D including the practice records between M learners and N questions. The practice records of each learner u is denoted by x u = { x u,i } , where x u,i equals 1 or 0 , representing that learner u answered question i correctly or not, respectively. Each question is related to at least one knowledge concept. The association relations between N questions and C knowledge concepts is represented by C = { c i } N i =1 , where c i ∈ R C and c i,c equals 1 or 0 denoting that question i is related to concept c or not. The practice records are regarded as the explicit inner-learning information in our context.

Besides, we consider the collaborative connections among learners with similar cognitive states, which provide the inter-learner information. We define collaborative connections as a graph structure G = ( V, E ) which contains a set of nodes (i.e., learners) V and a set of edges E where ( u, v ) ∈ E or ( u, v ) ∈ G indicates that the existence of a collaborative connection between learner u and v (i.e., u and v have similar latent cognitive states). Notably, the collaborative connections in educational

1

3

5

2

4

8

6

9

𝒩(𝝁 1 , 𝝈 (1)

)

𝒩(𝟎,𝜎0

2

𝐈)

𝒩(𝝁 1 , 𝝈 (1)

)

𝒩(𝟎,𝜎0

2

𝐈)

𝒩(𝝁 1 , 𝝈 (1)

)

𝒩(𝟎,𝜎0

2

𝐈)

𝑢

𝑢

𝑢

x𝑢,1

x𝑢,2

x𝑢,3

x𝑢,5

x𝑢,4

x𝑢,9

x𝑢,8

x𝑢,6

𝐳 u

(1)

𝐳 u

(2)

𝐳 u

(3)

Disentangled Cognitive Representation Encoding

Graph construction at (

k

-1)-th step

Collaborative Representation Learning

𝑟

𝑢

1 ,𝑘-1

𝑟

𝑢

2 ,𝑘-1

𝑟

𝑢

3 ,𝑘-1

𝐺(1)

𝐺(2)

𝐺(3)

Graph modeling

𝑢

𝐫 𝑢

(1)

𝐫 𝑢

(2)

𝐫 𝑢

(3)

1

3

5

2

4

8

6

9

x𝑢,1

x𝑢,2

x𝑢,3

x𝑢,5

x𝑢,4

x𝑢,9

x𝑢,8

x𝑢,6

ෝ

ෝ

ෝ

ෝ

ෝ

ෝ

ෝ

ෝ

…

…

…

…

Decoding and

Reconstruction

𝐷KL

𝐷KL

𝐷KL

Notations:

/      Question practicing correctly / incorrectly

Learner under one particular knowledge concept

𝑢

𝑢

𝑢

scenarios are generally not explicitly or directly available, and it needs to design an adaptive strategy to automatically infer similar learners from observed learning data during the training process.

To achieve cognitive state disentanglement, we initially assign C factorized representations to each learner, i.e., z u = [ z (1) u ; z (2) u ; . . . ; z ( C ) u ] ∈ R d × C with Gaussian Mixture initialization since the Gaussian distribution has long been recognized as a proper statistic model for the cognitive states of learners in educational psychology [4]. The component z ( c ) u is expected to capture the learner u 's cognitive state over knowledge concept c . We denote Θ as the set of trainable parameters for the proposed model. Based on the above setups, the goal of Coral is to learn co-disentangled representations ˜ Z = { ˜ z u } M u =1 for the M learners from both the inner-learner practice perspective and inter-learner collaborative perspective.

3.2 Disentangled Cognitive Representation Encoding

The practice response x u of each learner u provides valuable inner-learner insights regarding his/her proficiency since learners' performance on each question is assumed to be proportional to their cognitive proficiency on question-related knowledge concepts [8]. Therefore, we implement an encoder for encoding the disentangled cognitive state z u of each learner u by reconstructing their practice responses. For a learner u , we assume that his/her practice performance on candidate questions can be generated from the following distribution:

where p ( C ) = p D ( C ) and p Θ ( x u | z u , C ) is naturally a cognitive diagnosis procedure to predict practice performance. The key point of this task is to learn an optimal encoder p Θ ( z u ) via practice records x u to encode the cognitive state z u of each learner u . To optimize Θ , we introduce a variational distribution q Θ ( z u | x u ) to approximate p Θ ( z u ) , following the V AE literature [3], through maximizing a lower bound of log p Θ ( x u ) based on the following property.

Property 1. maxlog p Θ ( x u ) is bounded as follows:

See the Appendix A for the proof.

In Property 1, the first term reconstructs the true practice performance x u of learner u and the variational encoder q Θ ( z u | x u ) in the second term approximates the true encoder p Θ ( z u ) by minimizing the KL divergence D KL. The variational distribution q Θ ( z u | x u ) and the expectation E q Θ ( z u | X u ) are intractable, thus we employ the re-parameterization trick [20] for the model optimization.

Furthermore, the diagnosis procedure p Θ ( x u | z u , C ) is achieved by estimating how well a learner u answers question i from both the perspectives of cognitive states and comprehensive abilities. From the perspective of cognitive states, solving question i requires learner u to master all knowledge concepts related to this question. Regarding comprehensive abilities, each learner possesses a latent state reflecting their overall learning ability, which is shared when addressing different questions.

Formally, this process can be described as:

where ϕ Θ ( · ) and ψ Θ ( · ) : R d → R + are two shallow neural networks. ψ Θ ( · ) estimates the comprehensive ability of the learner and ϕ Θ ( · ) predicts the performance of a learner with a given cognitive state z ( c ) u and a comprehensive ability θ u over question i in terms of concept c . h i is a learnable latent representation for question i . Besides, to ensure psychometric interpretability of prediction, we set the weights of ϕ Θ ( · ) are positive values, i.e., ∂ϕ Θ ( · ) /∂ z u > 0 , assuming that the probability of correctly answering the question monotonically increases with learners' cognitive state. Please note that we found that the mean operation here can also be replaced with a neural network (i.e., ϕ ′ Θ : R C → R + ) with positive weights, formulated as ϕ ′ Θ ( c i · ( θ u · z ( c ) u -h i )) , as in [40], without affecting prediction performance. Particularly, in contrast to most methods that consider entangled cognitive factors as input, our diagnosis model can better capture learners' proficiency on each knowledge concept by disentangling cognitive states under each concept.

Furthermore, inspired by the outstanding performance of β -TCVAE [7] in disentanglement, we prompt statistical independence among its dimensions to obtain a better trade-off between the reconstruction accuracy and the quality of disentangled representation through q ( z ( c ) u ) = ∏ d j =1 q Θ ( z ( c ) u,j ) where q Θ ( z ( c ) u ) is the aggregated posterior of z u , i.e., q Θ ( z ( c ) u ) = ∫ q Θ ( z ( c ) u | x u ) p ( x u ) d x u where p ( x u ) = p data ( x u ) . This setup is encouraged by the term D KL ( · ) in Eq. (2) based on Property 2.

Property 2. The D KL ( · ) in Eq. (2) can be rewritten as:

See Appendix A for the proof. On one hand, I ( z u , x u ) maximizes the mutual information (MI) between z u and x u which obtains the useful information for the diagnosis task as much as possible according to the information bottleneck theory [2]. On the other hand, given a Gaussian distribution p Θ ( z ( c ) u ) = ∏ d j =1 p Θ ( z ( c ) u,j ) , the KL divergence term encourages independence among the dimensions of z ( c ) u by preventing each latent variable from deviating too far from specified priors. Compared to prior VAE-based CD models [50, 8], Coral additionally considers ability parameters from psychology [16] to enhance the expressive power of disentangled cognitive states.

Overall, we penalize Eq. (2) by a Lagrange multiplier β resulting in the following objective:

3.3 Collaborative Representation Learning

Collaborative information among similar learners provides an auxiliary inter-learner insight for cognitive representation learning. However, collaborative connections among learners with similar states are typically not readily accessible. To address this challenge, we design a context-aware graph construction strategy that searches similar neighbors automatically via the initial disentangled cognitive states. Based on the constructed collaborative graph, we can learn collaborative node representations by aggregating collaborative signals to generate collaborative cognitive states.

3.3.1 Context-aware Collaborative Graph Learning

The core goal of constructing the collaborative graph is to find K optimal neighbors for each learner node in V via their initial disentangled cognitive state { z ( c ) } C c =1 . For different knowledge concepts, the cognitive connections between the same learner pair are typically different. Thereby, it needs to search C groups of similar neighbors for each learner via each disentangled component z ( c ) . This means that we would generate C collaborative graphs, i.e., G = { G ( c ) } C c =1 . Each collaborative

graph G ( c ) under concept c is expected to characterize the cognitive similarities of learners regarding concept c . Formally, this task is defined as computing p Θ ( G | V, Z ) by identifying all the K similar neighbors for each learner covering each concept c . Let N ( c ) u denote the set of K similar neighbors for the learner u , the task can be described as:

where N ( c ) and Z ( c ) = { z ( c ) u } M u =1 are the neighbor set and feature set of all the learners regarding knowledge concept c , respectively. The number of N ( c ) u equals the combination of arbitrary K neighbors from all M learners for each learner node u under each concept c , i.e., ∣ ∣ ∣ N ( c ) u ∣ ∣ ∣ = M ! K !( M -K )! , thus Eq. (6) is computationally expensive especially for larger M and K . To facilitate computation, we transform the Eq. (6) that requires global MI maximization to the task of maximizing MI locally via locally available context information inspired by [22] and derive a lower bound of it as the following Property 3.

Property 3. maxlog p Θ ( G | V, Z ) is bounded as follows:

See the Appendix A for the proof. The Eq. (7) iteratively searches K neighbors for the learner u under each knowledge concept c from step k = 1 to K . L ( c ) ,k u is the well-known InfoNCE loss function [36]. Let r ( c ) ,k -1 u denote the current context at step ( k -1) (i.e., the set of ( k -1) neighbors selected from step 1 to ( k -1) ). b ( c ) ,k u is the affinity candidate learner in the ( M -k ) nonneighbor learners. Let V ( c ) u denote the current set of nonneighbor learners, and we hence have b ( c ) ,k u ∈ V ( c ) u . f ( c ) ( b ( c ) ,k u ; r ( c ) ,k -1 u ) is a matching function measuring the similarity between of nonneighbor b ( c ) ,k u and the current context r ( c ) ,k -1 u , where the higher the scalar score means the higher likelihood of b ( c ) ,k u is a new neighbor.

Furthermore, we have L ( c ) ,k u ∝ f ( c ) ( b ( c ) ,k u ; r ( c ) ,k -1 u ) . Thus, given the context of ( k -1) neighboring learners (i.e., we have found ( k -1) neighbors for the learner u ) and matching function f ( c ) ( · ) , our goal following the Property 3 is to find a learner b ( c ) ,k u from nonneighbor set V ( c ) u that can maximize the matching score f ( c ) ( · ) as the k -th neighbor of u . In other words, p ( G | V, Z ) can be optimized through maximizing the matching score f ( c ) ( · ) from k = 1 to K iteratively. Thereby, at each step k , we sort the scores of the nonneighbor learners and select the learner with the highest score to label as k -th neighbor b ( c ) ,k u , i.e., b ( c ) ,k u ← arg max v f ( c ) ( v ; r ( c ) ,k -1 u ) , v ∈ V ( c ) u . After obtaining the k -th neighbor b ( c ) ,k u , the context r ( c ) ,k -1 u is updated to r ( c ) ,k u by absorbing b ( c ) ,k u .

The calculation of matching score f ( c ) ( · ) usually relies on the node representations (i.e., learner cognitive states). However, the sub-optimal cognitive state learning during the initial training epochs probably results in the matching function exhibiting biases. To enhance the stability of model training, instead of directly aggregating node representations as the context r ( c ) ,k -1 u as many graph learning works, we denote it using relative representations w.r.t. the learner u [22]. Without loss of generality, we first establish relative collaborative coordinate systems with learner node u as the origin, and process relationship measurements between node u and each of its neighbors v as z ( c ) u,v = ∥ z ( c ) u -z ( c ) v ∥ 2 . Then the context-aware features can be generated by aggregating each node in the context r ( c ) ,k -1 u , i.e., rc ( c ) ,k -1 u = ∑ v ∈ r ( c ) ,k -1 u z ( c ) u,v . Thereby, let v k denote b ( c ) ,k u with feature z ( c ) v k , we have f ( c ) ,v k = f ( c ) ( b ( c ) ,k u ; r ( c ) ,k -1 u ) = z ( c ) v k T · rc ( c ) ,k -1 u .

3.3.2 Collaborative Graph Modeling

After iteratively searching K neighbors under each concept, we can obtain C collaborative graphs regarding each learner, i.e., { G ( c ) } C c =1 . Then, we consider collaborative modeling as a node representation learning task within each collaborative graph G ( c ) . It relies on a nonlinear kernel function φ Θ ( · ) to aggregate neighboring information and update each disentangled cognitive state, i.e., r ( c ) u = φ Θ ( z ( c ) u , { z ( c ) v : ( u, v ) ∈ G ( c ) } ) . Given the disentangled learner cognitive states generated by the variational posterior distribution q Θ ( z u | x u ) from Property 1, φ Θ ( · ) is naturally expected to contain C channels to extract different concept features from similar learners, though projecting the representation z u into different subspaces, i.e., ˆ z ( c ) u = σ ( W T ( c ) z ( c ) u + b ( c ) ) / ∥ σ ( W T ( c ) z ( c ) u + b ( c ) ) ∥ 2 , where W ( c ) ∈ R d and b ( c ) ∈ R d are learnable parameters of channel c and σ ( · ) is a nonlinear activation function (e.g., Sigmoid), and ∥ · ∥ 2 is L 2 normalization ensuring numerical stability. Then the collaborative learner representation modeling in terms of concept c can be described as:

where s ( c ) u,v is the attention weight between u and v , considering both the collaborative aggregation (the first term) commonly used in graph modeling works and the corresponding context-aware attention (the second term) calculated in the iterative graph construction process in Eq. (7). When K is set large in Eq. (7), there is a possibility of introducing non-collaborative noise. In such cases, s ( c ) u,v can assign lower values to non-collaborative neighbors to mitigate the negative impact of noise, allowing for the adaptive tuning of attention in graph modeling. During training, the channels will remain changing because different subsets of the neighborhood will be searched for dynamically aggregating neighbor information in different iterations.

With Gaussian Mixture initialization from the Disentangled Cognitive Representation Encoding (section 3.2), we derive the theorem on convergence as:

Theorem 1. The Collaborative Representation Learning (section 3.3) procedure is equivalent to an expectation-maximization (EM) algorithm [35] for the mixture model. In particular, it converges to a point estimate of { r ( c ) u } C c =1 that maximizes the marginal likelihood l ( { a ( c ) v : ( u, v ) ∈ G ( c ) } C c =1 ; { r ( c ) u } C c =1 ) , where a ( c ) u,v equals 1 or 0 denoting whether learner v is a collaborative neighbor of learner u regarding concept c or not. See the Appendix A for the proof.

3.4 Decoding and Reconstruction

Given the initial disentangled encoding via inner-learner information (section 3.2) and the collaborative representation learning via inter-learner information (section 3.3), this part encourages an alignment between the initial encode z u and collaborative state r u , formulating a co-disentangled representation as ˜ z u = z u + r u . This operation is inspired by the residual block [17] to address the second challenge, where r u can be treated as a disentangled auxiliary information of z u from collaborative graphs.

The decoding process predicts the practice performance of each learner u on candidate questions, given her co-disentangled representation ˜ z u = [ ˜ z (1) u , ˜ z (2) u , . . . , ˜ z ( C ) u ] , i.e., p Θ (ˆ x u ) = E p Θ ( C ) [ p Θ (ˆ x u | ˜ z u , C )] , similar to the reconstruction procedure in Eq. (1). Thus, putting Eq. (5) and Eq. (7) together, we have the overall training objective:

where BCE ( · , · ) is the binary cross entropy loss function between ground-truth practice behaviors x u and the reconstructed ones. α and β are hyper-parameters.

By optimizing with minimizing the above loss function Eq. (9), the cognitive state ˜ z u of each learner u can be jointly refined serving as the diagnostic results. During the testing phase, we evaluate the model performance by matching the difference between the predicted score p Θ (ˆ x u,i ) and the true score x u . Specifically, when a proficiency value is required instead of the vector ˜ z ( c ) u , we can obtain it by averaging each dimension of ˜ z ( c ) u .

4 Experiments

We empirically evaluate the performances of the proposed Coral model over three real-world datasets and conduct several experiments to prove its effectiveness.

4.1 Experimental Setup

Datasets We conduct experiments on three real-world datasets: ASSIST [11], Junyi [5] and NeurIPS2020EC [43]. The statistics of datasets are listed in Table 1. The details about datasets and preprocessing are depicted in the Appendix C.

Baselines The baselines include the matrix factorization-based model, i.e., PMF [34], the typical latent factor models derived from educational psychology, including IRT [16], MIRT [1], and the neural networks-based models, including NCDM [40], RCD [12], KaNCD [41] and DCD [8].

Evaluation Since cognitive states cannot be directly observed in practice, it is common to indirectly evaluate CDMs through the student performance prediction task on test datasets [4]. To evaluate prediction performance, we adopt ACC and AUC and F1-score as metrics from the perspective of classification, using a threshold of 0.5, and RMSE as metrics from the perspective of regression, following previous work [12, 23].

Settings We set the dimension size d as 20 , the layer of graph modeling as 2 , and the mini-batch size as 512 . In the training stage, we select the learning rate from { 0 . 002 , 0 . 005 , 0 . 01 , 0 . 02 , 0 . 05 } , select α from { 0 . 05 , 0 . 1 , 0 . 5 , 1 } and β from { 0 . 25 , 0 . 5 , 1 } , and select neighboring number K from { 1 , 2 , 3 , 4 , 5 , 10 , 15 , 20 , 15 , 30 , 25 , 40 , 45 , 50 } . All network parameters are initialized with Xavier initialization [15]. Each model is implemented by PyTorch [37] and optimized by Adam optimizer [19]. Specially, for the implementation of baselines, we set the dimensional sizes of each representation in PMF, NCDM, KaNCD, RCD and DCD as the number of knowledge concepts. All experiments are conducted on a Linux server equipped with two 3.00GHz Intel Xeon Gold 5317 CPUs and two Tesla A100 40G GPUs.

4.2 Experimental Results

Prediction Comparison We evaluate prediction performance of Coral against baselines under three setups: normal, sparse, and cold-start scenarios.

Table 2 reports the performance comparison under normal settings for all the models across three datasets on four evaluation metrics. In this setting, we split all the datasets with a 7:1:2 ratio into training sets, validation sets, and test sets. The proposed Coral model significantly outperforms most baselines. This demonstrates two key benefits of Coral. First, the iterative graph construction process effectively generates collaborative connections for modeling. Second, the co-disentangled representation learning successfully discovers disentangled cognitive states for each learner.

We extend our analysis to assess the performance of Coral in sparse scenarios. In order to simulate varied sparse environments, we systematically discard 80%, 60%, 40%, and 20% of the training data from the ASSIST dataset under the normal settings described above. The experimental results

shown in Figure 3 (a) reveal that Coral consistently outperforms baselines across a range of sparse environments. Moreover, our model exhibits robust performance consistently, demonstrating its adaptability and effectiveness in diverse sparse scenarios.

Moreover, we conduct an analysis of Coral's performance in a cold-start environment. To replicate this scenario, we retain solely the coldstart response data for each learner in the test set of Junyi, corresponding to the knowledge concepts they had not previously practiced in the training set. The Figure 3 (b) illustrates the experimental results, highlighting the exceptional performance of the Coral model (with K = 10 ) in a cold-start scenario.

Collaborate Graph Learning We investigate the influence of the generated neighbor number K on diagnostic performance. Figure 3 (c) displays the prediction performances for various values of K on Junyi under the normal scenario. The model performance exhibits improvement as K increases, particularly noticeable when K is small. This observation suggests that the interlearner information automatically retrieved by Coral contributes positively to the model. However, once K surpasses a threshold, the performance gain becomes less pronounced. This diminishing effect arises because users beyond the threshold (i.e., K = 10 in this dataset) may lack significant collaborative relationships, thus limiting the useful clues they can offer. We observe that when K exceeds the threshold, the model's

performance remains acceptable, and even the performance improves after K exceeds 30 . This indicates that Coral effectively perceives the similarity functions of the scenario and collaborative context. Consequently, it assigns lower similarity scores to non-collaborative neighbors, robustly adjusting the attention weight in graph modeling.

To obtain a more intuitive insight into the iterative graph construction process, we randomly select two learners (called target learners) from the Junyi dataset (with the normal setup) and visualize their neighbor selection process. Initially, we utilize t-SNE [39] to present the aggregated cognitive vector of each learner u , which can be obtained by aggregating each disentangled cognitive component learned by Coral (setting K = 30 ), i.e., ∑ C c =1 ˜ z ( c ) u . The embedding of the target learner is highlighted in red, while the nodes representing neighboring learners are color-coded based on the selection steps, with unselected points displayed in gray. The outcomes are illustrated in Figure 4 (a), showcasing how Coral organizes neighbors according to cognitive states and exemplifying a compelling strategy for neighbor selection that takes into account cognitive similarity.

Disentanglement We evaluate the disentanglement level achieved by assessing independence of dimensions within z u . The independence level IL ( u ) of each z u is quantified as IL ( u ) = 1 C ∑ C c =1 2 d ( d -1) ∑ 1 ≤ i,j ≤ d | z ( c ) u [ i ] -z ( c ) u [ j ] | , where z ( c ) u [ i ] represents the i th dimension of z ( c ) u , fol-

∑ M

lowing a prior methodology [48, 42, 49]. In Figure 3 (d), we depict IL = u =1 IL ( u ) and the corresponding model performances at different training epochs on ASSIST (with the normal setup). Notably, Coral (setting K = 10 ) gradually achieves a high degree of disentanglement during the training process, and the model performances generally exhibit a positive correlation with the degree of disentanglement. This observation reveals the effectiveness of the disentanglement process.

Additionally, we visualize the disentangled cognitive component representations (i.e., ˜ z ( c ) u ) of each learner u learned by Coral. We treat each knowledge component in the representation as independent points, with each component colored differently. For visual clarity, we randomly select 200 learners and 5 knowledge concept components for display. The results in Figure 4 (b) uses t-SNE to visualize

(a)

(b)

(c)

(d)

0.70

0.75

0.80

0.85

0.90

3

5

10

15

20

25

30

35

40

ACC

AUC

F1-score

0.55

0.65

0.75

0.85

1

2

3

4

5

6

7

ACC

F1-score

IL

Value of

𝑲

Epoch

0.65

0.75

0.85

80%

60%

40%

20%

IRT

MIRT

PMF

NCDM

KaNCD

RCD

DCD

Coral

F1 score

0.55

0.7

0.85

Coral

DCD

RCD

KaNCD

NCDM

PMF

MIRT

IRT

Ratio of discarding

ACC

(b)

(a)

𝒌

-steps neighbors

30

2

20

01

ASSIST

Junyi

NIPS2020

learners' cognitive states, with each color representing a distinct category of knowledge-related learner states. This illustrates Coral's ability to achieve well-separated representations.

Explainability We further investigate the interpretability of the cognitive diagnosis outputs based on Coral. We aim to explore whether Coral can provide reasonable predictions for knowledge concepts that learners have not practiced in the training set during actual inference. Firstly, we randomly select a target student u from the Junyi dataset and identify 5 knowledge concepts (denoted as A ∼ E ) that u has not learned in the training data. Subsequently, based on the refined model, we retrieve the top 4 most similar neighboring learners (i.e., u 1 ∼ u 4 ) to the target student u . Figure 5 depicts the assessed knowledge concepts ( A ∼ E ) and the corresponding mastery levels of selected neighbors using a radar chart. Table 3 presents the diagnostic outputs for the proficiency of u , along with 5 questions related to knowledge A ∼ E , the predicted scores answering correctly and the actual performances of u . We observe that, despite the cold-start nature of these knowledge concepts for Coral, the model effectively outputs cognitive states that align with the true performance of u by considering the mastery levels of collaborative learners with similar cognitive states. This sufficiently demonstrates the interpretability of Coral's diagnostic results.

0.4

0.6

0.8

A

B

C

D

E

0.4

0.6

0.8

A

B

C

D

E

0.4

0.6

0.8

A

B

C

D

E

0.4

0.6

0.8

A

B

C

D

E

𝑢1

𝑢2

𝑢3

𝑢4

5 Conclusion

We are pioneering the exploration of collaborative cognitive diagnosis by disentangling the implicit cognitive representations of learners. Extensive experiments demonstrate the superior performance of Coral, showcasing significant improvements over SOTA methods across several real-world datasets. We believe this endeavor marks a crucial step towards collaborative modeling for 'AI Education'. Furthermore, this work offers valuable insights into conscious-aware learner modeling, under the assumption that human learner proficiency can be effectively represented in a disentangled manner.

Acknowledgments

This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2021YFF0901003), the Key Technologies R & D Program of Anhui Province (No. 202423k09020039) and the Fundamental Research Funds for the Central Universities.

References

Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Statistics Reference Online , 2014.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations , 2022.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8): 1798-1828, 2013.

Haoyang Bi, Enhong Chen, Weidong He, Han Wu, Weihao Zhao, Shijin Wang, and Jinze Wu. Beta-cd: A bayesian meta-learned cognitive diagnosis framework for personalized learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 5018-5026, 2023.

Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In EDM , pages 532-535, 2015.

Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation. 2023.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Xiangzhi Chen, Le Wu, Fei Liu, Lei Chen, Kun Zhang, Richang Hong, and Meng Wang. Disentangling cognitive diagnosis with limited exercise labels. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems , 33:19314-19326, 2020.

Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning , pages 1436-1445. PMLR, 2019.

Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with an online system that tutors as it assesses. User modeling and user-adapted interaction , 19:243-266, 2009.

Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages 501-510, 2021.

Weibo Gao, Qi Liu, Hao Wang, Linan Yue, Haoyang Bi, Yin Gu, Fangzhou Yao, Zheng Zhangm Xin Li, and Yuanjing He. Zero-1-to-3: Domain-level zero-shot cognitive diagnosis via one batch of early-bird students towards three diagnostic objectives. arXiv preprint arXiv:2312.13434 , 2023.

Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang. Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 983-992, 2023.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Robert J Harvey and Allen L Hammer. Item response theory. The Counseling Psychologist , 27 (3):353-383, 1999.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yujia Huo, Derek F Wong, Lionel M Ni, Lidia S Chao, and Jing Zhang. Knowledge modeling via contextualized representations for lstm-based personalized exercise recommendation. Information Sciences , 523:266-278, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems , 27, 2014.

Rongfan Li, Ting Zhong, Xinke Jiang, Goce Trajcevski, Jin Wu, and Fan Zhou. Mining spatiotemporal relations via self-paced graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 936-944, 2022.

Rui Li, Liyang He, Qi Liu, Yuze Zhao, Zheng Zhang, Zhenya Huang, Yu Su, and Shijin Wang. Consider: Commonalities and specialties driven multilingual code retrieval framework. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8679-8687, 2024.

Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 world wide web conference , pages 689-698, 2018.

Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST) , 9(4):1-26, 2018.

Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering , 33(1):100-115, 2019.

Chung Kwan Lo. What is the impact of chatgpt on education? a rapid review of the literature. Education Sciences , 13(4):410, 2023.

Ting Long, Jiarui Qin, Jian Shen, Weinan Zhang, Wei Xia, Ruiming Tang, Xiuqiang He, and Yong Yu. Improving knowledge tracing with collaborative information. In Proceedings of the fifteenth ACM international conference on web search and data mining , pages 599-607, 2022.

Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18733-18741, 2024.

Xinwei Long, Jiali Zeng, Fandong Meng, Jie Zhou, and Bowen Zhou. Trust in internal or external knowledge? generative multi-modal entity linking with knowledge retriever. In Findings of the Association for Computational Linguistics ACL 2024 , pages 7559-7569, 2024.

Aleksandra Luszczynska and Ralf Schwarzer. Social cognitive theory. Fac Health Sci Publ , pages 225-51, 2015.

Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang. Enhancing cognitive diagnosis using un-interacted exercises: A collaboration-aware mixed sampling approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8877-8885, 2024.

Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. Advances in neural information processing systems , 32, 2019.

Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. Advances in neural information processing systems , 20, 2007.

Todd K Moon. The expectation-maximization algorithm. IEEE Signal processing magazine , 13 (6):47-60, 1996.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Emiko Tsutsumi, Ryo Kinoshita, and Maomi Ueno. Deep-irt with independent student and item networks. International Educational Data Mining Society , 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6153-6161, 2020.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuralcd: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering , 2022.

Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum codisentangled representation learning across multiple environments for social recommendation. In International Conference on Machine Learning , pages 36174-36192. PMLR, 2023.

Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Instructions and guide for diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007.12061 , 2020.

Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative llm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15077-15087, 2024.

Shangshang Yang, Haoyu Wei, Haiping Ma, Ye Tian, Xingyi Zhang, Yunbo Cao, and Yaochu Jin. Cognitive diagnosis-based personalized exercise group assembly via a multi-objective evolutionary algorithm. IEEE Transactions on Emerging Topics in Computational Intelligence , 7(3):829-844, 2023.

Fangzhou Yao, Qi Liu, Min Hou, Shiwei Tong, Zhenya Huang, Enhong Chen, Jing Sha, and Shijin Wang. Exploiting non-interactive exercises in cognitive diagnosis. Interaction , 100(200): 300, 2023.

Fangzhou Yao, Qi Liu, Linan Yue, Weibo Gao, Jiatong Li, Xin Li, and Yuanjing He. Adard: An adaptive response denoising framework for robust learner modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3886-3895, 2024.

Linan Yue, Qi Liu, Yichao Du, Yanqing An, Li Wang, and Enhong Chen. Dare: disentanglementaugmented rationale extraction. Advances in Neural Information Processing Systems , 35: 26603-26617, 2022.

Linan Yue, Qi Liu, Ye Liu, Weibo Gao, Fangzhou Yao, and Wenfeng Li. Cooperative classification and rationalization for graph generalization. In Proceedings of the ACM on Web Conference 2024 , pages 344-352, 2024.

Yunfei Zhang, Chuan Qin, Dazhong Shen, Haiping Ma, Le Zhang, Xingyi Zhang, and Hengshu Zhu. Relicd: A reliable cognitive diagnosis framework with confidence awareness. In 2023 IEEE International Conference on Data Mining (ICDM) , pages 858-867. IEEE, 2023.

Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang, Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. Fairlisa: Fair user modeling with limited sensitive attributes information. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Zheng Zhang, Qi Liu, Zirui Hu, Yi Zhan, Zhenya Huang, Weibo Gao, and Qingyang Mao. Enhancing fairness in meta-learned user modeling via adaptive sampling. In Proceedings of the ACM on Web Conference 2024 , pages 3241-3252, 2024.

Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. Understanding and improving fairness in cognitive diagnosis. Science China Information Sciences , 67(5):152106, 2024.

Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, and Jing Wang. Lane: Logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. arXiv preprint arXiv:2407.02833 , 2024.

Yan Zhuang, Qi Liu, GuanHao Zhao, Zhenya Huang, Weizhe Huang, Zachary Pardos, Enhong Chen, Jinze Wu, and Xin Li. A bounded ability estimation for computerized adaptive testing. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

A Proofs

Property 1. maxlog p Θ ( x u ) is bounded as follows:

The proof is as follows.

Proof.

which completes the proof.

Property 2. The D KL ( · ) in Eq. (2) can be rewritten as:

The proof is as follows.

Proof. Given that p ( x ) = p ( x ) and q ( z , x ) = q ( z | x ) p ( x ) , we then have

where I ( A ; B ) calculates mutual information (MI) between A and B , i.e. I ( A ; B ) = E p ( a , b ) [log p ( a | b ) p ( a ) ] . Therefore, the proof is completed.

Property 3. maxlog p Θ ( G | V, Z ) is bounded as follows:

The proof is as follows:

Proof. Given the following inequality,

we further derive the term I ( N ( c ) ; Z ( c ) ) that search the combination of K similar neighbors from the permutation perspective as follows:

where H ( A ) = -∑ p ( a ) · log p ( a ) . R ( c ) u denotes the permutation of neighbors representing routes to N ( c ) u in given orders, where ∣ ∣ ∣ R ( c ) u ∣ ∣ ∣ = M ! ( M -K )! . However, the search space of R ( c ) u is huge and even prohibitive. Inspired by the equivalent task [22], we next present the Eq. (16) in a heuristic

style by maximizing the MI between the context of selected similar learner neighbors and the next neighbor iteratively.

Specifically, the core goal of Eq. (15) is to find all K neighbors N ( c ) u for each learner u under a specific concept the data at once, from the perspective of maximizing MI globally. However, this task is challenging due to the especially large search spaces of N ( c ) u and R ( c ) u . Thereby, we decompose the globally optimal task into an equivalent task in an iterative local optimal process. Concretely, assume that we have found ( k 0 -1) optimal neighbors for the learner u formulating a route r ( c ) ,k 0 -1 u in a specific order from learner node 1 to ( k 0 -1) , we then search the k 0 -th neighbor b ( c ) ,k 0 u equally from the rest ( M -k 0 +1) learners for the learner u , i.e., p ( b ( c ) ,k 0 u ) = 1 M -k 0 +1 .

Given arbitrary k 0 optimal neighbors for learner u with a specific sub-route r ( c ) ,k 0 u , we can derive p Θ ( R ( c ) u , z ( c ) u ) in Eq. (16) as follows:

We can derive the log term log p Θ ( R ( c ) u | z ( c ) u ) in Eq. (16) as follows:

With Eq. (17) and (18), we can derive the first term in Eq. (16) as follows:

where ϵ ( M,K ) ≥ 0 is a constant term regarding M and K . Inspired by [36], we have the lower bound of I ( b ( c ) ,k u ; r ( c ) ,k -1 u ) as follows:

The Eq. (20) iteratively searches K neighbors for the learner u under each knowledge concept c from step k = 1 to K . L ( c ) ,k u is the well-known InfoNCE loss function [36]. Let r ( c ) ,k -1 u denote the

current context at step ( k -1) (i.e., the set of ( k -1) neighbors selected from step 1 to ( k -1) ). b ( c ) ,k u is the affinity candidate learner in the ( M -k ) nonneighbor learners. Let V ( c ) u denote the current set of nonneighbor learners, and we hence have b ( c ) ,k u ∈ V ( c ) u . f ( c ) ( b ( c ) ,k u ; r ( c ) ,k -1 u ) is a matching function measuring the similarity between of nonneighbor b ( c ) ,k u and the current context r ( c ) ,k -1 u , where the higher the scalar score means the higher likelihood of b ( c ) ,k u is a new neighbor.

Furthermore, we have L ( c ) ,k u ∝ f ( c ) ( b ( c ) ,k u ; r ( c ) ,k -1 u ) . Thus, given the context of ( k -1) neighboring learners (i.e., we have found ( k -1) neighbors for the learner u ) and matching function f ( c ) ( · ) , our goal following the Eq. (20) is to find a learner b ( c ) ,k u from nonneighbor set V ( c ) u that can maximize the matching score f ( c ) ( · ) as the k -th neighbor of u . In other words, p ( G | V, Z ) can be optimized through maximizing the matching score f ( c ) ( · ) from k = 1 to K iteratively. Thereby, at each step k , we sort the scores of the nonneighbor learners and select the learner with the highest score to label as k -th neighbor b ( c ) ,k u , i.e., b ( c ) ,k u ← arg max v f ( c ) ( v ; r ( c ) ,k -1 u ) , v ∈ V ( c ) u . After obtaining the k -th neighbor b ( c ) ,k u , the context r ( c ) ,k -1 u is updated to r ( c ) ,k u by absorbing b ( c ) ,k u .

Based on the Eq. (15), Eq. (16) and Eq. (20), we have

which completes the proof.

Theorem 1. With Gaussian Mixture initialization from the Disentangled Cognitive Representation Encoding (section 3.2), the Collaborative Representation Learning (section 3.3) procedure is equivalent to an expectation-maximization (EM) algorithm [35] for the mixture model. In particular, it converges to a point estimate of { r ( c ) u } C c =1 that maximizes the marginal likelihood l ( { a ( c ) v : ( u, v ) ∈ G ( c ) } C c =1 ; { r ( c ) u } C c =1 ) , where a ( c ) u,v equals 1 or 0 denoting whether learner v is a collaborative neighbor of learner u regarding concept c or not. The proof is as follows.

Proof. The collaborative modeling process can be approximatively equivalent to an expectationmaximization (EM) algorithm for the mixture model. Let A = { A ( c ) } C c =1 where A ( c ) = { a ( c ) v : ( u, v ) ∈ G ( c ) } , where a ( c ) u,v equals 1 or 0 denoting whether learner v is a collaborative neighbor of learner u regarding concept c or not, which is a type unknown factor in EM algorithm. Given R = { r ( c ) } C c =1 and Z = { z ( c ) } C c =1 , the EM algorithm maximizes the likelihood l ( A ; R ) = ∑ A l ( A, Z ; R ) . Let q ( A ) is the distribution over A , we then have

Let L ( R , q ( A )) denote ∑ A q ( A ) · l ( A, Z ; R ) q ( A ) , we can rewrite Eq. (22) as

where L ( R , q ( A )) is a lower bound of l ( A ; R ) since the KL divergence from l ( A | Z ; R ) towards q ( A ) is non-negative.

At every E-step, the EM algorithm aims to search the optimal q ( A ) that tightens the lower bound, which q ( A ) is set to l ( A | Z ; R ) since the KL divergence will become zero. Given that

and Property 3, the optimal q ( A ) that tightens the lower bound can be achieved through iteratively maximizing the following objective as

which performs the E-step.

After the E-step, an M-step is performed to maximize the lower bound L ( R , q ( A )) w.r.t. R with q ( A ) fixed found in the E-step. Given that

we optimize the r ( c ) upon ∂L ( R ,q ( A )) ∂ r ( c ) to zero, which is actually performing the M-step.

Let q ( A ) k and r k be the result of the k -th E-step and the k -th M-step, respectively, we then have

Hence, this can prove that the likelihood will increase monotonically and be upper-bounded by zero at the same time. Therefore, the algorithm converges, which completes the proof.

B Algorithm

To offer a more comprehensive description of Coral's structure, we outline the algorithm (see Algorithm 1) for the entire model, including four functions and a main function.

C Dataset Description and Preprocessing

We conduct experiments on three real-world datasets, i.e., ASSIST [11], Junyi [5] and NeurIPS2020EC [43]. The statistics of these datasets are summarized in Table 1. For all datasets, we preserve the first-time exercise-answering record for the same learner-question pairs to support cognitive diagnosis aligning with common settings used in previous related studies [40]. The detailed information on datasets and preprocessing method are depicted as follows:

ASSIST (ASSISTments 2009-2010 'skill builder') [11] This dataset is an open dataset collected by the ASSISTments online tutoring systems 2 , which has become one of the popular benchmark datasets for cognitive diagnosis. We preserve learners with more than 30 practice records for ASSIST to guarantee that each learner has enough data for diagnosis.

Junyi [5] This dataset contains learner online learning logs collected from a Chinese online educational platform called Junyi Academy 3 . Nowadays, Junyi is widely used in the evaluation of online education tasks [47, 8]. We randomly select 1,400 learners with more than 15 practice records from Junyi to guarantee that each learner has enough data for diagnosis.

)

NeurIPS2020EC [43] This dataset is originated from NeurIPS 2020 Education Challenge, which provides learners' practice logs on mathematical questions from Eedi 4 . We randomly select 1,000 learners with more than 15 practice records from NeurIPS2020EC to guarantee that each learner has enough data for diagnosis.

D Additional Experimental Results

D.1 Ablation Study

We additionally perform ablation studies to assess the impact of key components within Coral. The results in Table 4 depict the performances of Coral (setting K = 5 ) under various conditions: without the KL term for encoding (w/o KL), without the collaborative aggregation during decoding (w/o collar), and replacing the collaborative graph construction procedure using a knn-based methods (w/ knn) used in [13] on the ASSIST dataset. These findings show the effectiveness of each key component in enhancing the overall performance of Coral.

D.2 Efficiency Improvement

We additionally implement three efficiency optimization strategies to further reduce the complexity of Coral. These strategies cannot theoretically guarantee optimal performance, but they can enhance applicability and scalability of Coral through empirical balancing of efficiency and accuracy. We refer them as Coral with n -sample, Coral with m -selections, and Coral with full-kit, as follows:

Coral with n -sample: During the K iterations of searching for neighbors, randomly sample n subsets from all M learners to replace V in the original approach. This reduces computational efficiency from M × K to n × K , where n ≪ M .

Coral with m -selections: Based on the basic Coral, replace selecting one neighbor per iteration with selecting m neighbors. This decreases computational efficiency from M × K to M × K m , where m<K .

Coral with full-kit: A combination of Coral with n -sample and Coral with m -selections, further reducing computational efficiency from M × K to n × K m , where n ≪ M and m<K .

Following the three strategies outlined above, we conduct several experiments on the Junyi dataset, with K = 40 , to assess prediction performance. The results are summarized in the Figure 6. These experimental results demonstrate Coral's potential to improve computational efficiency while maintaining acceptable performance, as evidenced by the varying levels of accuracy achieved with different optimization strategies.

E Broader Impact and Limitation

This research delves into modeling human cognitive states within the realm of intelligent education. The proposed Coral model significantly enhances the diagnostic accuracy of implicit learners' knowledge states. This improvement not only provides effective insights for online personalized tutoring services, such as question recommendations but also lays the foundation for further research in this area. Moreover, the automatic construction strategy for collaborative connections among learners

0.86

0.87

0.87

0.88

0.88

1400 1000 800

500

200

F1 score

𝒏

0.870

0.875

0.880

1

3

5

10

20

F1 score

𝒎

0.850

0.860

0.870

0.880

1&1

1000&10 800&10

500&20

200, 20

F1 score

𝒏 & 𝒎

offers valuable insights that can contribute to subsequent investigations in this field. Lastly, we anticipate that the proposed techniques can be extended to other domains, including but not limited to user interest modeling and social network modeling. Although our method is effective both theoretically and empirically, it suffers from computational inefficiencies. We have explored preliminary optimization strategies in the Appendix D.2 and will focus on improving computational efficiency in future research. In addition, future research plans to consider issues of fairness [51, 52] and explore the integration of large language models and multi-modal knowledge to enhance interpretability [27, 29, 30]. In essence, our work is dedicated to advancing intelligent education and deepening the understanding of human cognitive proficiency. It cannot cause negative effects. We anticipate its crucial role in fostering progress in both pertinent technologies and societal advancements.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly claim our task (scope), contributions and solutions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide all the proofs in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide our code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a github repository to publish our work (https://github.com/bigdataustc/Coral).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the experimental setup (Section 4.1), we provide all the setups.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: In our paper, we provide the average scores. Given the impending submission deadline, we will provide error bars during the rebuttal phase by updating the README file in the anonymous GitHub repository.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide these details in the implementation details (Section 4.1).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: These information is discussed in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers or website links about the dataset and open-source codes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Coral, a model that enhances cognitive diagnosis by integrating collaborative signals among learners with disentangled representation learning. Coral addresses the importance of leveraging collaborative connections among learners to better understand human learning processes. Traditional cognitive diagnosis methods focus on individual attributions and explicit practice records, but they often overlook the potential of collaborative signals. The proposed Coral model aims to fill this gap by integrating collaborative and disentangled cognitive states. The disentangled state encoder separates the cognitive states of learners and reconstructs their practice performance from an inner-learner perspective. The collaborative representation learning dynamically constructs a collaborative graph by iteratively searching for optimal neighbors in a context-aware manner, effectively capturing collaborative signals. The model is evaluated on three real-world datasets: ASSIST, Junyi, and NIPS2020EC. The results show that Coral outperforms existing methods in terms of ACC, AUC, F1-score, and RMSE. Coral also performs well in sparse and cold-start scenarios, demonstrating its robustness and adaptability.\n\nSTRENGTHS:\n1. Coral introduces an integration of collaborative signals with disentangled representation learning, addressing a gap in existing cognitive diagnosis methods.\n2. Experimental results demonstrate performance improvements over state-of-the-art methods across various datasets and scenarios.\n3. The disentangled representation provides better explainability and controllability of cognitive states, which is crucial for personalized education.\n\nWEAKNESSES:\n1. The approach in this paper is not very innovative, and the motivation is not very convincing.\n2. Although visualizing the embeddings provides some level of interpretability, it still does not yield any explicit conclusions or observable phenomena. Rather than visualizing the embeddings (t-SNE has a certain degree of randomness), I would prefer to see some interesting conclusions based on the dataset.\n3. The paper mentions computational inefficiencies, which could limit the model's applicability in large-scale or real-time settings.\n4. The model requires careful tuning of hyperparameters, which may not be straightforward for all datasets and might require extensive experimentation.\n5. The paper could benefit from a more extensive comparison with a broader range of alternative approaches to strengthen its contributions.\n\nQUESTIONS:\nPlease see the weaknesses\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper presents a Collaborative Cognitive Diagnosis model, named Coral, which incorporates a disentangled state encoder, collaborative graphs, and a state decoder. The goal of Coral is to model collaborative and disentangled cognitive states using representation learning techniques.

STRENGTHS:
1. The introduction of collaborative connections to cognitive diagnosis is well-motivated and logical.

2. The paper is well-structured, and the methodology is explained in detail.

3. The release of the code enhances the reproducibility of the research.

WEAKNESSES:
1. While the authors identify the challenge of obtaining explicit collaborative connections, they do not provide clear solutions to address this issue.

2. The heavy use of theoretical notations makes the paper less readable.

3. The detailed presentation of the methodology makes the experimental section appear crowded.

4. The performance improvements demonstrated by Coral are moderate.

QUESTIONS:
1. What is the performance impact if the concepts are not disentangled? Currently, there is insufficient motivation for integrating collaborative signals into the disentangled representations.

LIMITATIONS:
There is no significant negative societal impact identified in this work. The authors acknowledge an inefficiency issue within the model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the Coral model, which combines disentangled representation learning with collaborative signals to enhance cognitive diagnosis in intelligent education. It focuses on identifying implicit collaborative links among learners to improve understanding of their cognitive states. The model integrates a context-aware collaborative graph learning mechanism to simulate the collaborative relationships between learners based on their cognitive states. By incorporating collaborative information into the disentangled cognitive states of learners, the model achieves collaborative disentanglement. Experiments on real datasets demonstrate Coral's performance compared to existing methods.\n\nSTRENGTHS:\n1. This paper will undoubtedly attract attention and excitement from researchers in various fields, e.g., personalized learning, cognitive diagnosis, knowledge tracing, and performance prediction. It represents an exploration of modeling user collaboration in educational settings. While the modeling of collaborative relationships among users is well-established in other fields (e.g., social networks, recommendations), the technological development in online education has lagged. The research line on modeling learner relationships is almost non-existent. I hope this work will advance the intelligent education community.\n2. Compared to previous cognitive diagnosis research, Coral introduces a new method that combines disentangled representation learning with collaborative signals, holding great potential for enhancing cognitive diagnosis.\n3. Theoretical Contribution: The authors have meticulously derived the optimization form of the lower bound of cognitive representation encoders, the approximate optimization objective of collaborative relationship modeling, and the model convergence. The theoretical proofs are reasonable and solid.\n4. The experimental results on real datasets in normal, data-sparse, and cold-start scenarios indicate that Coral outperforms existing methods, demonstrating its effectiveness in improving diagnostic accuracy.\n5. The writing is good. The proposed problem and model are described clearly.\n\nWEAKNESSES:\nI expect that this paper will comfortably clear the bar for acceptance. However, I believe two main issues should be addressed first. Due to these reasons, I have set my score at a relatively borderline level but anticipate increasing it after the authors' rebuttal.\n\n1. If two students have fewer interaction records, leading to increased similarity in their interaction patterns, they might exhibit different cognitive levels. Using the method outlined in the paper to learn their representations may yield similar results, but this might not accurately reflect the true situation.\n2. The paper employs GNN to learn network representations, but the graph-based baseline comparison only includes RCD and does not include some recent graph-based cognitive diagnosis methods, such as [1].\n\nOther/Minor issues (Optional):\n\n1. In the second paragraph, the authors describe an example: experienced teachers can summarize students' similarities. Similarly, could large language models be used instead of real human teachers to construct collaborative graphs? I look forward to the authors discussing the integration with LLMs. Although this does not affect the readability and acceptance of the paper, I believe readers will be very interested.\n2. In Table 6, \"w/ full-kit (m,m)\" should be \"w/ full-kit (n,m).\"\n3. In Table 1, why are there two results for DCD on the NIPS2020EC dataset? It seems like results from two different runs.\n\n```\n[1] Wang, Shanshan, et al. \"Self-supervised graph learning for long-tailed cognitive diagnosis.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.\n```\n\nQUESTIONS:\n1. The motivation of this paper is to consider collaborative information among learners. What are the specific differences between the collaborative information proposed in this paper and the collaborative information in recommendation systems?\n2. Do the ASSIST and NIPS2020EC datasets have similarities and prerequisite relationships among the knowledge concepts that support RCD?\n3. In the example of question recommendation in Appendix E, are these five questions answered by learners in comparable situations (i.e., without acquiring new knowledge during the answering process)?\n\nLIMITATIONS:\nThe authors discuss the limitations of the computational efficiency of the paper. Meanwhile, they also give 2 simple yet effective solutions to address current limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Coral, a Collaborative Cognitive Diagnosis model with Disentangled Representation Learning, aimed at improving our understanding of human learning by leveraging collaborative signals among learners. By disentangling cognitive states and dynamically constructing a collaborative graph, Coral captures implicit connections between learners, leading to more accurate cognitive diagnoses. The model's innovative approach involves an iterative process of encoding, collaborative representation learning, and decoding, which significantly enhances its performance over existing methods. Extensive experiments demonstrate Coral's superior effectiveness in real-world datasets, making it a promising tool for intelligent education systems.\n\nSTRENGTHS:\n1. Coral’s use of disentangled representation learning and dynamic graph construction is a novel approach in the field. This allows the model to capture implicit collaborative signals among learners, leading to more accurate and insightful cognitive diagnoses.\n\n2. The extensive experiments and superior performance across multiple real-world datasets highlight Coral’s practical utility.\nBy disentangling cognitive states and integrating collaborative information, Coral offers greater explainability and controllability in cognitive diagnosis. This may make it easier for educators and researchers to understand and interpret the learning patterns and cognitive states of students.\n\nWEAKNESSES:\n1. Coral's effectiveness relies heavily on the quality and quantity of learner data available. In scenarios where such data is sparse or noisy, the model's performance may not be as robust.\n\n2. While the model shows great promise, its computational requirements for dynamically constructing collaborative graphs and learning representations might pose scalability issues, especially in very large educational environments.\n\n3. There may lack some highly related baselines/papers for the cognitive diagnosis. \n\n[1] https://arxiv.org/pdf/2002.00276\n[2] https://ieeexplore.ieee.org/document/10027634\n[3] https://arxiv.org/pdf/2005.13107\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Coral, a novel model for collaborative cognitive diagnosis with disentangled representation learning, that can help in developing intelligent education systems. The key contributions are incorporating collaborative information among learners to improve cognitive state diagnosis by using disentangled representation learning to model cognitive states, and then using context-aware graph construction method to identify implicit collaborative connections. Coral demonstrates superior performance over state-of-the-art methods across several real-world datasets, showing particular strength in sparse and cold-start scenarios. The approach offers improved interpretability and adaptability in modeling learners' cognitive states, potentially advancing personalized tutoring and understanding of human learning processes.\n\nSTRENGTHS:\n* The paper introduces an intriguing dual-perspective approach to cognitive diagnosis. By examining both inner-learner and inter-learner aspects, it offers a more comprehensive view of cognitive states. The integration of context-aware collaborative graph learning with disentangled representation learning is particularly noteworthy, as it potentially allows for more nuanced insights into learner interactions.\n\n* The empirical validation using three diverse datasets (ASSIST, Junyi, NIPS2020EC) lends credibility to the model's effectiveness. Its superior performance across various metrics compared to existing models is encouraging. Moreover, the model's resilience in scenarios with sparse data and cold start environments, suggests practical applicability in real-world educational settings.\n\n* From a structural standpoint, the paper presents its ideas coherently. The authors have taken care to provide detailed explanations and mathematical formulations for each component of their model. The inclusion of key equations and visual aids, such as t-SNE plots, effectively illustrates the model's inner workings, particularly in terms of disentanglement and neighbor selection processes.\n\n* This research makes a valuable contribution to the field of cognitive diagnosis. By enhancing both the accuracy and interpretability of learner cognitive state assessments, it opens up new avenues for personalized education. The model's broad applicability across various educational datasets, coupled with its novel methodologies like context-aware graph learning and co-disentanglement, provides a solid foundation for future research in educational data mining and related fields.\n\nWEAKNESSES:\n* The model suffers from computational inefficiencies, which can limit its practical applicability, especially in large-scale educational settings. This issue is transparently acknowledged by the authors too, which is positive. \n\n* While the paper aims to achieve disentangled representations, the evaluation of disentanglement quality remains simple and limited. It will be valuable to add more ablation studies related to disentanglement quality and run qualitative analysis of how individual latent dimensions correspond to interpretable knowledge concepts across different datasets.\n\n* The paper doesn't adequately address potential limitations of the collaborative approach, such as negative transfer or performance in edge cases. It will be valuable to discuss scenarios where the collaborative approach might not be beneficial or could lead to negative transfer. E.g., a highly advanced learner who struggles in one specific area may be grouped with generally low-performing learners\n\nQUESTIONS:\n* How sensitive is the model performance to changes in key hyperparameters such as the number of neighbors (K) and the beta value? Adding a sensitivity analysis for these hyperparameters in the appendix can help assess the model's robustness and guide practitioners in tuning it for different datasets.\n\n* Can you provide a breakdown of Coral's performance as the percentage of questions answered by new learners varies (e.g., 0%, 5%, 10%, 20%)? This would give a clearer picture of the model's efficacy in true cold-start scenarios.\n\n* Is it possible to show how disentangled factors correspond to specific knowledge areas (since mapping to interpretable cognitive factors are not easy) across different datasets? This would strengthen the claim of improved interpretability over existing methods.\n\nLIMITATIONS:\n* The authors briefly mention computational inefficiencies as a limitation in Section 5. They also provide some preliminary optimization strategies in Appendix F.2, which is positive. However, it would be valuable to also expand it to other concerns like any potential bias in the modeling process, limitations in the datasets used or generalizability to other educational contexts, any limitations in the disentanglement approach to capture all relevant cognitive factors, etc\n\n* Also the authors do not fully address potential ethical concerns and mitigation strategies related to privacy and fairness in cognitive modeling. E.g., privacy concerns related to modeling learner cognitive states, potential bias in the model that could disadvantage certain groups of learners, etc", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel model called Coral, aimed at enhancing cognitive diagnosis in educational contexts by leveraging collaborative signals among learners. The authors argue that learners with similar cognitive states often exhibit comparable problem-solving performances, and thus, understanding these collaborative connections can significantly improve the diagnosis of individual cognitive states.\nKey Contributions - \n1. Disentangled Representation Learning: Coral introduces a disentangled state encoder to effectively separate the cognitive states of learners. This approach allows for clearer interpretations of individual knowledge proficiencies by disentangling the various cognitive factors that influence learning performance.\n2. Collaborative Graph Construction: The model incorporates a context-aware collaborative representation learning mechanism that dynamically constructs a graph of learners. This graph is built by identifying optimal neighbors based on their cognitive states, facilitating the extraction of collaborative information.\n3. Co-disentanglement Process: Coral achieves co-disentanglement by aligning the initial cognitive states with collaborative states through a decoding process. This integration enhances the model's ability to reconstruct practice performance while providing a more nuanced understanding of learners' cognitive states.\n4. Empirical Validation: The paper reports extensive experiments demonstrating that Coral outperforms state-of-the-art methods across several real-world datasets, highlighting its effectiveness in accurately diagnosing cognitive states through collaborative learning.\nOverall, this work addresses significant gaps in existing cognitive diagnosis methods by simultaneously modeling both individual and collaborative learning information, paving the way for more personalized educational interventions.\n\nSTRENGTHS:\nOriginality\nThe paper introduces a unique approach to cognitive diagnosis by integrating collaborative signals among learners, which has not been extensively explored in existing literature. The concept of using a disentangled representation to model cognitive states while simultaneously capturing collaborative learning dynamics is innovative. Prior models either focused solely on individual cognitive factors or did not adequately account for collaborative interactions. I think it reflects a creative synthesis of ideas from domains of educational psychology and collaborative filtering.\n\nQuality\nThe authors provide a comprehensive framework that includes a disentangled state encoder, a context-aware collaborative graph learning mechanism, and a co-disentanglement process. The empirical validation through extensive experiments on real-world datasets demonstrates the robustness of the model. The results indicate significant improvements over state-of-the-art methods, showcasing the effectiveness of the proposed approach. The paper also includes detailed mathematical formulations and algorithms, which enhance the reproducibility of the research.\n\nClarity\nThe paper is well-structured, with a logical progression from the introduction of the problem to the presentation of the model and its components. The use of clear diagrams and figures aids in the understanding of complex concepts, such as the collaborative graph construction and the disentanglement process. The performance comparison table is extremely effective in showcasing how the contribution is significantly better than the state-of-the-art methods on multiple datasets.\n\nSignificance\nThe work definitely shows potential impact on the field of intelligent education. By improving the accuracy of cognitive diagnosis, the Coral model can facilitate more personalized learning experiences. The integration of collaborative learning dynamics addresses a critical gap in existing models and could influence the design of educational technologies and interventions, making this research valuable not only academically but also practically in educational settings.\n\nWEAKNESSES:\n1. Limited Generalizability of Experiments\nWhile the paper demonstrates the effectiveness of the Coral model through extensive experiments on several real-world datasets, the choice of datasets may limit the generalizability of the findings. The experiments primarily focus on specific educational contexts, which may not capture the diverse learning environments and learner behaviors present in broader educational settings.\nActionable Insight: Future work should consider testing the model on a wider variety of datasets, including those from different educational domains (e.g., K-12, higher education, online learning platforms) and varying learner demographics. This would help validate the robustness of the model and its applicability across different contexts.\n2. Lack of Comparison with Baseline Models\nThe paper claims significant improvements over state-of-the-art methods, but it does not provide a comprehensive comparison with a more wider range of baseline models. Especially collaborative filtering methods and other machine learning approaches that could serve as alternatives.\nActionable Insight: The authors should include a more diverse set of baseline models in their experiments to provide a clearer picture of Coral's performance. Additionally, discussing the strengths and weaknesses of these models in comparison to Coral would enhance the reader's understanding of the model's contributions.\n3. Insufficient Exploration of Hyper-parameter Sensitivity\nThe model's performance could be sensitive to various hyper-parameters, such as the number of neighbors in the collaborative graph or the dimensions of the disentangled representations. However, the paper does not provide a thorough analysis of how these hyper-parameters impact the model's performance.\nActionable Insight: Conducting a sensitivity analysis on key hyper-parameters would help identify optimal settings and provide insights into the model's robustness. This could involve systematic experiments varying one parameter at a time while keeping others constant, followed by a discussion of the results.\n4. Explainability and Interpretability\nWhile the paper emphasizes the importance of disentangled representations for explainability, it lacks a detailed exploration of how the model's outputs can be interpreted in practical educational contexts.\nActionable Insight: The authors should include case studies or examples demonstrating how the model’s outputs can be interpreted and applied in real educational settings. This could involve visualizing the relationships between cognitive states and learner performance or providing guidelines for educators on how to utilize the model's insights.\n5. Potential Overfitting Concerns\nThe complexity of the Coral model, with its multiple components and parameters, raises concerns about potential overfitting, especially if the training datasets are not sufficiently large or diverse. The paper does not address how the model's complexity is managed or validated against overfitting.\nActionable Insight: Providing validation metrics would strengthen the paper's claims regarding the model's generalization capabilities. \n6. Limited Discussion on Ethical Considerations\nThe paper does not address potential ethical implications of using collaborative cognitive diagnosis models in educational settings, such as data privacy concerns or the impact on learner autonomy and agency.\nActionable Insight: Including a section on ethical considerations would enhance the paper's comprehensiveness and relevance. This could involve discussing how to ensure data privacy, the implications of algorithmic bias, and strategies for maintaining learner agency in personalized educational interventions.\n\nQUESTIONS:\nDataset Diversity: What criteria did you use to select the datasets for your experiments? How do you ensure that the datasets adequately represent the diversity of learning environments and learner behaviors?\n\nBaseline Comparisons: Can you provide more details on the baseline models you compared Coral against? Specifically, how do these models differ in their approach to cognitive diagnosis, and why were they chosen as benchmarks?\n\nHyper-parameter Sensitivity: Did you conduct a sensitivity analysis on the hyper-parameters of the Coral model? If so, what were the findings, and how do hyper-parameter choices impact the model's performance?\n\nInterpretability of Results: How can educators and practitioners interpret the outputs of Coral in practical settings? Are there specific examples or case studies that illustrate how the model's insights can be applied in educational contexts?\n\nOverfitting Concerns: What measures did you take to prevent overfitting in your model? Can you provide validation metrics or results from cross-validation to support the generalization capabilities of Coral? In your \"data_loader.py\" script you have provided a check for validation set on line 60 (\"val_set.json\") and also in the paper on line 279 under the Experimental setup --> Settings section you claim \"split all the datasets with a 7:1:2 ratio into training sets, validation sets, and testing sets\". However, no validation metrics or validation sets seem to be used.\n\nEthical Considerations: Have you considered the ethical implications of using collaborative cognitive diagnosis models in educational settings? How do you address potential concerns regarding data privacy and algorithmic bias?\n\nLIMITATIONS:\nThe authors have not addressed the limitations and potential negative societal impact of their work, but there are still areas that could be improved. They could further elaborate on the generalizability of their findings beyond the specific educational contexts covered in the experiments. The authors do not explicitly discuss the potential negative societal impact of their work. This is an important aspect that should be addressed, as cognitive diagnosis models can have significant implications for learners, educators, and educational institutions. An in depth explanation should be provided pertinent to limitations on dimensions like : Data Privacy, Algorithmic Bias, Misuse of Diagnostic Information etc.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}