Title: DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for  Cognitive Diagnosis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Cognitive Diagnosis

Intelligent Education

Disentanglement Learning

Robustness

Graph Learning
Abstract: Existing graph learning-based cognitive diagnosis (CD) methods have made relatively good results, but their student, exercise, and concept representations are learned and exchanged in an implicit unified graph, which makes the interaction-agnostic exercise and concept representations be learned poorly, failing to provide high robustness against noise in students' interactions. Besides,  lower-order exercise latent representations obtained in shallow layers are not well explored when learning the student representation. 
To tackle the issues, this paper suggests a meta multigraph-assisted disentangled graph learning framework for CD (DisenGCD), which learns three types of representations on three disentangled graphs: student-exercise-concept interaction,  exercise-concept relation, and concept dependency graphs, respectively. 
Specifically,  the latter two graphs are first disentangled from the interaction graph. 
Then, the student representation is learned from the interaction graph by a devised meta multigraph learning module; multiple learnable propagation paths in this module  enable current student latent representation to access  lower-order exercise latent representations,
which can lead to  more effective nad robust student representations learned; 
the exercise and concept representations are learned on the relation and dependency graphs by graph attention modules. 
Finally, a novel diagnostic function is devised to handle three disentangled representations for prediction.  Experiments show better performance and robustness of DisenGCD than state-of-the-art CD methods and demonstrate the effectiveness of the disentangled learning framework and meta multigraph module.The source code is available at https://github.com/BIMK/Intelligent-Education/tree/main/DisenGCD.

DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for Cognitive Diagnosis

Abstract

Existing graph learning-based cognitive diagnosis (CD) methods have made relatively good results, but their student, exercise, and concept representations are learned and exchanged in an implicit unified graph, which makes the interactionagnostic exercise and concept representations be learned poorly, failing to provide high robustness against noise in students' interactions. Besides, lower-order exercise latent representations obtained in shallow layers are not well explored when learning the student representation. To tackle the issues, this paper suggests a meta multigraph-assisted disentangled graph learning framework for CD (DisenGCD), which learns three types of representations on three disentangled graphs: student-exercise-concept interaction, exercise-concept relation, and concept dependency graphs, respectively. Specifically, the latter two graphs are first disentangled from the interaction graph. Then, the student representation is learned from the interaction graph by a devised meta multigraph learning module; multiple learnable propagation paths in this module enable current student latent representation to access lower-order exercise latent representations, which can lead to more effective nad robust student representations learned; the exercise and concept representations are learned on the relation and dependency graphs by graph attention modules. Finally, a novel diagnostic function is devised to handle three disentangled representations for prediction. Experiments show better performance and robustness of DisenGCD than state-of-the-art CD methods and demonstrate the effectiveness of the disentangled learning framework and meta multigraph module. The source code is available at https: //github.com/BIMK/Intelligent-Education/tree/main/DisenGCD .

1 Introduction

In the realm of intelligent education [43, 27, 47], cognitive diagnosis (CD) plays a crucial role in estimating students' mastery/proficiency on each knowledge concept [1], which mainly models the exercising process of students by predicting students' responses based on their response records/logs

A

B

C

D

E

A

e

4

e

2

e

3

e

1

e

5

A

B

C

D

E

concepts

Cognitive

Diagnosis

and the exercise-concept relations. As shown in Figure 1, student Bob completed five exercises { e 1 , e 2 , e 3 , e 4 , e 5 } and got corresponding responses, and his diagnosis result can be obtained through CDbased on his records and the plotted relations between exercises and concepts. With students' diagnosis results, many intelligent education tasks can be benefited, such as exercise assembly [42], course recommendation [34, 45], student testing [18], and targeted training [2], and remedial instruction [11].

In recent years, many cognitive diagnosis models (CDMs) have been suggested based on various techniques, generally classified into two types. The first type of CDMs is based on the theory of educational psychology, and the representatives include DINA [4], (IRT) [13], and MIRT [20]. Considering that the simple diagnostic functions of this type of models fail to model complex student interactions, recently, artificial intelligence researchers have tried to employ neural networks (NNs) to enhance and invent novel CDMs. Therefore, the second type of CDMs can be further divided into two sub-types: focusing on inventing novel diagnostic functions and enhancing the input representations of CDMs, respectively. The representatives of the first sub-type of CDMs include the well-known NCD [25], KSCD [15], and so on [44, 16, 12, 36, 38]. For the second sub-type of CDMs, many NN techniques are employed, including hierarchical attention NNs for ECD [48], long short-term memory networks for DIRT [3], graph neural networks (GNNs) for RCD [7], SCD [28], and GraphCDM [21].

Among existing CDMs, GNN-based ones exhibit significantly better performance than others, which is attributed to the rich information propagation and exchange brought by GNNs. These CDMs generally learn the student, exercise, and concept representations implicitly or explicitly on a unified graph [7, 28], which makes three types of representations be exchanged and aggregated fully even if they are distant regarding relations, thus generating better representations and providing better performance. However, learning exercise and concept representations should be student-interactionagnostic [31], while the above CDMs' learning for exercise and concept representations is easily affected by students' interactions, and the representations will be learned poorly especially when there exists noise in students' interactions. In short, existing GNN-based CDMs fail to provide high robustness against student interaction noise. In addition, these CDMs do not explore the use of lower-order exercise latent representations in shallow layers for learning the student representation due to the intrinsic defect of their GNNs. The learned student representations could be more robust and not sensitive to student interaction data change.

Therefore, we propose a meta multigraph-assisted disentangled graph learning framework for CD (DisenGCD) to learn robust representations against interaction noise. The contributions include

The disentangled graph learning framework DisenGCD learns three types of representations on three disentangled graphs. Specifically, the student representation is learned on the student-exerciseconcept interaction graph; the exercise and concept representations are learned on two disentangled graphs: the exercise-concept relation graph and the concept dependency graph. By doing so, the latter two learned representations are interaction-agnostic and robust against student interaction noise.

To make the best of lower-order exercise latent representations for learning the robust student representation, a modified meta multigraph module containing multiple learnable propagation paths is used, where propagation paths enable current student latent representation to access and use lower-order exercise latent representations. The exercise and concept representations are learned through common graph attention networks (GAT) on relation and dependency graphs, respectively. Finally, a novel diagnostic function is devised to handle three learned representations for prediction.

(3) Extensive experiments show the superiority of the proposed DisenGCD to state-of-the-art (SOTA) models regarding performance and robustness, and the effectiveness of the disentangled graph learning framework and the devised meta multigraph module is validated.

2 Related Work

Related Work on Cognitive Diagnosis . The above has given the classification of CDMs, and we will briefly introduce typical CDMs, especially GNN-based ones. As representatives in educational psychology, IRT [13] (MIRT [20]) utilizes single (multiple) variable(s) to denote student's ability with logistic function for prediction. For NN-based representatives, NCD [25] and KSCD [15] fed student, exercise, and concept vectors to an IRT-like NN as the diagnostic function for prediction; NAS-GCD [44] is similar to NCD, but its diagnostic function is automatically obtained.

For GNN-based CDMs, their focus is on obtaining enhanced representations. For example, RCD [7] learns the student representation along the relation of adjacent exercise nodes, learns the exercise representation along the relation of adjacent student and concept nodes, and learns the concept representation along the relation of adjacent exercise and concept nodes. Since each node's information can be propagated to any node, learning regarding three relations can be seen as learning on an implicit unified graph; Similarly, SCD [28] takes the same aggregation manner along the first two relations of RCD for learning student and exercise representations for contrastive learning to mitigate long-tail problems. Despite the success of GNN-based CDMs, their learning manners of exercise and concept representations are not robust against student interaction noise, because the representations are learned together with student interactions in a unified graph, where the noise will prevent the representations from being learned well.

Related Work on Meta Graph/Multigraph . As can be seen, existing GNN-based CDMs intuitively adopt classical GNNs (GAT [24] or GCN[9]) to learn three types of representations. These CDMs update the student representation by only using the exercise latent representation in the previous layer yet not exploring the use of lower-order exercise latent representations.

Compared to traditional GNNs, meta graph-based GNNs can make the target type of nodes access lower-order latent representations of their adjacent nodes. The meta graph is a directed acyclic graph built for a GNN, each node stores each layer's output (latent representations) of the GNN, and each edge between two nodes could be one of multiple types of propagation paths. The representatives include DiffMG [5] and GEMS [8]. The meta multigraph is the same as the meta graph, but its each edge could hold more than one type of propagation path, where the representative is PMMM [10]. To make the best of low-order exercise latent representations for learning effective and robust student representations, this paper adopts the idea of meta multigraph and devises a modified meta multigraph learning module for the updating of the student interaction graph.

Related Work on Disentangling Graph Representation Learning . Learning potential representations of disentangling in complex graphs to achieve model robustness and interpretability has been a hot topic in recent years. Researchers have put forward many disentanglement approaches (e.g., DisenGCN [19], DisenHAN [32], DGCF [29], DCCF [35], DcRec [33], etc.) to address this challenge. For example, in DisenHAN [32], the authors utilized disentangled representation learning to account for the influence of each factor in an item. They achieved this by mapping the representation into different spatial dimensions and aggregating item information from various edge types within the graph neural network to extract features from different aspects; DcRec [33] disentangles the network into a user-item domain and a user-user social domain, generating two views through data augmentation and ultimately obtaining a more robust representation via contrastive learning.

Despite many approaches suggested, they were primarily applied to bipartite graphs to learn different representations from different perspectives, for learning more comprehensive representations. While this paper aims to leverage disentanglement learning to mitigate the influence of the interaction noise in the interaction graph, and thus we proposed a meta multigraph-assisted disentangled graph cognitive diagnostic framework to learn three types of representations on three disentangled graphs. By doing so, the influence of the noise on exercise and concept learning can be well alleviated.

3 Problem Formulation

For easy understanding, two tables are created to describe all notations utilized in this paper including notations for disentangled graphs and notations for the meta multigraph, summarized in Table 4 and Table 5. Due to the page limit, the two tables are included in Appendix A.1.

3.1 Disentangled Graph

This paper only employs the student-exercise-concept interaction graph G I for students' representation learning, and disentangles two graphs from G I for the remaining two types of representation learning, which are the exercise-concept relation graph G R , and the concept dependency graph G D .

Student-Exercise-Concept Interaction Graph. With students' response records R , exerciseconcept relation matrix, and concept dependency matrix, the interaction graph G I can be represented as G I = {V , E} . The node set V = S ∪ E ∪ C is the union of the student set S , exercise set E , and concept set C , while the edge set E = R se ∪ R ec ∪ R cc contains three types of relations: rse ij ∈ R se represents the student s i ∈ S answered exercise e j ∈ E , rec jk ∈ R ec denotes the exercise e j contains concept c k ∈ C , and rcc km ∈ R cc denotes concept c k relies on concept c m .

Disentangled Relation Graph. To avoid the impact of students' interactions R se on exercise representation learning, the exercise-concept relation graph G R is disentangled from G I , denoted as G R = G I / { S, R se } = { E ∪ C, R ec ∪ R cc } .

Disentangled Dependency Graph. Similarly, a concept dependency graph G D is further disentangled from G I to learn the concept representation without the influence of interactions R se and exercises' relation R ec , which is represented by G D = G R / { E, R ec } = { C, R cc } .

3.2 Problem Statement

For the cognitive diagnosis task in an intelligent education online platform, there are usually three sets of items: a set of N students S = { s 1 , s 2 , . . . , s N } , a set of M exercises E = { e 1 , e 2 , . . . , e M } , and a set of K knowledge concepts (concept for short) C = { c 1 , c 2 , . . . , c K } . Besides, there commonly exists two matrices: exercise-concept relation matrix Q = ( Q jk ∈ { 0 , 1 } ) M × K (Qmatrix) and concept dependency matrix D = ( Q km ∈ { 0 , 1 } ) K × K , to show the relationship of exercises to concepts and concepts to concepts, respectively. Q jk = 1 denotes the concept c k is not included in the exercise e j and Q jk = 0 otherwise; similarly, D km = 1 denotes concept c k relies on concept c m and D km = 0 otherwise. All students' exercising reponse logs are denoted by R = { ( s i , e j , r ij ) | s i ∈ S, e j ∈ E,r ij ∈ { 0 , 1 }} , where r ij refers to the response/answer of student s i on exercise e j . r ij = 1 means the answer is correct and r ij = 0 otherwise.

The CD based on disentangled graphs is defined as follows: Given : students' response logs R and three disentangled graphs: interaction graph G I , relation graph G R , and dependency graph G D ; Goal : revealing students' proficiency on concepts by predicting students' responses through NNs.

4 Method

For better understanding, Figure 2 presents the overall architecture of the proposed DisenGCD, where three learning modules are used for learning three types of representations on three disentangled graphs and the diagnostic function makes the final prediction based on the learned representations. Specifically, these three learning modules are: (1) a meta multigraph-based student learning module, (2) a GAT-based exercise learning module, and (3) a GAT-based concept learning module. In each learning module, the corresponding graph's embedding is randomly first initialized. Then, the meta multigraph-based student learning module is employed to learn the student representation S i based on interaction graph G I ; the GAT-based exercise learning module is used to learn the exercise representation E j based on relation graph G R ; while the concept representation C k is learned by the GAT-based concept learning module on dependency graph G D or the naive embedding if G D is unavailable. Finally, a devised novel diagnostic function receives three learned representations S i , E j , and C k to get the prediction ˆ r ij of student s i on exercise e j .

Note that the first module employs a modified meta multigraph aggregator to learn student representations. Compared to traditional graph learning, the module contains multiple learnable propagation paths, which enable the student latent representation to be learned currently to access and use lowerorder exercise latent representations, leading to a more effective and robust student representation.

s

i

e

j

c

k

×

WS

×

WE

×

WC

Disentangled

Framework

Diagnostic

Function

Interaction Graph

𝒔𝒔 �

𝒊𝒊

s

i

GAT

c

k

Dependency graph

Student Learning Module

GAT-based Exercise

GAT-based Concept

𝒆𝒆 �

𝒋𝒋

e

j

Relation graph

GAT

Disentangle

…

Candidate  propagation

paths  in  AP

uv

AP 1P

AP 12

ℎ𝑟𝑟

𝑎𝑎

𝑤𝑤𝑤𝑤𝑤𝑤

>

𝜏𝜏

H (1)

H (2)

H (p)

…

AP 1p

H (v)

H (u)

H (P)

Aek

Ake

Ase

Aes

Akk

I

zero

Disentangle

𝒢𝒢

ℐ

𝒢𝒢

𝒟𝒟

𝒢𝒢ℛ

Meta-multigraph

Aggregator

�

𝒄𝒄

𝐤𝐤

4.1 The Meta Multigraph-based Student Learning Module

Based on interaction graph G I , this module is responsible for: learning the optimal meta multigraph structure in G I , i.e., optimal multiple propagation paths, and updating student nodes' representations based on the learned meta multigraph structure to get the representation S i ∈ R 1 × d .

To start with, all nodes (including students, exercises, and concepts) need to be mapped into a d -dimensional hidden space. The student embedding s i ∈ R 1 × d for student s i , exercise embedding e I j ∈ R 1 × d for exercise e j , and concept embedding c I k ∈ R 1 × d for concept c k can be obtained by

x S i ∈ { 0 , 1 } 1 × N , x E j ∈ { 0 , 1 } 1 × M , and x C k ∈ { 0 , 1 } 1 × K are three one-hot vectors for student s i , exercise e j , concept c k , respecitvely. W I S , W I E , and W I C are three learnable parameter matrices.

4.1.1 The Meta Multigraph Aggregator

To learn more useful representations from the above initial embedding, we construct a meta multigraph MG = { H , AP} to specify a set of propagation paths for all nodes' aggregation in graph learning.

As shown in Figure 2, the meta multigraph is actually a directed acyclic graph containing P hypernodes: H = { H (1) , . . . , H ( p ) , . . . , H ( P ) } . H ( p ) = { s p ∈ R N × d , e p ∈ R M × d , c p ∈ R K × d } is a set of latent representations stored in p -th hyper-node for all students, exercises, and concepts, whose i -th row s p i ∈ R 1 × d , j -th row e p j ∈ R 1 × d , and k -th row c p k ∈ R 1 × d are the p -th latent representations of student s i , exercise e j , and concept c k , respectively. Especially when p =1, s p i , e p j , and c p k equal initial embedding s i , e I j , and c I k .

Each edge AP uv ∈ AP between two hyper-nodes H ( u ) and H ( v ) contains multiple types ( | HR | ) of propagation paths, which can be denoted as AP uv = { ( hr a ∈ HR , hr wei a ∈ [0 , 1]) | 1 ≤ a ≤ | HR |} . Here hr a is a -th type of propagation path in candidate path set HR , and hr wei a represents the weight of hr a . This paper adopts | HR | =7 types of propagation paths as the candidate paths for HR : 1) the students to exercises path A se , 2) the exercises to students path A es , 3) the exercises to concepts path A ek , 4) the concepts to exercises path A ke , 5) the concepts to concepts path A kk , 6) the identity path I , 7) the zero path zero .

Here, path A se means updating the exercise latent representation along the propagation path of student nodes to exercise nodes, and other paths hold similar meanings. As a result, HR = { A se , A es , A ek , A ke , A kk , I, zero } , and the edge set AP in MG can be denoted by AP = { AP uv | 1 ≤ u < v ≤ P } , containing | HR | ∗ [ P ∗ ( P -1) / 2] propagation paths. With above propagation paths, H ( p ) can be updated by

where f ( · ) is the aggregation function of GCN. f ( AP up , H ( u ) ) refers to that obtaining the latent representations ( s p i , e p j , c p k ) in hyper-node H ( p ) based on all latent representations stored in previous hyper-nodes (i.e., H (1) to H ( p -1) ). It can be seen that the updating process of H ( p ) replies on latent representations in multiple hyper-nodes and multiple propagation paths, which compose the modified meta multigraph aggregator together.

For better understanding, here is an example: when p =3, AP 13 = { zero } and AP 23 = { A es , I } , the updating of three latent representations s 3 i , e 3 j , and c 3 k is denoted as

The first equation is to update three latent representations according to AP 13 , while the second is to get the updating based on AP 23 . The final latent representations are obtained by summing up both updated ones. Here Up ( · ) and Mess ( · ) are common update and message-passing functions.

4.1.2 Routing Strategy

To find suitable propagation paths, threshold τ ( u,v ) is created for each pair of hyper-nodes ( u, v ) :

Softmax ( AP uv )) normalizes weights of each type of propagation path regarding their hr wei a values. By doing so, the propagation paths of each pair of hyper-nodes will remain if their hr wei a values are greater than the corresponding threshold. Thus the updating process of H ( p ) can be rewritten as

Finally, the learned student representation s P in H P are used for the diagnosis, i.e, s P i is used as S i .

4.2 The GAT-based Exercise Learning Module and GAT-based Concept Learning Module

GAT-based Exercise Learning Module . This module is responsible for learning the exercise representation E j ∈ R 1 × d on the relation graph G R via a L -layer GAT network [24]. Firstly, the embedding of exercises and concepts in G R is obtained in the manner same as Eq.(1) through two learnable matrices W R E ∈ R M × d and W R C ∈ R K × d , i.e, e R j = x E j × W R E , c R k = x C k × W R C .

Afterward, the GAT neural network is applied to aggregate neighbor information to learn the exercise representation. The aggregation process of l -th layer ( 1 ≤ l ≤ L ) can be represented as

The first equation is to aggregate the information of exercise e j 's neighbors N e j to get its l -th layer's latent representation e R ( l ) j ∈ R 1 × d , while the second is to update the l -th layer's concept latent representation c R ( l ) k ∈ R 1 × d from its exercise neighbors N ec c k and concept neighbors N cc c k . α R ( l ) j ,

α R ( l ) k , and α R ( l ) ˆ k are the l -th layer's attention matrices for exercise e j 's concept neighbors, concept c k 's exercise neighbors, and c k 's concept neighbors. They can be obtained in the same manner, and the k -th row of α R ( l ) j can be obtained by

[ · ] is the concatenation and F ec ( · ) is a fully connected (FC) layer mapping 2 ∗ d vectors to scalars.

The latent representation e R (0) j and c R (0) k refer to e R j and c R k , and the L -th layer output e R ( L ) j is used as E j . By disentangling the interaction data, the learned exercise representation E j will be more robust against interaction noise.

GAT-based Concept Learning Module . Similarly, this module obtains the concept representation C k ∈ R 1 × d by applying a L -layer GAT network to the dependency graph G D . After obtaining the initial embedding c D k = x C k × W D C by a learnable matrix W D C ∈ R K × d , this module updates the l -th layer's latent representation c D ( l ) k by c D ( l ) k = ∑ m ∈ N cc c k α D ( l ) ˆ k ( m ) c D ( l -1) m + c D ( l -1) k . α D ( l ) ˆ k denotes the attention matrix, which can be computed as same as Eq.(7).

Here c D (0) k is equal to c D k , and the L -th layer output c D ( L ) k is used as C k . If graph G D is unavailable, this module will directly take the initial embedding c D k as C k . By further disentangling, the learned representation C k may be robust against noise in student interactions to some extent.

4.3 The Diagnosis Module

To effectively handle the obtained three types of representations, a novel diagnostic function is proposed to predict the response ˆ r ij of student s i got on exercise e j as follows:

where F si ( · ) , F ej ( · ) , and F simi ( · ) are three FC layers mapping a d -dimensional vector to another one, and Q k is a binary vector in the k -th row of Q-matrix.

Here h si ∈ R 1 × d can be seen as the student's mastery of each knowledge concept; while the obtaining of h ej ∈ R 1 × d aims to get the exercise difficulty of each concept; h simi is to measure the similarity between h si and h ej via a dot-product followed by an FC layer and a Sigmoid function σ ( · ) [41], where a higher similarity value in each bit represents a higher mastery on each concept, further indicating a higher probability of answering the related exercises; the last equation follows the idea of NCD to compute the overall mastery averaged over all concepts contained in exercise e j .

We can see that the proposed diagnostic function has as high interpretability as NCD, IRT, and MIRT.

Model Optimization. With the above modules, the proposed DisenGCD are trained by solving the following bilevel optimization problem through Adam [39]:

where L val ( · ) and L train ( · ) denote the loss on validation dataset D val and training dataset D train . ω denotes all model parameters, and α denotes the weights of learnable propagation paths AP in meta multigraph MG . The cross-entropy loss [40] is used for L val and L train .

5 Experiments

This section answers the following questions: RQ1 : How about the performance of DisenGCD compared to SOTA CDMs? RQ2 : How about the DisenGCD's robustness against noise and the disentangled learning framework's effectiveness in DisenGCD? RQ3 : How about the effectiveness of the devised meta multigraph learning module? RQ4 : How does the learned meta multigraph on target datasets look like, and how about their generalization on other datasets? In addition, more experiments to validate the proposed DisenGCD's effectiveness are in the Appendix.

5.1 Experimental Settings

Datasets. To verify the DisenGCD's effectiveness, we conducted experiments on two public datasets ASSISTments [6] and SLP [14], and one private dataset Math, whose statistics are in Table 1. Note that ASSISTments and Math [16] were used for most experiments to answer RQ1 to RQ3 , and SLP was used to answer RQ4 . More details of these datasets are presented in Appendix A.2 .

Comparison CDMs and Metrics. To verify the effectiveness of DisenGCD, five SOTA CDMs were compared, including traditional CDMs DINA and MIRT, NN-based CDMs NCD and ECD, and GNN-based RCD. Here SCD is not compared due to the failed run of its provided source code Three metrics were adopted to measure the performance of all CDMs [37], including area under the cure (AUC) [23], accuracy (ACC) [22], and root mean square error (RMSE) [46].

Parameter Settings. For the DisenGCD model, its dimension d was set to the number of concepts, P in the meta multigraph, the number of layers L in GAT, and λ were set to 5, 2, and 0.8. For its training, the learning rate and batch size were 1e-4 and 256. For comprehensive comparisons [17], four splitting ratios (40%/10%/50%, 50%/10%/40%, 60%/10%/30%, and 70%/10%/20%) were adopted to get training, validation, and testing datasets. All compared CDMs followed the settings in their original papers, and all experiments were executed on an NVIDIA RTX4090 GPU.

5.2 Overall Performance Comparison (RQ1)

To address RQ1 , the DisenGCD was compared with DINA, MIRT, NCD, ECD, and RCD on ASSISTments and Math datasets. Table 2 summarizes their performance in terms of AUC, ACC, and RMSE obtained under four dataset-splitting settings, where the best result of each column on one dataset was highlighted in bold. Besides, Table 6 in the Appendix compares DisenGCD with three SOTA CDMs, including SCD [28], KSCD [15], and KaNCD [26].

We can observe from the results in both tables that, DisenGCD holds better performance than all compared CDMs. Besides, we can also obtain two observations from Table 2: Firstly, GNN-based CDMs (RCD and DisenGCD) hold significantly better than NN-based CDMs (NCD and ECD), indicating the importance of learning representations through graphs, and DisenGCD outperforming RCD validates the superiority of DisenGCD's graph learning manner. Secondly, as the ratio changes,

DisenGCD

RCD

DisenGCD

RCD

1%

10%

30%

50%

1%

10%

30%

50%

Interaction Nosie Ratio

AUC

ACC

Interaction Nosie Ratio

(a)

Math

ASSISTments

DisenGCD(naï ve)

DisenGCD(mp)

DisenGCD(mg)

DisenGCD

ACC

AUC

ASSISTments

Math

(b)

0.76

0.80

0.75

0.78

0.74

0.76

0.73

0.74

0.72

0.71

0.72

0.70

0.70

the performance change of DisenGCD and RCD is much smaller than NCD and ECD, which signifies DisenGCD and RCD are more robust to different-sparsity data. To validate its sparsity superiority, more experiments are summarized in Appendix B.1 .

5.3 Effectiveness of Disentangled Learning Framework of DisenGCD (RQ2)

To investigate DisenGCD's robustness against interaction noise, we conducted robust experiments on the ASSISTments dataset under the ratio of 60%/10%/30%, where a certain amount of noise interactions were added to each student in the training and validation datasets. Figure 3a presents the ACC and AUC of DisenGCD and RCD under noise data of different percentages. As the noise data increases, the performance leading of DisenGCD over RCD becomes more significant, which reaches maximal when noise data of 50% was added. That demonstrates DisenGCD is more robust to student noise interactions than RCD, attributed to the disentangled learning framework of DisenGCD. Similar observations can be drawn from more experiments on other two datasets in Appendix B.2 .

To further analyze the framework effectiveness, four variants of DisenGCD were created: DisenGCD(I) refers to learning three representations only on the interaction graph G I ; DisenGCD(Is+Rec) refers to learning student representation on G I , but learning exercise and concept ones on the relation graph G R ; DisenGCD(Ise+Rc) refers to learning student and exercise representations on G I but learning concept one on G R ; while DisenGCD(Isc+Re) learns student and concept representations on G I but learns exercise one on G R . Table 3 compares the performance of RCD, DisenGCD, and its four variants on ASSISTments. As can be seen, the comparison between DisenGCD(I) and other variants indicates learning three representations in two disentangled graphs is more effective than in one unified graph, especially with textitDisenGCD(Is+Rec); the comparison between DisenGCD(Is+Rec) and DisenGCD indicates learning three representations in three disentangled graphs is more effective, further validating the above conclusion. Finally, we can conclude that the proposed disentangled learning framework is effective in enhancing DisenGCD's performance and robustness.

5.4 Effectiveness of Meta Multigraph Learning Module in DisenGCD (RQ3)

In Table 3, both DisenGCD(I) and RCD learn three types of representations in one unified graph, but DisenGCD(I) utilizes the meta multigraph aggregator. Therefore, the comparison between DisenGCD(I) and RCD proves the devised meta multigraph learning module is effective in improving the model performance to some extent.

To further validate this, we created three variants of DisenGCD: DisenGCD(naive) refers to the meta multigraph module replaced by the naive embedding (i.e., S i equal to s i in Eq.(1)); Dis-

H (1)

H (2)

H (4)

H (5)

Aek

I

Ase

Aes

Ake

H (3)

H (1)

I

Aes

Aek

Ake

I

I

I

Aes

I

Ake

Ake

Ake

Aes

Aes

Aes

H (2)

H (3)

H (4)

H (5)

H (1)

(a)

RMSE

0.41

0.42

0.43

0.44

0.45

65%

70%

75%

80%

85%

RCD

A1

A2

A3

M1

M2

M3

0.46

AUC

AUC

RMSE

(b)

enGCD(mp) refers to predefining the meta multigraph module's propagation paths according to HAN [30]; DisenGCD(mg) refers to this module's meta multigraph replaced by meta graph. Figure 3b presents the AUC and ACC values of DisenGDCN and its variants on two datasets under the ratio of 60%/10%/30%. As can be seen, the devised meta multigraph learning module enables DisenGDCN to hold significantly better performance than the naive embedding-based variant. The comparison results between DisenGCD and DisenGCD(mg) as well as DisenGCD(mp) validates the effectiveness of using meta multigraph and the automatically learned propagation paths (i.e., learned meta multigraph). Thus, the effectiveness of the devised meta multigraph module can be validated.

5.5 Visualization and Effectiveness of Learned Meta Multigraph (RQ4)

The comparison between DisenGCD(mp) and DisenGCD has revealed the effectiveness of the learned meta multigraph on the target dataset. Therefore, an intuitive doubt naturally emerged: Is the learned meta multigraph still effective on other datasets? Before solving this, Figure 4a gives the structure visualization of two learned meta multigraphs. We can see two learned meta multigraphs hold two distinct structures. That may be because two datasets are a bit different regarding the relations between exercises and concepts, where exercises in Math contain only one concept while exercises in ASSISTments may contain more than one concept. Thus, another doubt naturally emerged: Is the learned meta multigraph on the target dataset ineffective on a different type of dataset?

To solve the doubts, we applied six learned meta multigraphs (A1, A2, A3, M1, M2, and M3) to the SLP. A1-A3 and M1-M3 were learned by DisenGCD three times on ASSISTments and Math. SLP dataset is similar to Math, whose exercises only contain one concept. Figure 4b summarizes the results of RCD and six DisenGCD variants that utilize six given meta multigraphs. As can be seen, DisenGCD's performance under M1-M3 is promising and better than RCD, while DisenGCD under A1-A3 performed poorly. The observation can answer the above doubts to some extent: meta multigraphs learned by DisenGCD may be effective when the datasets to be applied are similar to target datasets.

In addition to the above four experiments, more experiments were executed to validate the devised diagnostic functions, analyze the DisenGCD's parameter sensitivity, analyze its execution efficiency, and investigate the effectiveness of the employed GAT modules in Appendix B.3 , Appendix B.4 , Appendix B.5 , and Appendix B.6 .

6 Conclusion

This paper proposed a meta multigraph-assisted disentangled graph learning framework for CD, called DisenGCD. The proposed DisenGCD learned student, exercise, and concept representations on three disentangled graphs, respectively. It devised a meta multigraph module to learn student representation and employed two common GAT modules to learn exercise and concept representations. Compared to SOTA CDMs on three datasets, the proposed DisenGCD exhibited highly better performance and showed high robustness against interaction noise.

References

Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. Engaging with massive online courses. In Proceedings of the 23rd international conference on World wide web , pages 687-698, 2014.

Joseph Beck. Difficulties in inferring student knowledge from observations (and why you should care). In Proceedings of the 13th International Conference of Artificial Intelligence in Education , pages 21-30, 2007.

Song Cheng, Qi Liu, Enhong Chen, Zai Huang, Zhenya Huang, Yiying Chen, Haiping Ma, and Guoping Hu. Dirt: Deep learning enhanced item response theory for cognitive diagnosis. In Proceedings of the 28th ACM international conference on information and knowledge management , pages 2397-2400, 2019.

Jimmy De La Torre. Dina model and parameter estimation: A didactic. Journal of educational and behavioral statistics , 34(1):115-130, 2009.

Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. Diffmg: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 279-288, 2021.

Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-adapted Interaction , 19(3):243-266, 2009.

Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages 501-510, 2021.

Zhenyu Han, Fengli Xu, Jinghan Shi, Yu Shang, Haorui Ma, Pan Hui, and Yong Li. Genetic metastructure search for recommendation on heterogeneous information network. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 455-464, 2020.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '20, page 639-648, New York, NY, USA, 2020. Association for Computing Machinery.

Chao Li, Hao Xu, and Kun He. Differentiable meta multigraph search with partial message propagation on heterogeneous information networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8518-8526, 2023.

Qi Liu, Shiwei Tong, Chuanren Liu, Hongke Zhao, Enhong Chen, Haiping Ma, and Shijin Wang. Exploiting cognitive structure for adaptive learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 627-635, 2019.

Sannyuya Liu, Qing Li, Xiaoxuan Shen, Jianwen Sun, and Zongkai Yang. Automated discovery of symbolic laws governing skill acquisition from naturally occurring data. Nature Computational Science , pages 1-12, 2024.

Frederique M Lord. Applications of Item Response Theory to Practical Testing Problems . LAWRENCE ERLBAUM ASSCCIAATES, 1980.

Yu Lu, Yang Pian, Ziding Shen, Penghe Chen, and Xiaoqing Li. SLP: A multi-dimensional and consecutive dataset from K-12 education. In Proceedings of the 29th International Conference on Computers in Education Conference , volume 1, pages 261-266. Asia-Pacific Society for Computers in Education, 2021.

Haiping Ma, Manwei Li, Le Wu, Haifeng Zhang, Yunbo Cao, Xingyi Zhang, and Xuemin Zhao. Knowledge-sensed cognitive diagnosis for intelligent education platforms. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 1451-1460, 2022.

Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang. Enhancing cognitive diagnosis using un-interacted exercises: A collaboration-aware mixed sampling approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8877-8885, 2024.

Haiping Ma, Yong Yang, Chuan Qin, Xiaoshan Yu, Shangshang Yang, Xingyi Zhang, and Hengshu Zhu. Hd-kt: Advancing robust knowledge tracing via anomalous learning interaction detection. In Proceedings of the ACM on Web Conference 2024 , pages 4479-4488, 2024.

Haiping Ma, Yi Zeng, Shangshang Yang, Chuan Qin, Xingyi Zhang, and Limiao Zhang. A novel computerized adaptive testing framework with decoupled learning selector. Complex & Intelligent Systems , pages 1-12, 2023.

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International Conference on Machine Learning , 2019.

Mark D. Reckase. Multidimensional Item Response Theory . Springer New York, 2009.

Yu Su, Zeyu Cheng, Jinze Wu, Yanmin Dong, Zhenya Huang, Le Wu, Enhong Chen, Shijin Wang, and Fei Xie. Graph-based cognitive diagnosis for intelligent tutoring systems. KnowledgeBased Systems , 253:109547, 2022.

Jianwen Sun, Fenghua Yu, Sannyuya Liu, Yawei Luo, Ruxia Liang, and Xiaoxuan Shen. Adversarial bootstrapped question representation learning for knowledge tracing. In Proceedings of the 31st ACM International Conference on Multimedia , pages 8016-8025, 2023.

Jianwen Sun, Fenghua Yu, Qian Wan, Qing Li, Sannyuya Liu, and Xiaoxuan Shen. Interpretable knowledge tracing with multiscale state representation. In Proceedings of the ACM on Web Conference 2024 , pages 3265-3276, 2024.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6153-6161, 2020.

Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuralcd: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering , 35(8):8312-8327, 2022.

Hao Wang, Tong Xu, Qi Liu, Defu Lian, Enhong Chen, Dongfang Du, Han Wu, and Wen Su. Mcne: An end-to-end framework for learning multiple conditional network representations of social network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1064-1072, 2019.

Shanshan Wang, Zhen Zeng, Xun Yang, and Xingyi Zhang. Self-supervised graph learning for long-tailed cognitive diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 110-118, 2023.

Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. Disentangled graph collaborative filtering. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , 2020.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The world wide web conference , pages 2022-2032, 2019.

Xinping Wang, Caidie Huang, Jinfang Cai, and Liangyu Chen. Using knowledge concept aggregation towards accurate cognitive diagnosis. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2010-2019, 2021.

Yifan Wang, Suyao Tang, Yuntong Lei, Weiping Song, Sheng Wang, and Ming Zhang. Disenhan: Disentangled heterogeneous graph attention network for recommendation. Proceedings of the 29th ACM International Conference on Information & Knowledge Management , 2020.

Jiahao Wu, Wenqi Fan, Jingfan Chen, Shengcai Liu, Qing Li, and Ke Tang. Disentangled contrastive learning for social recommendation. New York, NY, USA, 2022. Association for Computing Machinery.

Zhengyang Wu, Ming Li, Yong Tang, and Qingyu Liang. Exercise recommendation based on knowledge concept prediction. Knowledge-Based Systems , 210:106481, 2020.

Jiashu Zhao Dawei Yin Xubin Ren, Lianghao Xia and Chao Huang. Disentangled contrastive collaborative filtering. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2023.

Shangshang Yang, Haiping Ma, Ying Bi, Ye Tian, Limiao Zhang, Yaochu Jin, and Xingyi Zhang. An evolutionary multi-objective neural architecture search approach to advancing cognitive diagnosis in intelligent education. IEEE Transactions on Evolutionary Computation , 2024.

Shangshang Yang, Haiping Ma, Cheng Zhen, Ye Tian, Limiao Zhang, Yaochu Jin, and Xingyi Zhang. Designing novel cognitive diagnosis models via evolutionary multi-objective neural architecture search. arXiv preprint arXiv:2307.04429 , 2023.

Shangshang Yang, Linrui Qin, and Xiaoshan Yu. Endowing interpretability for neural cognitive diagnosis by efficient kolmogorov-arnold networks. arXiv preprint arXiv:2405.14399 , 2024.

Shangshang Yang, Xiangkun Sun, Ke Xu, Yuanchao Liu, Ye Tian, and Xingyi Zhang. Hybrid architecture-based evolutionary robust neural architecture search. IEEE Transactions on Emerging Topics in Computational Intelligence , 2024.

Shangshang Yang, Ye Tian, Cheng He, Xingyi Zhang, Kay Chen Tan, and Yaochu Jin. A gradient-guided evolutionary approach to training deep neural networks. IEEE Transactions on Neural Networks and Learning Systems , 33(9):4861-4875, 2021.

Shangshang Yang, Ye Tian, Xiaoshu Xiang, Shichen Peng, and Xingyi Zhang. Accelerating evolutionary neural architecture search via multifidelity evaluation. IEEE Transactions on Cognitive and Developmental Systems , 14(4):1778-1792, 2022.

Shangshang Yang, Haoyu Wei, Haiping Ma, Ye Tian, Xingyi Zhang, Yunbo Cao, and Yaochu Jin. Cognitive diagnosis-based personalized exercise group assembly via a multi-objective evolutionary algorithm. IEEE Transactions on Emerging Topics in Computational Intelligence , 2023.

Shangshang Yang, Xiaoshan Yu, Ye Tian, Xueming Yan, Haiping Ma, and Xingyi Zhang. Evolutionary neural architecture search for transformer in knowledge tracing. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Shangshang Yang, Cheng Zhen, Ye Tian, Haiping Ma, Yuanchao Liu, Panpan Zhang, and Xingyi Zhang. Evolutionary multi-objective neural architecture search for generalized cognitive diagnosis models. In 2023 5th International Conference on Data-driven Optimization of Complex Systems (DOCS) , pages 1-10. IEEE, 2023.

Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. Dataset regeneration for sequential recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3954-3965, 2024.

Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and llm performance. arXiv preprint arXiv:2407.06645 , 2024.

Xiaoshan Yu, Chuan Qin, Dazhong Shen, Shangshang Yang, Haiping Ma, Hengshu Zhu, and Xingyi Zhang. Rigl: A unified reciprocal approach for tracing the independent and group learning processes. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4047-4058, 2024.

Yuqiang Zhou, Qi Liu, Jinze Wu, Fei Wang, Zhenya Huang, Wei Tong, Hui Xiong, Enhong Chen, and Jianhui Ma. Modeling context-aware features for cognitive diagnosis in student learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2420-2428, 2021.

Appendix A More details about Proposed Method and Datasets

A.1 Notations Summary

For convenient reading and understanding, we have summarized all the notations discussed in the paper in the following two tables. Table 4 summarizes all the notations in the three disentangled graphs, while Table 5 describes the notations of the meta multigraph.

A.2 Statistics of Datasets

We evaluated our method on three real-world datasets: ASSISTments, Math and SLP, which both provide student-exercise interaction records and the exercise-knowledge concept relational matrix.

ASSISTments is a public dataset collected by the assistant online tutoring systems in the 2009-2010 acadaemic year.

Math is a private dataset collected by a well-known online learning platform that contains math practice records and test records for elementary and middle school students.

SLP is another publicdata set that collects data on learners' performance in eight different subjects over three years of study, including maths, English, physics, chemistry, biology, history and geography.

Notably, in both the Math dataset and the SLP dataset, they provide relationships between concepts. For these three datasets, we filtered students with fewer than 15 answer records to ensure there was enough data for the learning of the model. We compared our model with five previous diagnostic models, including two classic models based on educational psychology, DINA [4] and MIRT [20], two neural network-based models, NCD [25] and ECD [48], and a graph-based diagnostic model, RCD [7].

DINA [4] is a classical CDM, which uses a binary variable to characterize whether and whether a student has mastered a specific concept.

MIRT [20] is an extension of the irt model, which uses multidimensional vectors to characterize students' abilities and the difficulty of exercises, and uses linear functions to model the interactions.

NCD [25] is one of the recent CDMs based on deep learning, which uses neural networks to model higher-order student-exercise complex interactions.

ECD [48] incorporates the impact of the educational environment in students' historical answer records into the diagnostic model to achieve diagnostic enhancement. Here due to the lack of educational background, we use random initialization vectors to characterize cognitive states.

RCD [7] is one of the most advanced models, which introduces the relationship between concepts into cognitive diagnosis and models the relationship through a graph structure.

Appendix B Additional Experiments

B.1 Experiments on Dataset with Different Levels of Sparsity

As shown in Section 5.2, the proposed DisenGCD shows better performance than RCD even when the splitting ratios change, which indicates the better robustness of DisenGCD to different-sparsity data.

To further verify whether the proposed DisenGCD is robust against sparse data (i.e., missing data or sparse interaction patterns), we conducted corresponding experiments on ASSISTments and Math datasets under the splitting setting of 60%/ 10% / 30%. In the experiments, for each dataset, we randomly deleted 5%, 10%, and 20% of the students' answer records (i.e., the interaction data) in the training sets, respectively, thus providing three variant training sets; the proposed DisenGCD

ACC

RMSE

AUC

DisenGCD

RCD

ACC

RMSE

AUC

Math

SLP

0.3

0.4

0.5

0.6

0.7

0.40

0.60

0.45

0.50

0.55

0.4

0.5

0.6

0.7

0.55

0.75

0.70

0.65

0.60

0.42

0.44

0.46

0.48

0.50

0.55

0.60

0.65

0.70

0.75

0.80

1

10

30

50

1

10

30

50

1

10

30

50

1

10

30

50

1

10

30

50

1

10

30

50

was compared with NCD and RCD, and their overall performance comparisons were summarized in Table B.1.

As can be seen, under which deleting ratio, the proposed DisenGCD exhibits the best performance, and its performance does not drop very significantly when the deleting ratio increases. Even when deleting 20% of students' interaction data, DisenGCD still achieves an AUC value of 0.7812, which is greatly higher than NCD and RCD. It demonstrates that the proposed DisenGCD is effective on sparse interaction data, which is mainly attributed to the devised meta-multigraph learning module. That module enables the DisenGCD to access and use lower-order exercise latent representations, thus providing more accurate and robust students' representations, especially when partial data is lacking.

In summary, the proposed DisenGCD is effective and more robust against sparse interaction data, which benefits from the devised meta-multigraph learning module, and thus the effectiveness of the meta-multigtaph learning module can be indirectly demonstrated.

B.2 More Experiments for Robustness Validation

In Section 5.3, we have validated the robustness of the proposed DisenGCD against interaction noise on the ASSISTments dataset. To further show its robustness superiority, we executed the same experiments on other two datasets, i.e., Math and SLP, under the same settings, where a certain ratio of noise interactions were added to each student in the training and validation datasets. Figure 5

presents the results of DisenGCD and RCD obtained under different proportions of noise: the orange polyline denotes DsienGCD's results and the blue polyline denotes RCD's results.

As can be seen, the proposed DisenGCD always performs better than RCD in both datasets under different noise ratios. Besides, as the noise ratio increases, the performance leading of the proposed DisenGCD to RCD becomes more significant. That further demonstrates the proposed DisenGCD is more robust against student interaction noise and thus validates the effectiveness of the proposed disentangled graph learning framework.

B.3 Validation of the Devised Diagnostic Function

In this paper, in addition to the disentangled learning framework and the devised meta multigraph module, we also designed a novel diagnostic function to adopt them. To validate its effectiveness, we compared it with the diagnostic functions of MIRT, NCD, and RCD on the Math dataset, where four splitting settings were considered. Table 8 summarizes their overall performance regarding AUC, ACC, and RMSE.

We can get the following two observations. Firstly, the comparisons of MIRT and DisenGCD-MIRT, NCD and DisenGCD-NCD, as well as RCD and DisenGCD-RCD show that the representations learned by the proposed DisenGCD are effective. Secondly, the performance superiority of DisenGCD to other variants indicates the devised diagnostic function is effective. To sum up, the effectiveness of the devised diagnostic function is validated.

B.4 Hyperarameter Sensitivity Analysis

For the proposed DisenGCD, there is an important hyperparameter, i.e., the number of hyper-nodes in the meta multigraph. To investigate its influence on theDisenGCD, the hyper-node number was set to 4, 5, 6, and 7, respectively, and we executed the experiments on the ASSISTments and SLP datasets. Figures 6 and 7 plot the results regarding AUC and RMSE.

As can be seen in both two datasets, the DisenGCD can achieve optimal results in terms of RMSE and AUC, when the number of hyper-nodes is set to 5. When the number of hyper-nodes is less than 5, the meta multigraph contains too few propagation paths, adversely affecting the learning of student representations. This results in suboptimal performance compared to the scenario where the number of hyper-nodes is 5. Conversely, when the number of hyper-nodes is greater than 5, there is an abundance of propagation paths, making the aggregated information overly complex and hindering effective student representation learning. Besides, more hyper-nodes cause more computational complexity. Therefore, this paper set the number of hyper-nodes to 5 for the proposed disenGCD.

Furthermore, we aim to explore the influence of the number of hyper-nodes on the robustness of the proposed DisenGCD. Therefore, we also executed the experiments on Math datasets under different ratios of noise interaction(1%,30%,50%). Figures 8 plot the results regarding AUC and ACC. As can be seen, when the number of hyper nodes is equal to 5, the robustness of DisenGCD is the most promising, whose performance under different ratios of noises is balanced better than RCD.

0.758

0.766

0.762

0.770

0.422

0.423

0.424

0.425

0.426

6

8

18

12

14

16

10

20

6

8

18

12

14

16

10

20

AUC

RMSE

num=4

num=5

num=6

num=7

0.755

0.760

0.770

0.780

0.408

0.416

0.412

0.404

6

8

18

12

14

16

10

20

6

8

18

12

14

16

10

20

num=4

num=5

num=6

num=7

AUC

RMSE

B.5 Efficiency Analysis of DisenGCD

To investigate the computational efficiency of the proposed approach, we have compared it with RCD on ASSISTments and Math datasets in terms of model inference time and training time. Table 9 presents the overall comparison, where the time of the proposed DisenGCD under different hyperparameters P is also reported.

As can be seen, DisenGCD's inference time is better than RCD's. This indicates that the proposed DisenGCD is more efficient than RCD, further signifying that it is promising to extend DisenGCD to

B.6 Ablation Experiments on Graph Representation Learning

To verify the superiority of the employed GAT on the proposed DisenGCD, three GNNs (GCN, GraphSage, and HAN) are used to replace the GAT in the proposed DisenGCD, which are termed DisenGCD(GCN), DisenGCD(GraphSage), and DisenGCD(HAN), respectively. Then, we compared the three approaches with the proposed DisenGCD on the Math dataset. The comparison of them is presented in Table 10 As shown in Table 10, the GCN-based DisenGCD achieves the worst performance, followed by DisenGCD(HAN). The GraphSage-based DisenGCD holds a competitive performance to yet is still worse than the proposed DisenGCD (based on GAT).

The above results show the GAT is a suitable and optimal choice among these four GNNs for the proposed DisenGCD, but we think the GAT may not be the best choice because there exist other types of GNNs that can be integrated with DisenGCD, showing better performance.

In summary, it is reasonable and effective for the proposed DisenGCD to adopt the GAT to learn the representations.

Appendix C Limitation Discussion

There are still some limitations in the proposed DisenGCD.

High Computational Complexity and Poor Scalability. The complexity of DisenGCD consists of three parts: the aggregation of P -hypernode meta multigraph on N + M + K -node interaction graph (average A 1 neighbors), L-layer GAT aggregation on the M + K -node relation graph (average A 2 neighbors), and L-layer GAT aggregation on the K -node dependency graph (average A 3 neighbors). Suppose each node is d dimensional, L-layer GAT on K -node graph contains: computing nodes' attention from neighbors O ( L × K × A 3 × d ) and nodes' linear transformation O ( L × K × d 2 ) , totally equalling O ( L × K × d × ( A 3 + d )) ; meta multigraph on N + M + K -node graph contains: computing nodes' attention O ( P × ( N + M + K ) × A 1 × d ) , computing path's attention O ( P × ( N + M + K ) × d ) , and nodes' linear transformation O ( P × ( N + M + K ) × d 2 ) . As a result, DisenGCD's complexity equals O ( P ( N + M + K ) d ( A 1 +1+ d ))+ O ( L ( M + K ) × d ( A 2 + d ))+ O ( L × K × d ( A 3 + d )) . As analyzed above, the time complexity of our model may be high and not applicable in some large data sets.

Potentially Poor Task Transferability. The proposed method primarily targets cognitive diagnosis tasks and is designed to handle such graphs as the student-exercise-concept graph, which may not be readily applicable to other tasks. It is specifically tailored for modeling tasks similar to the student-exercise-concept triad diagram.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [TODO] In the abstract and introduction section, we describe the motivation of our research and the problems to be optimized

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [TODO] In the appendix, we discuss some of the shortcomings of our model and the directions for further optimization.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [TODO] not applicable

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [TODO] In the experimental chapter and the appendix chapter, we introduce the experimental environment and details in detail.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [TODO] We will include the required data set in the code that supplements the file

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [TODO] In the experimental chapter and the appendix chapter, we introduce the experimental environment and details in detail

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: [TODO] We did not analyze the error in the experimental results

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: [TODO] We describe our experimental environment in detail in the experimental chapter

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [TODO] Our experiment conforms to the experimental requirements of NIPS Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: [TODO] Our experiments help the development of AI in the field of education

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [TODO] not applicable

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [TODO] The dataset we use is free from the interference of related issues we introduce it in the experimental section

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

[TODO] not applicable

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO] not applicable

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO] not applicable

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies cognitive diagnosis based on graph learning. The proposed method utilizes multiple graphs to assist the representation learning. Specifically, the authors further disentangle another two graphs from the most comprehensive student-exercise-concept interaction graph by removing certain node and edge types. Therefore, three kinds of disentangled representations are learned based on the three graphs. Lastly, a diagnostic function fuses the three representations for final predictions.\n\nSTRENGTHS:\n1. The problem that the paper aims to solve is well formulated and it is of practical value. \n2. The authors give very detailed illustrations of the methodology part. \n3. Comprehensive experiments are conducted to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. Since the proposed method learns the graph representations based on disentangled graphs, the authors are encouraged to add a section about disentangled graph representation learning in the related work part to summarize the recent works on this problem and illustrate the relation of this work with previous ones. \n2. A more detailed introduction to incorporated datasets should be given. Specifically, both accuracy and RMSE are employed as evaluation metrics, the authors should provide a clearer illustration of the target tasks of those datasets. \n3. The authors should include more mode variants to demonstrate the effectiveness of the added graphs (e.g., w/o G_R, w/o G_D, and w/o both). \n4. GAT is used in this work to learn exercise and concept representations. The authors are encouraged to introduce the reason for this choice or conduct experiments based on other graph model backbones. \n5. There are many typos in the paper, e.g., line 16 and line 254. The authors should check the grammar carefully.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a meta multigraph assisted disentangled graph cognitive diagnosis model. Its main contribution is to propose a disentangled graph framework, which disentangles the student-exercise-concept dependency graph into an exercise-concept interaction graph and a concept-concept relationship graph to improve the robustness of representation modeling. Additionally, the concept of meta multigraph is introduced in the student-exercise-concept dependency graph to assist in better modeling of student representation. DisenGCD has demonstrated good performance compared to previous models in multiple datasets, and has shown good robustness in the presence of noise interference or sparse datasets. Finally, the authors also discussed some limitations of DisenGCD in terms of computational complexity and task transferability.\n\nSTRENGTHS:\n1.\tThe paper is well-organized and clearly written, making it accessible to readers. Besides, the authors also present mathematical formulations, detailed explanations of the proposed models.\n2.\tThe authors focus on improving the robustness of the CD model and provide a detailed analysis of some problems found in previous graph cognitive diagnosis works. \n3.\tThe experiments are very thorough. The authors demonstrate the robustness of the model through decoupling experiments, sparse dataset experiments, and noisy dataset experiments\n\nWEAKNESSES:\n1. Although DisenGCD has demonstrated excellent performance across multiple datasets, it still appears to have the drawback of high computational complexity, which may limit its application in practical scenarios.\n2. Although the model has shown improved performance, its interpretability may be lacking. In real-world applications within the education field, transparent and interpretable models are more likely to gain acceptance and trust. However, the paper does not address methods for enhancing the interpretability of the model.\n3. The paper lacks detailed discussion on the challenges and potential issues that may arise when applying the model in practical educational settings. For instance, it does not address the strategies for handling data privacy and security concerns, which are critical considerations in the implementation of such models.\n\nQUESTIONS:\n1. KaNCD seems to have been found to work equally well in some papers. Can you compare DisenGCD and KaNCD?\n2. The paper utilizes graph attention networks (GAT) to learn representations of exercises and concepts. However, it would be valuable to explore and compare alternative graph neural network (GNN) models, such as graph convolutional networks (GCN) or LightGCN, to assess their performance in the same context.\n3. How are the multiple learnable propagation paths in the meta-multigraph learning module designed and selected? Have you tried different numbers or types of propagation paths and how well have they worked?\n\nLIMITATIONS:\n\\", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DisenGCD, a new framework for cognitive diagnosis in educational contexts. The authors make several contributions:\n1) They propose a disentangled graph learning approach, separating the typically unified graph into three distinct graphs: student-exercise-concept interactions, exercise-concept relations, and concept dependencies.\n2) They develop a meta multigraph module for learning student representations, which innovatively allows access to lower-order exercise representations.\n3) They employ GAT-based modules to learn exercise and concept representations on the disentangled graphs.\n4) They design a new diagnostic function to effectively combine the learned representations from the different graphs.\nThe authors evaluate DisenGCD on multiple datasets, demonstrating performance improvements and enhanced robustness compared to state-of-the-art methods in cognitive diagnosis.\n\nSTRENGTHS:\nThe paper presents several notable strengths:\n1) The proposed disentangled graph learning approach is a novel contribution to the field of cognitive diagnosis. By separating different types of information (student-exercise-concept interactions, exercise-concept relations, and concept dependencies) into distinct graphs, the authors have created a more robust model that appears less susceptible to noise in the data. This is a significant advancement over existing unified graph approaches.\n2) The meta multigraph module introduces an innovative mechanism for student representation learning. By allowing access to lower-order exercise representations, this module potentially captures more nuanced relationships in the data, which could be particularly valuable in educational contexts where student knowledge evolves over time.\n3) The empirical evaluation is comprehensive and well-executed. The authors conduct experiments on multiple datasets (ASSISTments, Math, and SLP), demonstrating consistent improvements in performance and robustness over state-of-the-art methods. This thorough evaluation strengthens the paper's claims and increases confidence in the generalizability of the proposed method.\n4) The paper includes detailed ablation studies and analyses that effectively validate the different components of DisenGCD. These studies provide valuable insights into the contribution of each component and help justify the design choices made by the authors.\n\nWEAKNESSES:\n1) While the disentangled graph approach is innovative, it may lose some valuable cross-entity interactions. The separation of student-exercise interactions from exercise-concept relations could potentially limit the model's ability to capture complex, multi-hop relationships that might exist in a unified graph structure.\n2) The meta multigraph module, although showing improvements, adds considerable complexity to the model. It's not clear if this added complexity is always justified, particularly for simpler datasets or scenarios with limited noise. A more thorough analysis of the trade-off between model complexity and performance gains would strengthen the paper.\n3) The paper lacks a comprehensive discussion on how DisenGCD handles cold-start problems, especially for new students or exercises that don't have established interactions in the graph structure. This is a common challenge in educational settings and needs to be discussed.\n\nQUESTIONS:\n1) How does DisenGCD adapt to dynamic changes in the knowledge structure, such as the introduction of new concepts or the discovery of new relationships between existing concepts? Does the disentangled structure present any challenges in updating the model under these scenarios? \n2) In Section 4.1, you describe the meta multigraph as containing P hyper-nodes, and in your experiments you set P to 5. How did you determine this value? Have you explored the impact of different values of P on the model's performance and computational efficiency?\n3) While your experiments demonstrate improved robustness to interaction noise, how does DisenGCD perform when noise is introduced in the concept dependency graph or exercise-concept relation graph? This could simulate errors in curriculum design or expert knowledge.\n4) Have you investigated whether the disentangled approach leads to any loss of performance in scenarios where complex, cross-entity relationships are crucial for accurate diagnosis? Are there cases where a unified graph approach might outperform DisenGCD?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents DisenGCD, a novel cognitive diagnosis model designed to enhance the robustness and accuracy of student, exercise, and concept representations by leveraging a meta multigraph-assisted disentangled graph learning framework. DisenGCD constructs and disentangles three specific graphs for interactions, relations, and dependencies, thus mitigating the negative impact of noise in students’ interactions. The framework incorporates a meta multigraph learning module and GAT to improve the learning of latent representations, demonstrating superior performance and robustness compared to state-of-the-art CDMs on multiple datasets.\n\nSTRENGTHS:\n1. The use of a meta multigraph-assisted disentangled graph learning framework is a novel approach that significantly enhances the robustness of cognitive diagnosis models.\n2. DisenGCD shows better performance in terms of AUC, accuracy, and RMSE compared to other state-of-the-art CDMs.\n3. The model effectively handles noise in student interactions, ensuring that the learning process remains accurate and reliable.\n4. The experiments cover multiple datasets and different data splits, providing a thorough validation of the model’s effectiveness.\n5. The diagnostic function of DisenGCD maintains high interpretability, comparable to traditional models like NCD, IRT, and MIRT.\n\nWEAKNESSES:\n1. The implementation of the meta multigraph learning module and the disentangled graph learning framework could be complex and resource-intensive.\n2. The paper does not report error bars or other statistical significance measures, which are important for validating experimental results. \n3. The comparison does not include some recent models like SCD, which could provide a more comprehensive evaluation.\n4. While some limitations are discussed, a more detailed analysis of potential weaknesses and future improvements could enhance the paper’s transparency and reliability.\n\nQUESTIONS:\n1. Can the model be adapted to handle real-time data for dynamic cognitive diagnosis?\n2. How does the choice of hyperparameters affect the model’s performance and robustness?\n3. Can the disentangled graph learning framework be integrated with other neural network architectures for further performance enhancement?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}