Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation
Authors: Zheng Zhang
Wei Song
Qi Liu
Qingyang Mao
Yiyan Wang
Weibo Gao
Zhenya Huang
Shijin Wang
Enhong Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Cognitive Diagnosis
Intelligent Education
Monotonicity Assumption
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Intelligent education stands as a prominent application of machine learning. Within this domain, cognitive diagnosis (CD) is a key research focus that aims to diagnose students' proficiency levels in specific knowledge concepts. As a crucial task within the field of education, cognitive diagnosis encompasses two fundamental requirements: accuracy and fairness. Existing studies have achieved significant success by primarily utilizing observed historical logs of student-exercise interactions. However, real-world scenarios often present a challenge, where a substantial number of students engage with a limited number of exercises. This data sparsity issue can lead to both inaccurate and unfair diagnoses. To this end, we introduce a monotonic data augmentation framework, CMCD, to tackle the data sparsity issue and thereby achieve accurate and fair CD results. Specifically, CMCD integrates the monotonicity assumption, a fundamental educational principle in CD, to establish two constraints for data augmentation. These constraints are general and can be applied to the majority of CD backbones. Furthermore, we provide theoretical analysis to guarantee the accuracy and convergence speed of CMCD. Finally, extensive experiments on real-world datasets showcase the efficacy of our framework in addressing the data sparsity issue with accurate and fair CD results.

Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation
Zheng Zhang 1.2
Wei Song 1.2
Li Qingyang Mao Yiyan Wang
Weibo Gao Zhenya Huang! Shijin Wang** Enhong Chen 1.2
1: University of Science and Technology China
2: State Key Laboratory of Cognitive Intelligence
3: Beijing Normal University
(zhangzheng, sww,, ma2,,maa00050, weibogao) veiiboga)]@maill.. @mail. ustc.edu.cm; {qiliuql, huangzhh, cheneh) Sestmeheeutuee @ustc. edw..edd.ucco cn;
wangyiy Soggyyyyn@mail.. bnu. edu. cn; cn;
s sjwang3eifytek. com
Abstract
Intelligent education stands prominent application of machine learning. Within this domain, cognitive diagnosis (CD) is key research focus that aims to diagnose students' proficiency levels in specific knowledge concepts. As crucial task within the field of education, cognitive diagnosis encompasses two fundamental requirements: aceuracy and fairness. Existing studies have achieved significant success by primarily utilizing observed historical logs of student-exercise interac- tions. However, real-world scenarios often present a challenge, where a substantial number of students engage with a limited number of exercises. This data sparsity issue can lead to both inaccurate and unfair diagnoses. To this end, we introduce monotonic data augmentation framework, CMCD, to tackle the data sparsity issue and thereby achieve accurate and fair CD results. Specifically, CMCD inte- grates the monotonicity assumption, fundamental educational principle in CD, to establish two constraints for data augmentation. These constraints are general and can be applied to the majority of CD backbones. Furthermore, we provide theoretical analysis to guarantee the accuracy and convergence speed of CMCD. Finally, extensive experiments on real-world datasets showcase the efficacy our framework in addressing the data sparsity issue with accurate and fair CD results.
Introduction
Intelligent education is a significant domain within machine learning that focuses on exploring students" learning patterns. In the past decades, research in this interdisciplinaty field has garnered substantial attention from scholars across various disciplines (29 26 [0]45), including education. machine learning, and psychology. Within intelligent education, cognitive diagnosis (CD) stands out as a crucial research area that aims to measure students' proficiency levels in specific knowledge domains, such as Geometry [39 13 21 For instance, illustrated in Figure students practice some exercises 61, 62, 63 and obtain associated responses indicating correctness, which are then utilized perform CD to infer the students' mastery levels of the corresponding concepts. With a comprehensive understanding of students' abilities, CD can be applied to various applications, including student assessment 56) and educational recommendation systems [19].
As a crucial task within the field of education, CD encompasses two fundamental requirements: I) Accurate Diagnoses: is imperative to precisely evaluate students' mastery of knowledge facilitate
*Correspondngg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Excreisc
Knowledge Concepts
Suc
Bob
Legend
A: Equation
RR
v
X
B
B: Statistics
C: A.solute Value
?
?
D: Coordinates
practice
Cognitive Diagnosis
Bob
E: Geometry
?
Diagnostic Report
Correct
X Wrong
No intoraction
Figure 1: An illustrative example of cognitive diagnosis.
various downstream applications. For instance, this accuracy enables teachers to deliver personalized instruction, tailored to the specific needs of each individual student. 2) Fair Diagnoses: CD plays pivotal role in high-stakes examinations like the GRE 300, significantly influencing individuals' developmental opportunities. Therefore, ensuring fairness in CD is of paramount importance. In this context, fairness means different student groups divided by sensitive attributes (e.g.. gender, race) should be treated fairly and equally.
To accomplish these dual objectives, prior research has made notable strides by harnessing historical logs that document students' interactions with exercises. However, in practical scenarios, many students can only interact with limited number of exercises within the vast exercise pool. leading to the data sparsity issue that may cause both inaccurate and unfair diagnoses. detailed analysis is available in Section 4 Currently, several approaches have been proposed mitigate this issue from a model-centric perspective, which focus on developing more complex architectures to address the challenges posed by data scarcity [14 77) However, the incorporation of additional architectures often compromises the model's ' interpretability. rendering it unsuitable for high-impact educational environments. For instance, in high-stakes exams like the GRE, only IRT (29), a classical CD model, has been applied due to their statistical superiority [30]. While other model-based models, despite mitigating data sparsity to some extent, remain unused because of their lack of interpretability.
In contrast to these model- based approaches. in this paper, we tackle the data sparsity issue from the perspective of data augmentation without altering the model architecture. However, have encountered the following challenges: f) Monotonicity Assumption. The Monotonicity Assumption is fundamental theoretical premise in the field of CD. Specifically, it posits that student's proficiency exhibits a monotonic relationship with the probability of providing a correct response to an exercise. Using Figure an example, Sue, who correctly responds to exercise e is considered to possess a higher proficiency level in the related concept C (i.e., Absolute Value) compared to Bob, who provides an incorrect response. This ensures the interpretability of CD, contributing its widespread acceptance and application. Therefore, maintaining the model's monotonicity assumption during the data augmentation process is crucial. 2) Theoretical Guarantees. As data augmentation emerges as a new paradigm in the CD domain, it becomes imperative to establish corresponding theoretical guarantees for this novel approach.
To address these challenges, we propose a monotonic data augmentation CD framework, CMCD, complemented by theoretical guaran- tees. Specifically, we integrate two data aug- mentation constraints confronting the mono- tonicity assumption. For each student, we generate fake students by reversing one of his responses while keeping other records un- changed to train CD models. As illustrated in Figure? Bob's wrong answer to é1 re- versed to generate C-1 while his right answer to es is reversed to generate C-2. Following the monotonicity assumption, we assume that Bob's proficiency the corresponding knowledge concept is lower than that of C-1 yet higher than that of C-2. Moreover, we provide theoretical analysis to guarantee CMCD's advantage accuracy and convergence speed. Finally, we conduct extensive experiments on real-world datasets, demonstrating the effectiveness of our method across various CD models. Our key contributions can be summarized as follows:
es
es
e
eq
C-1
X
V
V
Bob
X
x
C-2
x
x
Figure 2: An example of monotonic augmentation.
Problem Emphasis. Our paper strongly emphasizes the connection between data sparsity and inaccurate and unfair diagnoses. To the best of our knowledge, we are the first to highlight the relationship between data sparsity and educational faimess.

---

Framework Design. From the perspective of data augmentation, we combine the mono- tonicity assumption in CD propose a general framework, CMCD, which comes with theoretical guarantees of accuracy and convergence speed.
Experimental Evaluations. Extensive experiments on real-world datasets demonstrate the effectiveness of our approach.
Related Works
2.1
Cognitive Diagmosis
CD plays a crucial role in various real-world educational scenarios 48 49)
including student
assessment [36 and educational recommender systems 24 19]. Initially rooted in psychometries, CD models such as Item Response Theory (IRT) 22 have been widely used, particularly in the context of GRE 30 Later, Multidimensional Item Response Theory (MIRT) (35] was introduced to extend the single-trait features in IRT to multidimensions. While these models were effective and provided interpretable diagnostic results based on psychometric theories, they heavily relied on handcrafted interaction functions and could only leverage users' numerical response records, which were often affected by data sparsity (25]. With the advancements machine learning, researchers began developing CD models from machine-learning learning perspective. One notable model in this regard is NCDM [39]. which employed neural networks to Iearn the interactions between students and exercises, yielding satisfactory results. However, the majority of existing works have primarily focused on designing more complex architectures to alleviate data sparsity. In contrast to traditional model-based approaches, this paper embraces data-centric perspective and introduces CMCD, which comes with strong theoretical guarantees and adaptable for integration with all cognitive diagnosis models.
2.2 Data Augmentation
Data augmentation is a classical technique to address the problem of insufficient training data in the machine learning community 42]. Over the past few years, this concept has achieved significant success in the fields of neural language processing (NLP) I3 57 32 and computer vision (CV) Later, this idea was applied user modeling tasks, which necessitated the simulation of user-item interaction data. The work most similar to ours involves data augmentation of user-item interactions in recommendation systems. They typically answer questions such as: "What would happen if...?" Through these responses, they can generate additional virtual data, thereby alleviating the issue of data sparsity. For instance, Wang et al. 44 addressed the issue of data sparsity in the sequential recommendation and provided an answer to the question: "What would user like to buy if their previously purchased items had been different?". Similarly, Xiong et al. [44] focused on review-based recommendation and answered the question: "What would be the user's decision their feature-level preference had been different?" Although these works have shown promising results, when applied to cognitive diagnosis, they overlook the monotonicity assumption, classic theory in education. Additionally, they lack corresponding theoretical guarantees. In this paper, we aim to bridge these gaps to alleviate data sparsity in cognitive diagnosis more effectively.
38
6
3 Preliminaries
3.1 Cognitive Diagnosis and Models
In this subsection, we formally define the CD problem. Assume there are m students, n exercises, which are denoted as S = {ss m ). {e,} Assume each student s; has records on exercises indexed by Qi C {1, 2. ... n2.. the response logs R; of student s; a set of triplets (5 Eji Mr.g). where j E Qi Vi.j E {0, 1} is the score obtained by student s; on exercise ey. Given response logs R, of student s, the goal of CD is mine the proficiency 0,-
Cognitive Diagnosis Models (CDMs) are developed to depict student's proficiency level on specific knowledge concepts based on her responses to several test items. To do this, an objective function is used to train CDMs on the student performance prediction task. More concretely, CDMs are expected to minimize the difference of the predicted probability py (0.) of student s, giving the right response
3

---

to the exercise between the true response yt ()(()). In this paper. following the classical works we adopt the crosseentrppy loss, the goal is,
1 m m Σ (((:))
L
Σ
Σ JEQ,
Mi, log (0:) + +1 VV.,) log(1 (,()))
(I)
In the past decades, lots of CDMs have been proposed such as IRT 29 contain two parts: (1) the representations of trait features and (2) the interaction function. For example. IRT models each student as a proficiency variable 0, each exereise as discriminating factor a; and difficulty factor 3j. and logistic function used forecast the likelihood that student s; will answer exercise e, correctly based on logistic function Fl i.e., Py (0.) 1/(1 + (0,(0-
MIRT. Generally, CDMs
3.2 Fairness in Cognitive Diagnosis
With the advancement of machine learning technologies 20 91. such as large language models (551, which have found widespread applications in many important scenarios 522, trustworthy Al has become a very important topic [18
17 Among these concerns, the issue of fairness Given that cognitive diagnosis holds a fundamental position in the field of education and is extensively applied in high-stakes exams such as the GRE [30], which significantly shapes individuals' developmental opportunities, ensuring fairness in cognitive diagnosis becomes of utmost importance. this paper, we follow the widely accepted group fairness definition proposed by Li et al. [23]. which states that a fair model should provide the same level of utility performance for different user groups.
27
28
34
33
has garnered widespread attention 22
37
53
For analytical convenience, we focus on the case where the sensitive attribute is binary, which can be easily extended to multiple values. The student group be divided into two groups based on their sensitive attributes, denoted as Gp and Gi, where = Go GI, Go n G Ø. We denote the number of samples in each group as mg and mi. Inspired by previous works [54 50 23 in this paper, fairness in cognitive diagnosis is defined as follows, where a lower value indicates better fairness performance:
Definition 3.1 (Fairness in Cognitive Diagnosis).
1 Σ M(s:) GF
Σ M(si)
710
m
B.EGo
S.EG,
where M represents a metric for evaluating utility performance, such as MAE or MSE score, and M(s;) denotes the utility performance for student s;.
After introducing the fairness definition in cognitive diagnosis, our objective extends beyond accu- rately identifying students' proficiency, We should also strive to meet faimess requirements, with the goal of minimizing GF.
4 Data Sparsity Analysis
In this section, we explore data sparsity in cognitive diagnosis through the real-world dataset, Math (detailed information will be introduced in the Experiment). Firstly, we conduct an analysis of data sparsity within the dataset, partitioning it based on the number of responses per student. The statistical data illustrated in Figure a). From the figure, it is evident that the majority of users have low log count, highlighting the widespread presence of data sparsity. Following the methodology outlined in [46]. we classify records with fewer than 50 responses as the sparse group. Next, we validate whether data sparsity affects the accuracy and faimnsss of cognitive diagnosis.
Inaccurate Result From the perspective of accuracy, we conduct comparative analysis between the sparse and non-sparse groups, as illustrated in Figure bbb. The results indicate a notable decrease in performance for the sparse group across different backbones compared to the non-sparse group. This outcome strongly implies that data sparsity can result inaccurate diagnostic results.
Herr adopt two- woopprrmettr logistic IRT model.

---

=00
loy
1000
New 003140
1201
AD0
Areal
Amsr
4300
(a) Data Sparsity.
(b) Inaccurate Result.
(c) Unfair Result.
(d) Data Sparsity Disparity.
Figure 3: The impact of data sparsity on CD models
Unfair Result From the perspective of fairesss, examine the performance of the two groups divided by sensitive attributes. In the Math dataset, the student group is divided into Areal and Area2 by sensitive attribute region, with Areal representing the disadvantaged group. The results are depicted in Figure unveiling significant variations in performance outcomes among different groups and indicating the presence of unfairness. Following this, we delve into the disparity in sparsity levels between the two groups, as shown in Figure d). The results highlight significantly divergent sparsity levels, notably with the disadvantaged groups exhibiting higher sparsity. This discrepancy is attributed to the likelihood that disadvantaged groups may have fewer opportunities to engage in relevant online learning platforms compared to their advantaged counterparts. This statistical outcome directly implies that data sparsity has greater impact on the disadvantaged group (Areal). Based on this observation, we can conclusively state that data sparsity can lead unfair diagnostic results.
So far, we have validated that data sparsity in cognitive diagnosis can result in inaccurate and unfair diagnostic outcomes. Consequently, mitigating the issue of data sparsity in cognitive diagnosis is of paramount importance. In the following sections, we will demonstrate how address data sparsity from the perspective of data augmentation.
5 The Proposed Framework
In this section, propose monotonic data augmentation framework, CMCD, as depicted in Figure 4 It can address the issue of data sparsity and ensure accurate and fair diagnos- tic results. Firstly, present the constraints of our proposed monotonic data augmentation approach. Subsequently, offer theoretical as- surances regarding the effectiveness of CMCD.
Dyputhedk
0
Q2
du
5.1 Monotonic Data Augmentation
S1
Hyputheds
In CD, there is fundamental assumption called the monotonicity assumption [1], which plays a vital role in ensuring interpretability. In this paper, we introduce novel approach that integrates the monotonicity assumption with data aug- mentation. This combined technique allows us to generate more realistie student response data that conforms to educational principles. As a result, can effectively address the challenge of data sparsity. The monotonicity assumption is defined as follows:
Figure 4: The CMCD framework.
Monotonicity Assumption The probability of correct response to the exercise is monotonically increasing any dimension of the student's 's knowledge proficiency.
Building upon the monotonicity assumption, suppose two students 8; and 8y have answered exactly the same exercises, i.e., Q = Q - Q. we propose two specific hypotheses as follows:
Hypothesis 1: If answered exercise ej correctly, while answered éj incorrectly, and answers for other exercises are the same, then the proficiency level of 8; should not be lower than 8y. Formally,
for only one Q, ri.j 1, Tr 0, while for Q, k * Ta k Tv.k, we have 0₃ 2 Og.
5

---

Hypothesis 2: If 8; answered exercise ej incorrectly, while sy answered e, correctly, and answers for other exercises are the same, then the proficiency level of s; should not be greater than sy. Formally, for only one j E ri.j = 0, j = 1, while for & E Q, k + J. Ti = kr, we have 0; S Og.
To leverage these hypotheses and enhance the training data mitigate the data sparsity issue, we first need to construct suitable interaction records for students. Specifically, for student 8;, we randomly select one of his answered exercise Ej E Ei and flip the corresponding score on the interaction record (changing I to O or 0 to 1), and keep other records unchanged. By doing so, we generate the records for sp, Assume the proficiency of sy is Bs.g. the fitting function for student Sy can be expressed as:
4.5(0
Σ IALK pa Pa (O + +1 Vick)) log(1 - pk (O(a)))
kE KEQUAFS
(3)
[(1 V4.5) log Py (0.,)) + Di. log(1 (O
Next, we consider the partial order relationship between 8;, sp. Two scenarios need to be considered.
According to Hypothesis 1, when 0: 2 O2g should hold but 0g < Oz, in fact, we need to add a regularization term (0..) - 0,)² to impose the constraint. Similarly, as for Hypothesis 2, when 0, S fg should hold but 0, > f,., in fact, we need add regularization term (0, 0,,,) ) as penalty. As result, we establish two regularization terms:
2i (0,, 0..)) c0,
0.)², A2(0,, 0..)) 00, (0: 0z5)²
where 1 is the indicator function. Then we summarize the final regularization term as:
M( 0g 01.j) = yt. + (1 vi,,
(5)
Moreover, to enhance the monotonicity assumption, for each student 8;. we consider generating multiple fake students and comparing them with s; by randomly sample a subset C, C Q. For each exercise in C. we perform flipping operation on the corresponding score while keeping the other records unchanged, resulting |C;] versions student 8,.
As result, the final loss function in CMCD is:
L
4(0.) + Σ
(Os)( + ((I.),0,00,))
(6)
jEC.
where e represents the strength of constraint.
5.2 Theoretical Guarantees
In this section, we provide theoretical guarantees of CMCD's effectiveness in terms of accuracy and convergence speed. For theoretical analysis, we will focus the most classic cognitive diagnostic model, IRT (introduced in Section 3.2), which serves as the foundation for many other cognitive diagnostic models and has been widely implemented in GRE [30]. Please note that the following theoretical analysis can be easily extended to other CDMs.
Firstly, we analyze the accuracy of CMCD. Given that the core objective of cognitive diagnostic tasks is to estimate the true proficiency 0, of student 8;, to assess the accuracy of our CMCD, we theoretically analyze the Maximum Likelihood Estimation (MLE) of 0; through the optimization of A (Eq. (6). Let θ (0,, 022 ..), we assert that CMCD has accurate estimation stated as:
Proposition 5.1 (Accuracy). CMCD can accurately estimate the student proficiency level θ.
To validate the above proposition, we prove that the MLE under CMCD is consistent estimation: Theorem 5.2. Suppose student 8; with ability 0, has records R; exercises indexed by Qi and 0% denote the MLE of 0; under f.A. then 0 is consistent estimation of 0,, formally,
VE > 0,8 > 0, 3no E Z-, 1Qal 2 S.t.P (10; 0,1 2 <
Detailed proofs for Theorem 5.2 can be found in Appendix A.1 The consistency estimation ensures the accuracy of CMCD. Next, we explore the superiority of CMCD in convergence speed compared to traditional CD models (the loss function L is introduced in Eq.

---

Proposition 5.3 (Convergence Speed). In terms of the estimation of proficiency level e CMCD has faster convergence speed than the original approach.
To validate the above proposition, we first notice that C and LA are both convex. Then, in the specific zone that violates the monotonicity assumption, prove that La strong-conye in terms of e. Finally we can analyze and compare the convergence speed in optimizations of L and LA- Theorem 5.4. L and A convex in terms of (. In addition, LA is strong-coni convex when
+ E D {(01, 02 ......,0)llve E [mm] 3j E Ci Vi. 10. + (1 Vk 1}, (8)
and, meanwhile, we can also conclude 4 A V when E
Detailed proofs for Theorem 5.4 are
provided in Appendices A.2 When
performing the gradient descent algo- rithm optimize the loss function, is widely known that strong-convex functions converge faster than convex functions [3]. Overall, this represents the primary distinction between C, and C. By examining the frequency of occurrences of this expression dur- ing the optimization process, we can analyze the convergence speeds the two functions. The more frequent occurrences, the faster the conver- gence of CA. Simultancously, this ex- pression also indicates a violation of hypotheses, necessitating the in- corporation of regularization. To validate the superior convergence speed, we calculated instances of hypothesis violation across different datasets, with the results illustrated in Figure It reveals prevalent occurrence of strong convexity in L across various datasets and backbones. Therefore, by optimizing CA, CMCD converges faster.
20031
7090
0000
4900
100000
19000
Pyat
Rook
(a) Math dataset.
(b) ASSIST dataset.
Figure 5: The number of instances violating our hypotheses during the optimization process.
6 Experiments
In this section, we first introduce the dataset and experimental setup. Then, we conduct extensive experiments real-world datasets to answer the following questions:
RQ1: Does CMCD learn fair and accurate cognitive diagnosis results?
RQ2: How does the constraint of data augmentation impact CMCD?
RQ3: Does CMCD have faster convergence speed compared to other baseline models?
RQ4: How do different sypee-parameter settings (i.e. c, |C/)) affect the performance?
The code is released https///ghhbbbcooo/mMeeeraaaaaCa
6.1 Experimental Setup and Baselines
this paper, we conduct experiments two diagnostic datasets, namely Math AS- SIST. The Math dataset was collected from online educational system that provides homework, and evaluation for students. It collected records junior high school students on mathematical exercises. In this dataset, take the region the sensitive attribute. The ASSIST dataset (ASSISTments 2009-2010 "skill builder") is an open dataset collected by the ASSISTments online tutoring systems [2]. which includes student response logs and knowledge concepts Given that the ASSIST dataset does not provide sensitive attribute information, we follow (23) and categorize users into two groups based whether the number of their responses exceeds 50. Regarding the dataset division, allocate 80% of each student's 's response log for training and the remaining 20% for testing. The overview of basic statistics can be found in Appendix Table
Datasets

---

Table I: The utility (U) and fairness (F) results Math datasets. The best results on each backbone are highlighted in bold. + indicates the lower, the better. f indicates the higher, the better. We compare
CMCD with the Origin and get the p-value in the t-test. (* p 0.001, p < 0.01,
p < 0.05.)
IRT
MIRT
NCDM
Approach
AUCT
ACCT
MIXET
AUCT
ACC
AUCT
ACCT
Ongin
0402
0333
0.211
0.251
0.409
0.309
0322
0.757
0444
0.317
0X822
0,248
Ch5-Reg
0412
0.32
0.54
0,747
0,409
0.310
0.331
0.753
0416
0.548
0.80%
0,744
CD+BO
0412
CDOP
0.499
0.329
0888
0.249
0.410
0.314
0.819
0.753
0.444
0.331
0.811
0.248
CF-IRT
0.02
0.114
0.531
0.733
0,411
0.304
0.31
0.755
0.400
0.316
0.800
0.245
0.088
0.324
0,417
0,300
0.866
0.808
CE-CCDM
0.406
0.318
0,824
0.756
0.406
0.312
0.836
0,758
0.437
0.312
0.80M
0.267
CMC
0.84
0772
0,406
0229
1.413
0.316
0.14
T.740F
Appronch
ARMSRI
AMARI
AAUCI
SACCL
ARMSEL
AMARL
AAUCI
AACCI
ARMSEL
AMART
AAUCI
AACCL
Oriin
0008
0.022
0.021
0,058
0.032
0.035
0.018
0.015
0.04
0,070
0.032
0,054
0007
0049
0.056
03032
03030
CD&RO
0041
0.041
0.022
0.062
0.036
0.025
0.020
0.047
0.640
IL054
0.027
0.057
CD+DP
0.088
0.089
0.033
0,059
0,033
0,036
0.020
0.013
0.035
0.076
0.052
0,050
CR-RT
0.88
0.422
0.025
0.034
0.040
0.033
0,047
0,042
0.039
CF-MIRT
0.34
0.088
0.020
0.050
0.040
0.092
0.022
0.047
0.038
0.067
0.057
0.066
CF-NCDM
0.034
0.044
0,022
0,050
0,031
0,042
0.018
0.043
0.039
0.055
0,034
0,052
0,034
0.028
0.020
0,029
0,033
0,055Fe9
Evaluation. In this paper, we evaluate the results of user modeling by predicting scores. This evaluation can be divided into two aspects: utility and fairness evaluation. For utility evaluation, following previous works [4] we adopt different metrics from the perspectives of both regression (MAE, RMSE) and classification (AUC, ACC). For fairness evaluation, adopt the Definition introduced in Section 3.2. For the .M in Eq. 2 we adopt MAE, RMSE, AUC, ACC (denoted as AMAE, ARMSE, AAUC, AACC in the experiment).
Baseline approaches. CMCD is versatile framework applicable to various CD models. To validate its generality, we employed CMCD in three classical CD backbones, IRT (29] MIRT [5].
NCDM 39. For the baselines, we compare our methods with two categories of methods. Firstly, we
compare our methods with the data augmentation baseline. To the best of our knowledge, there are
currently no data augmentation methods specifically applied to diagnostics. Therefore, following the approach [44], we constructed the following baselines: CF-IRT, CF-MIRT, and CF-NCDM. Secondly, our model enhances fairness, we compare it with several classical fairness baselines, such as CD+GF [23,, CD+EO 47), CD+DP 47). Further details regarding these baselines and the implementation of our methods can be found in Appendix A.3
6.2
Experimental Results
RQ1. In this section, we discuss whether CMCD framework can alleviate the issue of data sparsity, thereby enhancing both the accuracy and fairness of the model. Specifically, conducted compara- tive experiments with baselines on the Assistment and Math datasets, and the results of Math datasets results are presented in Table [ the results of Assistment datasets can be found in Appendix A.. From these results, we can draw the following findings:
(1) From the perspective of utility, we observed significant enhancement in CMCD across different datasets and various backbones. This underscores the effectiveness our approach. Furthermore, we noted that, compared to backbone models like NCDM that rely on neural networks, CMCD demonstrates more pronounced improvements on traditional backbones (IRT). We attribute this to the fact that models dependent neural network backbones, such as NCDM, exhibit superior data fitting capabilities, thereby alleviating issues related to data sparsity. Consequently, the gains achieved by CMCD on NCDM are less conspicuous. In contrast, traditional IRT models are more susceptible to the impact of data sparsity, making CMCD's s influence more substantial in such cases.
(2) From the perspective of fairness, we observed that CMCD can, to certain extent, alleviate unfair phenomena cognitive diagnostic models. Particularly noteworthy is its performance on the Math dataset, where our model achieved the best fairness outcomes across different backbones (IRT, MIRT). This underscores that CMCD, while mitigating data sparsity issues, concurrently promotes fairness in model outcomes.
8

---

(3) From the perspective of the orfde-off between faimess and utility, our model consistently surpasses the baselines with data augmentation both performance and fairness outcomes. While compared to fairness-awand baselines, our model may not always outperform dedicated fairness-enhancion methods on specific datasets, it is important to note that these baselines often come at the expense of utility. In contrast, CMCD not only enhances fairness but also improves performance. This highlights that CMCD achieves the optimal trade-off between fairness and utility.
RQ2. To address the issue of data sparsity, we introduce two hypotheses for data augmentation. In this section, we validate the effectiveness of these two hypotheses. Specifically, duct ablation experiments on the Math dataset, where systematically re- moved the corresponding hypothesis strategies. The results are illustrated in Figure 6 It is evident that after removing different hypothesis strate- gies, both Fairness and Utility perfor- mances exhibited varying degrees of decline, providing evidence for the ef- ficacy each strategy. Additionally, we observed that these two hypotheses yield different effects various backbones. In comparison to NCDM, a neural network-dependent model, hypotheses perform better on traditional models like IRT and MIRT. We attribute this to the powerful data-fitting capability of NCDM, which alleviates issues related to data sparsity. In contrast, models such as IRT and MIRT, constrained by data sparsity, exhibit limited capabilities.
01900
Meyallon
CMED thpottale
con-
0.30
GECACC)
0.200
0.20
0,22
Utiliy
Math
(b) Fairuese Math
Figure 6: The ablation experiment on Math dataset.
RQ3. In Section 5.2, we theoreti- cally demonstrated that CMCD ex- hibits a faster convergence rate. In this section, we empirically validate the superiority of CMCD in terms of convergence speed. Specifically, we compared the convergence speeds (the epoch at which early stopping occurs) of different baselines various back- bones using the Math dataset, as de- picted in Figure 7 It is observed that in the early stages, CMCD shows min- imal differences compared to other baselines. We attribute this to the fact that, initially, the effectivvness of the backbone models is not optimal, resulting in less precise enhancement of the monotonicity assumption. However, as the number of epochs increases, CMCD significantly outpaces other baselines, demonstrating a faster convergence.
Dega:
DUCD
CD+
CD+BB
C3- 19-121
CS-MBBT
CR-NCD
Spedi
Roch
Epech
Figure 7: Convergence against training epoch for different backbones Math dataset.
RQ4. In CMCD, two hyperparameters control the effectiveness. Specifically, e controls the strength of the constraints, and |Ci| regulates the number of generated students. In this section, investigate the impact of adjusting these two hyperparameters on CMCD using the Math dataset. The specific results are presented in Figure S and Figure 9 In Figure a consistent trend is observed across different backbones. Initially, with the strengthening of c. both faireess and utility show improvement, indicating the efficacy of our monotonicity enhancement. However, as e reaches a certain intensity, both fairness and utility experience decline. We attribute this to the possibility that an excessively large e might interfere with the model's normal training. leading to decrease in effectiveness. Moving to Figure [ diverse trends are identified across different backbones. Specifically. in the case of IRT, as the number of contrasts increases, both fairness and utility consistently improve. Conversely, for NCDM, as the quantity of contrasts grows, there is a continuous decline in utility. We attribute these observations to inherent differences in the nature of the models. IRT is more affected by data sparsity, enhancing C, effectively mitigates data sparsity, yielding improved results. On

---

0.42
0,749
Uniliy
0.041
00082
0,708
00000
A.551
Unity
Falmess
Unilty
Faimess
Uailly
Falmess
2 0,707
0.748
10000
0.029
0.266
10,040
0.037
0.765 01
03 6iS Dia "665"
006
0.765
01
02" 0f
10d 03
0.747
6. 0.3 "Too"" Imprect of (NCDM)
05
60 Erggnet ofe (IRT)
Impact of (MIRT)
Figure 8: The impact of for different CD backbones the Math. For utility, the higher the better; For fairness, the lower the better.
0.007
Uniliy
0.77
0142
0.06
Faimess
0.002
0.040
0.147
3.034
0.713
0.038
Loumess
Ediltty
0.003
EsStty
30,000
Falmes
0.771
0.037
01016
0.002
0.745
0.15
00.031
0.048
0.033
0.743
0.767
an12
0.36
0.030
0.765
0.028
0,741
0.006
(a) Impact of (IR)
(b) Impact of (C) (MIRT)
(6) Impact of [C] (NCDM)
Figure 9: The impact of |C;] for different CD backbones on the Math. For utility, the higher the better; For fairness, the lower the better.
the other hand, NCDM, benefiting from its neural network's 's fitting capacity, partially alleviates data sparsity issues. Consequently, an increase in contrasts may interfere with intrinsic performance.
7 Conclusion and Discussion
In this paper, conducted focused study addressing data sparsity in CD. Initially, empirically validated that data sparsity leads to inaccurate and unfair diagnostic results. Subsequently. by integrating data augmentation and the monotonicity assumption, introduced constraints to alleviate the issue of data sparsity. Moreover, we provided theoretical guarantees regarding accuracy and convergence speed. The experiments real-world datasets demonstrated the effectiveness of our approach in addressing the data sparsity issue, culminating in fair and accurate diagnosis results.
CD is highly significant task in intelligent education. The generated diagnostic results be applied to various areas, e.g., personalized tutoring and intelligent question recommendations. This can reduce the burden teachers and students, providing effective learning experiences for students. Our CMCD mitigates the unfairness and inaccuracy issucs arising from data sparsity, thereby promoting educational equity to certain extent. Simultaneously, response records that might be inaccessible due to privacy concerns. the future, we will contemplate incorporating federated learning to enhance diagnostic services and ensures student privacy.
recognize that CD relies students'
Acknowledgments and Disclosure of Funding
This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2021 YFF0901003), the Key Technologies R&D Program of Anhui Province (No. 202423k09020039) and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Statistics Reference Online, 2014.
[2] Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4561-4569, 2019.
[3] Amir Beck, First-order methods optimization. SIAM, 2017.
[4] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model- model-agnostic framework for computerized adaptive testing. In 2020 IEEE International Conference on Data Mining (ICDM), pages 42-51, IEEE, 2020.
[5] R Philip Chalmers. mirt: A multidimensional item response theory package for the r environ- ment. Journal of statistical Software, 2012.
[6] Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4613-4623, 2019.
[7] Xiangzhi Chen, Le Wu, Fei Liu, Lei Chen, Kun Zhang, Richang Hong, and Meng Wang. Disentangling cognitive diagnosis with limited exercise labels. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[8] Shuang Cui, Kai Han, Shaojie Tang. Feng Li, and Jun Luo. Fairness in streaming submodular maximization subject a knapsack constraint. In ACM Knowledge Discovery and Data Mining (SIGKDD), pages 514-525, 2024.
[9] Shuang Cui, Kai Han, Tianshuai Zha, Jing Tang, Benwei Wu, and He Huang. Randomized algorithms for submodular function maximization with k-system constraint. International Conference Machine Learning (ICML), pages 2222-2232, 2021.
[10] Jimmy De La Torr. Dina model and parameter estimation: A didactic. Journal of educational and behavioral statistics, pages 115-130, 2009.
[11] Susan E Embretson. Item response theory. Psychology Press, 2013.
[12] Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with an online system that tutors as it assesses. User modeling and user-adapted interaction, pages 243-266, 2009.
[13] Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mita- mura, and Eduard Hovy. A survey of data augmentation approaches for nip. arXiv preprint arXiv:2/05.03075. 2021.
[14] Weibo Gao. Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang. Jianhui Ma. Shijin Wang, and Yu Su. Red: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 501-510, 2021.
[15] Weibo Gao, Qi Liu, Hao Wang, Linan Yue, Haoyang Bi, Yin Gu, Fangzhou Yao, Zheng Zhang, Xin Li, and Yuanjing He. Zero-1-to-3: Domain-level zero-shot cognitive diagnosis via one batch of carly-bird students towards three diagnostic objectives. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8417-8426, 2024.
[16] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1331, 2016.
[17] Rundong He, Zhongyi Han, Xiushan Nie, Yilong Yin, and Xiaojun Chang. Visual out-of- distribution detection in open-set noisy environments. International Journal of Computer Vision, pages 1-18, 2024.

---

[18] Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et Trustllm: Trustworthiness large language models. arXiv preprint arXiv:2401.0556/ 2024.
[19] Zhenya Huang, Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management. pages 1261-1270, 2019.
[20] Michael Jordan and Tom M Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255-266 2015.
[21] Jiatong Li, Qi Liu, Fei Wang, Jiayu Liu, Zhenya Huang, Fangzhou Yao, Linbo Zhu, and Yu Su. Towards the identifiability and explainability for personalized learner modeling: An inductive paradigm. In Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pages 3420-344 3. ACM, 2024.
[22] Peizhao Li and Hongfu Liu. Achieving fairness at utility cost via data reweighing with influence. In International Conference on Machine Learning, pages 12917-12930. PMI.R, 2022.
[23] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. User-oriented fairness in recommendation. In Proceedings of the Web Conference 2021, pages 624-632, 2021.
[24] Zuoquan Lin and Hanxuan Chen. Recommendation over time: a probabilistic model of time- aware recommender systems. Science China Information Sciences, 62(11):1-14, 2019.
[25] Qi Liu. Towards a new generation of cognitive diagnosis. In Proc. of IJCAI, pages 4961-4964, 2021.
[26] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering,
2019.
[27] Yuhan Liu, Xiuying Chen, Xiaoqing Zhang, Xing Gao, Ji Zhang, and Rui Yan. From skepticism to acceptance: Simulating the attitude dynamics toward fake news. In International Joint Conference on Artificial Intelligence, 2024.
[28] Yuhan Liu, Zirui Song, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. From a tiny slip to a giant leap: An Ilm-based simulation for fake news evolution. 2024.
[29] Frederic Lord. A theory of test scores. Psychometric monographs, 1952.
[30] Robert McKinley and Neal Kingston. Exploring the use of irt equating for the gre subject test in mathematics. ETS Research Report Series, 1987.
[31] Craig N Mills. The gre computer adaptive test: Operational issues. In Computerized adaptive testing: Theory and practice. 2000.
[32] Xingyu Peng, Zhenkun Zhou, Chong Zhang, and Ke Xu. Online social behavior enhanced detection of political stances in tweets. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 1207-1219, 2024.
[33] Liang Qu, Ningzhi Tang, Ruiqi Zheng, Quoc Viet Hung Nguyen, Zi Huang, Yuhui Shi, and Hongzhi Yin. Semi-decentralized federated ego graph learning for recommendation. In Proceedings of the ACM Web Conference 2023, pages 339-348, 2023.
[34] Liang Qu, Wei Yuan, Ruiqi Zheng, Lizhen Cui. Yuhui Shi, and Hongzhi Yin. Towards personalized privacy: User-governed data contribution for federated recommendation. In Proceedings of the ACM on Web Conference 2024, pages 3910-3918, 2024.
[35] Mark Reckase. Multidimensional item response theory models. Multidimensional item response theory, pages 79-112. Springer, 2009.
12

---

[36] Lawrence M Rudner. Implementing the graduate management admission test computerized adaptive test. In Elements of adaptive testing, pages 151-165. Springer, 2009.
[37] Pengyang Shao, Le Wu, Kun Zhang. Defu Lian, Richang Hong, Yong Li, and Meng Wang. Average user-side counterfactual faimess for collaborative filtering. ACM Transactions on Information Systems, 42(5):1-26, 2024.
[38] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[39] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6153-6161. 2020.
[40] Yifan Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. A survey on the fairness of recommender systems. ACM Transactions on Information Systems, 41(3): 133) -4,,40223 2023.
[41] Yifan Wang, Peijie Sun, Weizhi Ma, Min Zhang, Yuan Zhang, Peng Jiang, and Shaoping Ma. Intersectional two-sided fairnsss in recommendation. In Proceedings of the ACM on Web Conference 2024, pages 3609-3620, 2024.
[42] Zaitian Wang. Pengfei Wang, Kunpeng Liu, Pengyang Wang, Yanjie Fu, Chang-Tier Lu, Charu C Aggarwal, Jian Pei, and Yuanchun Zhou. A comprehensive survey on data augmenta- tion. arXiv preprint arXiv:2405.09591, 2024.
[43] Zhenlei Wang. Jingsen Zhang, Hongteng Xu, Xu Chen, Yongfeng Zhang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Counterfactual data-augmented sequential recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 347-356 2021.
[44] Kun Xiong, Wenwen Ye, Xu Chen, Yongfeng Zhang, Wayne Xin Zhao, Binbin Hu, Zhiqiang Zhang, and Jun Zhou. Counterfactual review-based recommendation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2231-22400 2021.
[45] Shangshang Yang, Xiaoshan Yu, Ye Tian, Xueming Yan, Haiping Ma, and Xingyi Zhang Evolutionary neural architecture search for transformer in knowledge tracing. Advances in Neural Information Processing Systems, 36, 2024.
[46] Fangzhou Yao, Qi Liu, Min Hou, Shiwei Tong, Zhenya Huang, Enhong Chen, Jing Sha, and Shijin Wang. Exploiting non-interactive exercises in cognitive diagnosis. In Proc. of IJCAI, 2023.
[47] Sirui Yao and Bert Huang. Beyond parity: Fairness objectives for collaborative filtering. Advances in neural information processing systems, 30, 2017.
[48] Xiaoshan Yu, Chuan Qin, Dazhong Shen, Haiping Ma. Le Zhang, Xingyi Zhang, Hengshu Zhu, and Hui Xiong. Rdgt: Enhancing group cognitive diagnosis with relation-guided dual-side graph transformer. IEEE Transactions on Knowledge and Data Engineering, 36(7)3342933442 2024.
[49] Xiaoshan Yu, Chuan Qin, Dazhong Shen, Shangshang Yang, Haiping Ma, Hengshu Zhu, and Xingyi Zhang. Rigl: A unified reciprocal approach for tracing the independent and group learning processes. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge 2024.
Discovery and Data Mining, pages
[50] Dacao Zhang, Kun Zhang, Le Wu, Mi Tian, Richang Hong, and Meng Wang. Path-specific causal reasoning for fairness-aware cognitive diagnosis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4143 4154, 2024.
[51] Hongbo Zhang, Pei Chen, Xuelong Xie, Zhaoqu Jiang, Zihong Zhou, and Lingyun Sun. A hybrid prototype method combining physical models and generative artificial intelligence to support creativity in conceptual design. ACM Transactions on computer-Humace Interaction, 2024.
13

---

[52] Hongbo Zhang, Pei Chen, Xuelong Xie, Chaoyi Lin, Lianyan Liu, Zhuoshu Li, Weitao You, and Lingyun Sun. Protodreamer: A protooype tool combining physical model and generative ai to support conceptual design. In Proceedings of the 37th Annual ACM Symposium User Interface Software and Technology, pages 1-18, 2024.
[53] Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang. Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. Fairlisa: Fair user modeling with limited sensitive attributes information. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
[54] Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. Understanding and improving fairness in cognitive diagnosis. Science China Information Sciences, 67(5):152100 2024.
[55] Hongke Zhao, Likang Wu, Yuqing Shan, Zonghan Jin, Yuanpei Sui, Zipeng Liu, Nan Feng, Minqiang Li, and Wei Zhang. A comprehensive survey of large language models in management: Applications, challenges, and opportunities. Challenges, and Opportunities (August 14, 2024), 2024.
[56] Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv. Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of Ilms: An adaptive testing perspective. arXiv preprint 1.XXi::3306.051222. 2023.
[57] Ran Zmigrod, Sabrina J Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. arXiv preprint 1r0iv::906..45500 2019.
A Appendix
A.1 Proof of Theorem 5.2.
Theorem A.1. Suppose student s, with ability 0g has records R& on exercises indexed by Qs and O denote the MLE of 0; under L. then 6 is consistent estimation of 0;, formally
WE 0,8 0, 3no E Z= 12dl 2 no, s.t.P (10: 0,1 2 < <.
(9)
Proof. We denote 0, as the maximum likelihood estimate (MLE) of 0; for the loss function C (Eq. The mean negative log-likee fog-likelihood function of student 8; is
(0)
Σ JEQ.
[yn. log (0) + (1 - vk.)) log(1 (0))
(10)
We can further validate that
((()) Σ
[assuns (0)))
jEQ,
(11)
(0) = Σ ()); PT(()).
jEQ.
On the other hand, the MLE 0 satisfies ((((,)) = 0, by taking Taylor expansion,
0 = ((())) = (2() + (E)) -0),
(12)
where & is between 0 and 0, We can further get
(()).) (E)
Eyeo as(ıi - P), Ejeo, (E), py (E.))
00. 0.
(13)
We can that {yr,) : Qi} are independent and Elg,,,1 ()())). Through Chebyshev's 's Law of
Large Numbers, VE > 0, > 0, there exists ny such that
P
ajPi. - Q
Σ (0,)
2
2
< 6,
(14)
Qi
4
JEQ:
JEQ,
14

---

when 12:l 2 holds, here a = max, aj. Considering
1
Σ
a}p,(E)(L -1, -
Σ
aj
-
Σ JEQ.
a
1
(15)
1Q
4Q2
4
JEQU
jEQ.
we can see when [Qal 2 no. with probability less than 6,
a2e/4 E,
0:-0:
(16)
a /4
which means 0g is a consistent estimation of 0z. Then, we denote Org arg mino (0), and we have
(0.) (2(6,) - (4, - 4) = -a,(1 - 2ys.)).
(17)
If Mig = 1, (0.) = > 0 and it means 0, is bigger than the zero point, i.e., 0, > Ò.j. If yo. = O. (0.) - aa < and means 0, is smaller than the zero point, i.e., 0, < Ōz.-- Overall, we can check that (2 0, 81.)) O.
In this way, 0, makes 1,) the smallest, d.gj makes (0) the smallest, and with them the regularized terms are always Therefore, such and 0z,5 are the minimum points of C and this fact results in 0r 0j. Combining the conclusion that 0: is consistent estimation of 0; we can conclude that 0 is a consistent estimation of 0,.
A.2 Proof of Theorem 5.4.
Theorem A.2. L and L are convex in terms O. In addition, L is strong-convex when
Oe {(11, (01,002...,,mmnn Om))Vi [m], Ej C₁ Vi.i (0 (1 Mr.))(0.>00.11 = 1}, (18)
and, meanwhile, we can also conclude V2 L θ e D.
Proof. For the loss function (Eq.
(p). the mean negative log- likelihood function of student is
(0) Σ
[e (0) (1 y(.) log(1 p,))
(19)
JEQ:
We can further validate that
((()) = Σ
[0y - ((() = Σ (1) 2 0.
(20)
JEQ:
JEQ;
For any i.j we have In addition, we see that
8 80
0 and
8c 0, hence V 2 and L is convex in terms of θ. 80,00;
LL =
JC
0C
+ 2c Σ
(uollp.coanl + Wi. (1.20.)))))))
2 0,
(21)
003
002
000-
jEC.
and C for all i.j. Hence V2LA V2C 0 and CA is also convex in terms of θ
09,00,
When θ E D. we can see that
2c
Σ (Visl + yi.) 9..)) > 0.
(22)
jEC.
Based on Eq. 21 we can conclude that LA is strong-convex and V2L > 7LL when θ E D. In
fact, we have L > 2c min; Bi 10. <0.,) + (1 11 when θ D. D
15

---

Table 2: The statistics of the datasets.
Math
ASSIST
#Students
7.177
4.163
#Exercises
12,129
17,746
#Knowledge concepts
2,076
123
#Response logs
384,356
324.572
Sparsity
99.558%
99.561 %
A.3 Experiment Details
Baseline approaches. CMCD is versatile framework applicable to various CD models. To
validate its generality, we employed CMCD in three classical CD backbones, IRT 29 MIRT
NCDM 39 The details of baselines are as follows:
IRT (29 is cognitive diagnosis method that models the cognitive processes from students'
exercise records with a logistic-like function.
MIRT 5 is variant of the basic IRT model, which extends the unidimensional latent traits of students and exercises in IRT to multidimensional vectors.
NCDM [3 is a deep neural cognitive diagnosis framework that models the complex interaction from students' exercising records with multilayer perceptron (MLP).
CMCD can alleviate the issue of data sparsity in diagnostic models, enhancing both accuracy and fairness. To validate the effectiveness of our approach, we compare it with two categories methods. The first category is data augmentation methods. To the best of our knowledge, there are currently no data augmentation methods specifically applied to diagnostics. Therefore, we adapt the data augmentation concept "What would ... if ...?" from traditional recommendation systems to the cognitive diagnostic field. Specifically, following the approach [44], we apply a causal rule: "What would the student's is response be if they encountered characteristics CD domain, we make appropriate adaptations by using classic IRT, MIRT, and questions. Ultimately, extend the data augmentation
question?" Additionally, considering the
NCDM models predict responses baseline to CF-IRT, CF-MIRT, and CF-NCDM. The second category is fairness-aware methods. Since model enhances fairness, compare it with several classical faimess baselines, CD+GF 23 CD+EO 47 CD+DP [7) The details of baselines are as follows:
CD+GF: well-known fairness improvement strategy that considers the faimess metric
regularization for the loss and has been used in prior fairness works 47
23
In our work,
we use Eq. (4) as regularization for Eq. (2);
CD+EO: method regards Equal opportunity (EO) regularization
16
CD+DP: method regards Demographic Parity (DP) regularization [16]
Implementation detail. In terms of model parameter configuration, NCDM adheres to the settings outlined in (39.. The fully connected layers have dimensions of 512, 256, and 1. The sigmoid function serves as the activation function for all layers. MIRT's dimension of student proficiency parameters matches those in NCDM. For all models, set the learning rate to 0,001 and the dropout rate to 0.2. We apply Adam as the optimization algorithm to update the model parameters. We implement all models with PyTorch and conduct all experiments on four 2.0GHz Intel Xeon E5-2620 CPUs and a Tesla K20m GPU.
A.4 The results on Assistment dataset (RQ1)
The results for the Assistment datasets can be found in Table They exhibit similar patterns those introduced in RQI.
16

---

Table 3: The utility (U) and fairesss (F) results Assistment datasets. The best results each backbone are highlighted in bold. 1. indicates the lower, the better. ? indicates the higher, the better.
We compare our CMCD with the Origin and get the p-value in the t-test. (* p 0.001, ** p
0.01, * p < 0.05.)
MIRT
NCDM
Approach
AUCT
ACCT
AUCT
ACC
AUCT
ACCT
Ongin
0440
0372
0.707
0.095
0.447
0.324
0.733
0,707
0A33
0232
0745
0.255
CIs-Reg
0.46
0.93
0.713
0.693
0.451
0.353
0.719
0.08
0.433
0.72
0742
0.735
CD+BO
CLOD
0451
0.389
0493
0.685
0.450
0,350
0.233
0.701
0.434
0.362
0.41
0.721
CF-IRT
0448
0.51
0.70
0.691
0,449
0.340
0.730
0.700
0.44
0.19
0748
0.736
CE-CCD
0.448
0.377
0.707
0.695
0.448
0.344
03332
0,700
0.436
0.336
0.748
0.725
CMC
8.745
0715
0337
1L758
1L72
0.22
0.727
Appronch
ARMSRI
AMAR]
AAUCI
SACCL
ARMSEL
AMARL
AAUCI
AACC1
ARMSEI
AMARI
AAUCI
AACCI
Ongin
00055
0.013
0,028
0.057
0,039
0,042
0.032
0.02
0.019
0.026
0.006
01027
CDiRkgg
0041
01029
CD&RO
0035
0.031
0.018
0.064
0.038
0.024
0.012
0.844
0.000
0.03
0.005
0.300
CD+DP
0.034
0.018
0.031
0,060
0,036
0,038
0.022
0.019
0.021
0.038
0.003
0.039
0.41
0.048
0.400
0.025
CF-MIRT
0.37
0.41
0.027
0.060
0.043
0.040
0.032
0,033
0.017
0.025
0.410
0.066
CF-NCDM
0.085
0.042
0,027
0.057
0,044
0,058
0.048
0.062
0.018
0.026
0.005
0.024
0033009
0.048
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ) In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
17

---

The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides the full set of assumptions and complete proof section 5.2 and section 9.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
18

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results in section 6
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper provides open access to the code in section 6
Guidelines:
19

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss//nips.c cc/
wwt.suubiiiiioooooooooooo..... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.ccal wwtpccccppccg.u..e..eeecc /tnpi////uuddee/eeeuuuubiiiii soiobiissiiiooccycccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details in section 6.1 and section
9.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper report error bars suitably and correctly defined in Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
20

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information on the computer resources in section
9.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://////.errpsssccccppp/tttt/
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts are introduced in section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
21

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't use data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The dataset used is introduced in Section 6.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswwthcode cww/co//////ooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't 't introduce assets.
22

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/codc/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: subjects.
We don't conduct crowdsourcing experiments and research with human
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't conduct experiments by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "bA5E9ru58j", "reviewer": "Reviewer_A4mA", "summary": "This paper discusses the issues of fairness and accuracy in cognitive diagnostic tasks, which hold significant societal value and impact the fairness of education. Through an experimental perspective, the paper analyzes how data sparsity can lead to unfairness and inaccuracy in cognitive diagnostic tasks. It also leverages the specific assumption of monotonicity in the field of education to alleviate the challenges posed by data sparsity. Notably, the authors theoretically prove the convergence and efficacy of the proposed method. Finally, an analysis is conducted on real-world datasets, accompanied by thorough validation, providing substantial evidence for the efficacy of the approach.", "questions": "See above weaknesses.", "limitations": "The authors have extensively discussed the limitations of their approach.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- S1: This paper delves into a highly significant societal concern - the accuracy and fairness in cognitive diagnosis. To the best of my knowledge, this paper is the first to unveil how data sparsity can yield inaccuracies and unfairness in cognitive diagnosis.\n- S2: From a data perspective, this paper effectively mitigates the challenge of data sparsity, steering clear of compromising the model's interpretability from a modeling standpoint. The motivation is sound, and the method is indeed compelling.\n- S3: The paper undergoes validation from two key standpoints - experimental and theoretical. The validation methods are thorough and comprehensive, ensuring a robust evaluation.", "weaknesses": "- W1: I noticed that the experimental results at the end show an improvement in both the fairness and accuracy of diagnostics simultaneously, which seems to contradict the prevailing trade-off between fairness and accuracy. It would be beneficial for the paper to delve deeper into this issue, exploring whether this approach can be extended to other domains such as the field of recommender systems.\n- W2: In the experimental section, I observed that the utility and fairness metrics are aligned. While I understand that fairness analysis is based on utility outcomes, this alignment could potentially mislead readers to some extent. Clarity on this relationship would enhance the understanding for the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: I noticed that the experimental results at the end show an improvement in both the fairness and accuracy of diagnostics simultaneously, which seems to contradict the prevailing trade-off between fairness and accuracy. It would be beneficial for the paper to delve deeper into this issue, exploring whether this approach can be extended to other domains such as the field of recommender systems.**\n\nThank you for your constructive feedback. In fact, in the realm of fairness research, there does exist a trade-off between fairness and accuracy. This implies that while traditional fairness-aware methods can enhance fairness, they also tend to reduce utility. We have validated this phenomenon through experiments. Specifically, Table 1 displays the accuracy results of CMCD and the baselines, while Table 2 showcases the fairness results. Among these baselines, CD+Reg, CD+EO, and CD+DP are all considered fairness baselines. We observe that these baselines have to some extent improved the fairness of the original cognitive diagnostic model but have simultaneously decreased its performance.\n\nWhat sets our work apart in this article is our departure from traditional fairness approaches. We have approached the issue from a data perspective and, through extensive data analysis on real datasets, discovered that data sparsity can lead to inaccurate and unfair outcomes. The specific data analysis is presented in Section 4. This finding suggests that mitigating data sparsity in cognitive diagnostic tasks can alleviate the trade-off between fairness and utility. However, it is worth noting that in terms of the improvement in fairness, CMCD does not consistently outperform traditional fairness baselines and achieve state-of-the-art results. This discrepancy is due to our work considering both accuracy and fairness, highlighting the inherent trade-off between fairness and accuracy. \n\nFinally, we discuss the potential extension of our work to other domains, such as recommender systems. The data distribution in cognitive diagnostic fields and recommender systems exhibits similar trends. Therefore, alleviating data sparsity issues can potentially enhance both fairness and accuracy simultaneously, making the data-driven approach transferrable. However, our method relies on the unique monotonicity assumption in cognitive diagnostic fields to enhance data, combined with the cognitive diagnostic paradigm to provide corresponding theoretical guarantees. This approach cannot be directly extended. Thank you once again for your feedback. We will incorporate the relevant discussion in the revision.\n\n\nTable 1. The accuracy result of CMCD and baselines on NCDM （for the RMSE, MAE metric, the lower the better, for the AUC, ACC metric, the higher the better）\n| Baselines  | RMSE | MAE | AUC | ACC |\n| -------- | ------- | ------- | ------- | ------- | \n| origin  | 0.414 | 0.317 | 0.812 | 0.748 |\n| CD+Reg |  0.416 | 0.348 | 0.806 | 0.744 |\n| CD+EO |  0.420 | 0.321 | 0.803 | 0.740 |\n| CD+DP |  0.414 | 0.331 | 0.811 | 0.748 |\n| CMCD | **0.413** | **0.316** | **0.814** | **0.749** |\n\n\nTable 2. The fairness result of CMCD and baselines on NCDM (the lower, the better)\n\n| Baselines  | GF(RMSE) | GF(MAE) | GF(AUC) | GF(ACC) |\n| -------- | ------- | ------- | ------- | ------- | \n| origin  | 0.034 | 0.070 | 0.032 | 0.054 |\n| CD+Reg |  0.036 | 0.066 | 0.030 | 0.052 |\n| CD+EO |  0.040 | **0.054** | **0.027** | 0.057 |\n| CD+DP |  0.035 | 0.076 | 0.032 | **0.050** |\n| CMCD | **0.033** | 0.059 | 0.028 | **0.050** |\n\n\n> **W2: In the experimental section, I observed that the utility and fairness metrics are aligned. While I understand that fairness analysis is based on utility outcomes, this alignment could potentially mislead readers to some extent. Clarity on this relationship would enhance the understanding for the readers.**\n\nSorry for the confusion. In this paper, the calculation of fairness metrics is based on user-oriented group fairness [1], which asserts that a fair model should deliver an equal level of utility performance across different user groups. Therefore, we express the performance gaps between the two groups accordingly. To simplify, we directly use performance metrics as a representation. To avoid any confusion, we will change our metric names in the revision. For example, in the fairness table, we will replace RMSE, MAE, AUC, ACC with GF(RMSE), GF(MAE), GF(AUC), GF(ACC).\n\n[1] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented\n fairness in recommendation. In Proceedings of the Web Conference 2021. 624–632.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5RcFwkHTgi", "reviewer": "Reviewer_ujU9", "summary": "This paper focuses on fairness and accuracy issues in Cognitive Diagnosis. Unlike model-based methods, the approach in this paper tackles these issues from a data-driven perspective. Leveraging the unique monotonicity assumption in cognitive diagnosis, the authors propose a general monotonic data augmentation method that can be applied to all cognitive diagnosis models. This method comes with both theoretical guarantees and empirical validation.", "questions": "What is the relationship between the two proposed constraints for data augmentation? \nCan the data augmentation methods suggested be applied to other domains, thereby enhancing the generalizability of the proposed approach?", "limitations": "This paper discusses the privacy concerns related to user data in its limitations section and suggests federated learning as a potential future research direction. Given this, is it feasible to integrate the proposed method in this paper with federated learning?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper investigates a significant issue in the field of intelligent education, the fairness and accuracy problem in cognitive diagnosis, which is utilized in high-stakes examinations, holds considerable research value.\nThe data augmentation method proposed in this paper integrates educational domain knowledge - the monotonicity assumption, which is intuitive and rational. Moreover, the authors provide theoretical guarantees for the accuracy and convergence speed of the proposed method.\nFurthermore, the paper has released the corresponding code as open source and conducted extensive experiments to validate the proposed method.", "weaknesses": "This paper proposes two constraints for data augmentation, which are actually quite similar in nature. I am curious about the relationship between these two augmentation methods. The authors need to engage in further discussion and analysis on this aspect.\nBy leveraging the unique monotonicity assumption in the field of cognitive diagnosis, this paper applies data augmentation techniques that have shown promising results in both accuracy and fairness. Cognitive diagnosis falls under the subfield of user modeling. Is it possible to extend these methods to other user modeling tasks? How do these methods differ from data augmentation techniques used in other user modeling tasks? I recommend that the authors delve into further discussions on these points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: The relationship between the two proposed constraints for data augmentation**\n\nA1: Thank you for your explanation. The constraints of these two data augmentations reflect the two states in the monotonicity assumption: when a student answers a question correctly, their ability is assumed to be higher than when they answer the same question incorrectly (Hypothesis 1), and when a student answers a question incorrectly, their ability is assumed to be lower than when they answer the same question correctly (Hypothesis 2). These two constraints complement each other and work synergistically to better address the issue of data sparsity, thereby achieving fairer and more accurate cognitive diagnostics. To validate this point, we conducted corresponding experiments. Specifically, we performed ablation experiments on the Math dataset where we systematically removed the corresponding hypothesis strategies. The results are presented in Table 1 and Table 2.\n\nFrom the tables, we observe that both constraints corresponding to the hypotheses can enhance both fairness and accuracy. This indicates that both constraints, which align with the monotonicity assumption, can facilitate more accurate and fair cognitive diagnostics. Importantly, when both constraints are applied (i.e., CMCD), better results are achieved compared to applying only one constraint. This suggests a synergistic effect between the two constraints, where they enhance each other's performance.\n\n\nTable 1. The accuracy result of CMCD on IRT  (the higher, the better)\n\n| Baselines  | ACC | \n| -------- | ------- | \n| CMCD  | 0.772 | \n| CMCD w/o Hypothesis 1 |  0.769 |\n| CMCD w/o Hypothesis 2 |  0.770 |\n| CMCD w/o Hypothesis 1 and 2 |  0.752 |\n\n\nTable 2. The fairness result of CMCD on IRT  (the lower, the better)\n\n| Baselines  | GF(ACC) | \n| -------- | ------- | \n| CMCD  | 0.034 | \n| CMCD w/o Hypothesis 1 |  0.036 |\n| CMCD w/o Hypothesis 2 |  0.038 |\n| CMCD w/o Hypothesis 1 and 2 |  0.058 |\n\n> **Q2: How do these methods differ from data augmentation techniques used in other user modeling tasks?**\n\n\nA2: Sorry for the confusion. The differentiation of our CMCD approach from standard data augmentation techniques is primarily manifested in two key aspects:\n\n* Monotonicity Assumption: The Monotonicity Assumption stands as a fundamental theoretical cornerstone in the realm of Cognitive Diagnostics (CD). It specifically asserts that a student's proficiency demonstrates a monotonic correlation with the likelihood of providing a correct response to an exercise. Building upon this premise, we have introduced two data augmentation constraints, with experimental results validating the efficacy of our approach.\n* Theoretical Guarantees: We have provided theoretical assurances regarding the accuracy and convergence of the proposed data augmentation strategies. Detailed information on these theoretical guarantees can be found in section 5.2 of our work.\n \nIn terms of practical application, we have also adapted some traditional data augmentation methods to the domain of cognitive diagnostics (specific baseline details can be found in A.3). Comparative analyses between these traditional methods and our CMCD approach have revealed superior performance of CMCD in both accuracy and fairness. Furthermore, our method demonstrates accelerated convergence rates (specific details can be found in section 6.2, RQ1, RQ2). We intend to emphasize these pivotal points in the revision.\n\n> **Q3: Is it possible to extend these methods to other user modeling tasks?**\n\nA3: Thank you for your thoughtful question. The core contribution of our paper is reflected in two aspects. Firstly, we conducted an analysis from a data perspective and identified that data sparsity could lead to unfairness and inaccuracy. We proposed achieving more accurate and fair cognitive diagnostics through the lens of data augmentation. Secondly, based on the unique monotonicity assumption in the field of cognitive diagnostics, we introduced two data augmentation constraints and provided theoretical guarantees.\n\nRegarding the first point, drawing from our past experiences, we observed that the data distributions in other user modeling tasks, such as recommendation systems, and the field of cognitive diagnostics generally exhibit similar trends. This suggests that the approach of addressing data sparsity to enhance accuracy and fairness in user modeling can be transferred. However, in tackling the challenge of data sparsity, our paper uniquely utilized the cognitive diagnostics domain's distinctive monotonicity assumption to augment data. By integrating this with the cognitive diagnostics paradigm, we provided corresponding theoretical guarantees. This method cannot be directly extended. In the future, we will explore how to leverage the specific characteristics of user modeling tasks to effectively address data sparsity issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAG08kdcXz", "reviewer": "Reviewer_xrHo", "summary": "The paper \"Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation\" introduces a framework called CMCD to address data sparsity in cognitive diagnosis, which affects accuracy and fairness. The authors integrate the monotonicity assumption to ensure data augmentation remains interpretable. They provide theoretical guarantees for accuracy and convergence speed and validate their method with extensive experiments on real-world datasets, showing improved performance in both accuracy and fairness. However, the paper has notable methodological flaws, limited novelty, and practical application challenges, making it less relevant for NeurIPS.", "questions": "1. Validation on Diverse Datasets: Have you considered validating your model on more diverse and larger datasets to enhance the robustness and generalizability of your findings?\n\n2. Practical Implementation: Can you provide more details on the practical challenges and solutions for implementing CMCD in real-world educational environments?\n\n3. Theoretical Guarantees: How do you plan to empirically demonstrate the theoretical guarantees of accuracy and convergence speed provided in the paper?\n\n4. Ethical Considerations: What measures have you considered to address data privacy and informed consent issues in future studies?", "limitations": "The paper lacks a detailed discussion on the potential negative societal impact, such as bias in cognitive diagnosis.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel framework, CMCD, addressing the critical issue of data sparsity in cognitive diagnosis by integrating the monotonicity assumption. The approach maintains interpretability while enhancing accuracy and fairness, as demonstrated through theoretical analysis and experiments. The use of real-world datasets and extensive experiments validates the efficacy of CMCD across various cognitive diagnosis models, providing a robust solution to a common problem in intelligent education systems.", "weaknesses": "The paper has significant methodological flaws, including the reliance on small, homogeneous datasets, limiting the generalizability of the findings. The theoretical guarantees for accuracy and convergence speed are not convincingly demonstrated, and the practical implementation of the proposed framework is inadequately discussed. Additionally, the novelty of the contribution is limited, with the method being an incremental improvement over existing approaches rather than a groundbreaking innovation. The lack of detailed ethical considerations and practical deployment strategies further diminishes the paper's overall impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Validation on Diverse Datasets: Have you considered validating your model on more diverse and larger datasets to enhance the robustness and generalizability of your findings?**\n\nA1: Thank you for your suggestion. Firstly, we would like to clarify that the two datasets used in this paper are widely utilized in the field of cognitive diagnostics and are highly representative. They have been collected from real online learning scenarios and are extensively referenced in many papers, such as [1][2][3][4].\n\nFurthermore, to address your concern, we have validated the effectiveness and robustness of our model on additional datasets. Specifically, we conducted new experiments on the ASSISTments2012 dataset, which was collected from ASSISTments, an online tutoring system in the United States between 2012 and 2013. The information regarding ASSISTments2012 is presented in Table 1:\n\nTable 1. The statics of ASSISTments2012 dataset.\n\n| Students  | Exercises | Response logs |\n| -------- | ------- | ------- | \n| 46,674  | 179,999 | 6,123,270 |\n\nThe results of CMCD on ASSISTments2012 are shown in Table 2 and Table 3. From these results, we observe that CMCD effectively enhances the accuracy and fairness of cognitive diagnostic models, demonstrating the effectiveness of our model.\n\nTable 2. The accuracy result of CMCD and baselines (the higher, the better)\n| Baselines  | AUC | ACC |\n| -------- | ------- | ------- | \n| IRT  |  0.771 | 0.742 |\n| CMCD-IRT |  0.780 | 0.748|\n\nTable 3. The fairness result of CMCD and baselines (the lower, the better)\n\n| Baselines  | GF(AUC) | GF(ACC) |\n| -------- | ------- | ------- | \n| IRT  |  0.0516 | 0.0149 |\n| CMCD-IRT |  0.0386 | 0.0147|\n\n\n[1] Wang F, Liu Q, Chen E, et al. Neural cognitive diagnosis for intelligent education systems[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 6153-6161.\n\n[2] Wang F, Gao W, Liu Q, et al. A Survey of Models for Cognitive Diagnosis: New Developments and Future Directions[J]. arXiv preprint arXiv:2407.05458, 2024.\n\n[3] Liu S, Shen J, Qian H, et al. Inductive Cognitive Diagnosis for Fast Student Learning in Web-Based Intelligent Education Systems[C]//Proceedings of the ACM on Web Conference 2024. 2024: 4260-4271.\n\n[4] Yu X, Qin C, Shen D, et al. Rdgt: enhancing group cognitive diagnosis with relation-guided dual-side graph transformer[J]. IEEE Transactions on Knowledge and Data Engineering, 2024.\n\n> **Q2: Practical Implementation: Can you provide more details on the practical challenges and solutions for implementing CMCD in real-world educational environments?**\n\nA2: Thank you for your feedback. In real-world educational settings, it is crucial to have access to users' response records, which may sometimes be restricted due to privacy considerations. \n**It's important to note that for all cognitive diagnostic tasks currently, gathering users' response records is fundamental to research in the entire cognitive diagnostic field, the privacy issue is not within the scope of our current discussion in this paper**. To address this privacy issue, we intend to integrate our approach with federated learning and differential privacy. This limitation has been discussed in Section 7, Conclusion and Discussion, of the paper.\n\n\n> **Q3: Theoretical Guarantees: How do you plan to empirically demonstrate the theoretical guarantees of accuracy and convergence speed provided in the paper?**\n\nA3: Sorry for the confusion. In fact, besides theoretically validating the accuracy and convergence speed of our method, **we have empirically verified these aspects as well**.\n\n* Regarding accuracy, we conducted experiments on different datasets and different cognitive diagnostic backbones (IRT, MIRT, NCDM), where we found that our method enhances the accuracy of the baseline on real datasets. These results are detailed in Section 6.2 RQ1. Furthermore, we performed ablation experiments (Section 6.2 RQ2) and discovered the effectiveness of the data augmentation strategy in improving accuracy within CMCD.\n\n* Regarding convergence speed, we conducted experiments on real datasets. Specifically, we compared the convergence speeds (the epoch at which early stopping occurs) of different baselines on various backbones. The experiments revealed that we can achieve faster convergence significantly, as demonstrated in Section 6.2 RQ3.\n\n> **Q4: Ethical Considerations: What measures have you considered to address data privacy and informed consent issues in future studies?**\n\nA4: Thank you for your thoughtful suggestion. We plan to integrate our method with the following three perspectives.\n\n* Encrypted Communication: Ensure the use of secure encrypted channels during data transmission in the CD. This helps prevent data from being intercepted or tampered with during the transmission process, safeguarding the privacy and integrity of students' response records.\n\n* Differential Privacy: By incorporating differential privacy techniques, we can introduce controlled noise to the data during the model training process. This aids in protecting the privacy of individual student data, preventing malicious users from inferring specific student information through the model's outputs.\n\n* Federated Learning: By employing federated learning methods, we can distribute the model training process across multiple local devices rather than centralizing it on a single server. In federated learning, only encrypted parameters are shared during the model update phase, not the raw data. This approach ensures that student response records remain local and decentralized, safeguarding student privacy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers' time and efforts in reviewing our paper.  We extend our gratitude to each of them for offering constructive and valuable feedback, which we will use to enhance this work. Meanwhile, we are encouraged by the positive comments from reviewers, including:

* **Motivation:** "hold significant societal value and impact the fairness of education" (Reviewer A4mA), "a highly significant societal concern" (Reviewer A4mA), "a significant issue in the field of intelligent education"(Reviewer ujU9), "holds considerable research value" (Reviewer ujU9), "addressing the critical issue of data sparsity in cognitive diagnosis" (Reviewer xrHo)

* **Theoretical Contribution:** "the authors theoretically prove" (Reviewer A4mA), "The paper undergoes validation from two key standpoints - experimental and theoretical." (Reviewer A4mA), "provide theoretical guarantees" (Reviewer ujU9), "as demonstrated through theoretical analysis and experiments" (Reviewer xrHo)

* **Method:** "effectively mitigates the challenge of data sparsity" (Reviewer A4mA), " the method is indeed compelling" (Reviewer A4mA), "intuitive and rational" (Reviewer ujU9), " maintains interpretability while enhancing accuracy and fairness"(Reviewer xrHo)

* **Experimental Results:** "accompanied by thorough validation" (Reviewer A4mA), "The validation methods are thorough and comprehensive, ensuring a robust evaluation" (Reviewer A4mA), "released the corresponding code as open source and conducted extensive experiments"(Reviewer ujU9), "The use of real-world datasets and extensive experiments validates the efficacy of CMCD" (Reviewer xrHo), "providing a robust solution to a common problem" (Reviewer xrHo).

We have provided specific responses to each reviewer. We hope our responses can clarify all your confusion and alleviate all concerns. We thank all reviewers again. Looking forward to your reply!

Meta-review: The submitted paper was reviewed by 3 knowledgeable reviewers all of whom recommended acceptance of the paper. The reviewers appreciate the importance of the considered problem for intelligent education, leveraging the monotonicity assumption for improved data augmentation, and the theoretical as well as extensive experimental evaluation demonstrating the benefits of the proposed approach. Thus, in line with the reviewers' unanimous recommendation, I recommend acceptance of the paper. 
The authors are asked to prepare the camera-ready version by considering the reviewers' comments and in particular include the clarifications made during the discussion-phase as well as new results in the final paper.
Decision: Accept (poster)