Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation

Authors: zheng  zhang
wei  song
qi  liu
qingyang  mao
yiyan  wang
weibo  gao
zhenya  huang
shijin  wang
enhong  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 09:56:01

Abstract: Intelligent education stands as a prominent application of machine learning. Within this domain, cognitive diagnosis (CD) is a key research focus that aims to diagnose students' proficiency levels in specific knowledge concepts. As a crucial task within the field of education, cognitive diagnosis encompasses two fundamental requirements: accuracy and fairness. Existing studies have achieved significant success by primarily utilizing observed historical logs of student-exercise interactions. However, real-world scenarios often present a challenge, where a substantial number of students engage with a limited number of exercises. This data sparsity issue can lead to both inaccurate and unfair diagnoses. To this end, we introduce a monotonic data augmentation framework, CMCD, to tackle the data sparsity issue and thereby achieve accurate and fair CD results. Specifically, CMCD integrates the monotonicity assumption, a fundamental educational principle in CD, to establish two constraints for data augmentation. These constraints are general and can be applied to the majority of CD backbones. Furthermore, we provide theoretical analysis to guarantee the accuracy and convergence speed of CMCD. Finally, extensive experiments on real-world datasets showcase the efficacy of our framework in addressing the data sparsity issue with accurate and fair CD results.

Main paper: 
 introduction: 
 Intelligent education is a significant domain within machine learning that focuses on exploring students' learning patterns. In the past decades, research in this interdisciplinary field has garnered substantial attention from scholars across various disciplines [29, 26, 10, 45] , including education, machine learning, and psychology. Within intelligent education, cognitive diagnosis (CD) stands out as a crucial research area that aims to measure students' proficiency levels in specific knowledge domains, such as Geometry [39, 15, 7, 21] . For instance, as illustrated in Figure 1 , students practice some exercises e 1 , e 2 , e 3 and obtain associated responses indicating correctness, which are then utilized to perform CD to infer the students' mastery levels of the corresponding concepts. With a comprehensive understanding of students' abilities, CD can be applied to various applications, including student assessment [4, 56] and educational recommendation systems [19] .As a crucial task within the field of education, CD encompasses two fundamental requirements: 1) Accurate Diagnoses: It is imperative to precisely evaluate students' mastery of knowledge to facilitate Figure 1 : An illustrative example of cognitive diagnosis. various downstream applications. For instance, this accuracy enables teachers to deliver personalized instruction, tailored to the specific needs of each individual student. 2) Fair Diagnoses: CD plays a pivotal role in high-stakes examinations like the GRE [30] , significantly influencing individuals' developmental opportunities. Therefore, ensuring fairness in CD is of paramount importance. In this context, fairness means different student groups divided by sensitive attributes (e.g., gender, race) should be treated fairly and equally.To accomplish these dual objectives, prior research has made notable strides by harnessing historical logs that document students' interactions with exercises. However, in practical scenarios, many students can only interact with a limited number of exercises within the vast exercise pool, leading to the data sparsity issue that may cause both inaccurate and unfair diagnoses. A detailed analysis is available in Section 4. Currently, several approaches have been proposed to mitigate this issue from a model-centric perspective, which focus on developing more complex architectures to address the challenges posed by data scarcity [14, 7] . However, the incorporation of additional architectures often compromises the model's interpretability, rendering it unsuitable for high-impact educational environments. For instance, in high-stakes exams like the GRE, only IRT [29] , a classical CD model, has been applied due to their statistical superiority [30] . While other model-based models, despite mitigating data sparsity to some extent, remain unused because of their lack of interpretability.In contrast to these model-based approaches, in this paper, we tackle the data sparsity issue from the perspective of data augmentation without altering the model architecture. However, we have encountered the following challenges: 1) Monotonicity Assumption. The Monotonicity Assumption is a fundamental theoretical premise in the field of CD. Specifically, it posits that a student's proficiency exhibits a monotonic relationship with the probability of providing a correct response to an exercise. Using Figure 1 as an example, Sue, who correctly responds to exercise e 1 , is considered to possess a higher proficiency level in the related concept C (i.e., Absolute Value) compared to Bob, who provides an incorrect response. This ensures the interpretability of CD, contributing to its widespread acceptance and application. Therefore, maintaining the model's monotonicity assumption during the data augmentation process is crucial. 2) Theoretical Guarantees. As data augmentation emerges as a new paradigm in the CD domain, it becomes imperative to establish corresponding theoretical guarantees for this novel approach. To address these challenges, we propose a monotonic data augmentation CD framework, CMCD, complemented by theoretical guarantees. Specifically, we integrate two data augmentation constraints confronting the monotonicity assumption. For each student, we generate fake students by reversing one of his responses while keeping other records unchanged to train CD models. As illustrated in Figure 2 , Bob's wrong answer to e 1 is reversed to generate C-1 while his right answer to e 6 is reversed to generate C-2. Following the monotonicity assumption, we assume that Bob's proficiency the corresponding knowledge concept is lower than that of C-1 yet higher than that of C-2. Moreover, we provide theoretical analysis to guarantee CMCD's advantage in accuracy and convergence speed. Finally, we conduct extensive experiments on real-world datasets, demonstrating the effectiveness of our method across various CD models. Our key contributions can be summarized as follows:• Problem Emphasis. Our paper strongly emphasizes the connection between data sparsity and inaccurate and unfair diagnoses. To the best of our knowledge, we are the first to highlight the relationship between data sparsity and educational fairness.• Framework Design. From the perspective of data augmentation, we combine the monotonicity assumption in CD to propose a general framework, CMCD, which comes with theoretical guarantees of accuracy and convergence speed.• Experimental Evaluations. Extensive experiments on real-world datasets demonstrate the effectiveness of our approach.2 Related WorksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
cognitive diagnosis: 
 CD plays a crucial role in various real-world educational scenarios [48, 49] , including student assessment [36, 31] and educational recommender systems [24, 19] . Initially rooted in psychometrics, CD models such as Item Response Theory (IRT) [29] have been widely used, particularly in the context of GRE [30] . Later, Multidimensional Item Response Theory (MIRT) [35] was introduced to extend the single-trait features in IRT to multidimensions. While these models were effective and provided interpretable diagnostic results based on psychometric theories, they heavily relied on handcrafted interaction functions and could only leverage users' numerical response records, which were often affected by data sparsity [25] . With the advancements in machine learning, researchers began developing CD models from a machine-learning perspective. One notable model in this regard is NCDM [39] , which employed neural networks to learn the interactions between students and exercises, yielding satisfactory results. However, the majority of existing works have primarily focused on designing more complex architectures to alleviate data sparsity. In contrast to traditional model-based approaches, this paper embraces a data-centric perspective and introduces CMCD, which comes with strong theoretical guarantees and is adaptable for integration with all cognitive diagnosis models.Data augmentation is a classical technique to address the problem of insufficient training data in the machine learning community [42] . Over the past few years, this concept has achieved significant success in the fields of neural language processing (NLP) [13, 57, 32] and computer vision (CV) [38, 2, 6] . Later, this idea was applied to user modeling tasks, which necessitated the simulation of user-item interaction data. The work most similar to ours involves data augmentation of user-item interactions in recommendation systems. They typically answer questions such as: "What would happen if...?" Through these responses, they can generate additional virtual data, thereby alleviating the issue of data sparsity. For instance, Wang et al. [43] addressed the issue of data sparsity in the sequential recommendation and provided an answer to the question: "What would a user like to buy if their previously purchased items had been different?". Similarly, Xiong et al. [44] focused on review-based recommendation and answered the question: "What would be the user's decision if their feature-level preference had been different?" Although these works have shown promising results, when applied to cognitive diagnosis, they overlook the monotonicity assumption, a classic theory in education. Additionally, they lack corresponding theoretical guarantees. In this paper, we aim to bridge these gaps to alleviate data sparsity in cognitive diagnosis more effectively. 
cognitive diagnosis and models: 
 In this subsection, we formally define the CD problem. Assume there are m students, n exercises, which are denoted as S = {s i } m i=1 , E = {e j } n j=1 . Assume each student s i has records on exercises indexed by Q i ⊂ {1, 2, . . . , n}, the response logs R i of student s i are a set of triplets (s i , e j , y i,j ), where j ∈ Q i , y i,j ∈ {0, 1} is the score obtained by student s i on exercise e j . Given response logs R i of student s i , the goal of CD is to mine the proficiency θ i . Cognitive Diagnosis Models (CDMs) are developed to depict student's proficiency level on specific knowledge concepts based on her responses to several test items. To do this, an objective function is used to train CDMs on the student performance prediction task. More concretely, CDMs are expected to minimize the difference of the predicted probability p j (θ i ) of a student s i giving the right response to the exercise e j between the true response y i,j = p j (θ i ). In this paper, following the classical works [14] , we adopt the cross-entropy loss, the goal is,EQUATIONIn the past decades, lots of CDMs have been proposed such as IRT [29, 11] , MIRT. Generally, CDMs contain two parts: (1) the representations of trait features and (2) the interaction function. For example, IRT models each student s i as a proficiency variable θ i , each exercise as a discriminating factor α j and a difficulty factor β j , and a logistic function is used to forecast the likelihood that student s i will answer exercise e j correctly based on a logistic function2 , i.e., p j (θ i ) = 1/(1 + e αj (θi-βj ) ).With the advancement of machine learning technologies [20, 9] , such as large language models [55] , which have found widespread applications in many important scenarios [51, 52] , trustworthy AI has become a very important topic [18, 27, 28, 34, 33, 17] . Among these concerns, the issue of fairness has garnered widespread attention [22, 37, 8, 41, 53, 47, 40] . Given that cognitive diagnosis holds a fundamental position in the field of education and is extensively applied in high-stakes exams such as the GRE [30] , which significantly shapes individuals' developmental opportunities, ensuring fairness in cognitive diagnosis becomes of utmost importance. In this paper, we follow the widely accepted group fairness definition proposed by Li et al. [23] , which states that a fair model should provide the same level of utility performance for different user groups.For analytical convenience, we focus on the case where the sensitive attribute is binary, which can be easily extended to multiple values. The student group can be divided into two groups based on their sensitive attributes, denoted as G 0 and G 1 , whereU = G 0 ∪ G 1 , G 0 ∩ G 1 = ∅.We denote the number of samples in each group as m 0 and m 1 . Inspired by previous works [54, 50, 23] , in this paper, fairness in cognitive diagnosis is defined as follows, where a lower value indicates better fairness performance: EQUATIONwhere M represents a metric for evaluating utility performance, such as MAE or MSE score, and M(s i ) denotes the utility performance for student s i .After introducing the fairness definition in cognitive diagnosis, our objective extends beyond accurately identifying students' proficiency. We should also strive to meet fairness requirements, with the goal of minimizing GF . 
data sparsity analysis: 
 In this section, we explore data sparsity in cognitive diagnosis through the real-world dataset, Math (detailed information will be introduced in the Experiment). Firstly, we conduct an analysis of data sparsity within the dataset, partitioning it based on the number of responses per student. The statistical data is illustrated in Figure 3 (a). From the figure, it is evident that the majority of users have a low log count, highlighting the widespread presence of data sparsity. Following the methodology outlined in [46] , we classify records with fewer than 50 responses as the sparse group. Next, we validate whether data sparsity affects the accuracy and fairness of cognitive diagnosis.Inaccurate Result From the perspective of accuracy, we conduct a comparative analysis between the sparse and non-sparse groups, as illustrated in Figure 3 (b). The results indicate a notable decrease in performance for the sparse group across different backbones compared to the non-sparse group. This outcome strongly implies that data sparsity can result in inaccurate diagnostic results. Unfair Result From the perspective of fairness, we examine the performance of the two groups divided by sensitive attributes. In the Math dataset, the student group is divided into Area1 and Area2 by sensitive attribute region, with Area1 representing the disadvantaged group. The results are depicted in Figure 3 (c), unveiling significant variations in performance outcomes among different groups and indicating the presence of unfairness. Following this, we delve into the disparity in sparsity levels between the two groups, as shown in Figure 3(d) . The results highlight significantly divergent sparsity levels, notably with the disadvantaged groups exhibiting higher sparsity. This discrepancy is attributed to the likelihood that disadvantaged groups may have fewer opportunities to engage in relevant online learning platforms compared to their advantaged counterparts. This statistical outcome directly implies that data sparsity has a greater impact on the disadvantaged group (Area1). Based on this observation, we can conclusively state that data sparsity can lead to unfair diagnostic results.So far, we have validated that data sparsity in cognitive diagnosis can result in inaccurate and unfair diagnostic outcomes. Consequently, mitigating the issue of data sparsity in cognitive diagnosis is of paramount importance. In the following sections, we will demonstrate how to address data sparsity from the perspective of data augmentation.5 The Proposed Framework In this section, we propose a monotonic data augmentation framework, CMCD, as depicted in Figure 4 . It can address the issue of data sparsity and ensure accurate and fair diagnostic results. Firstly, we present the constraints of our proposed monotonic data augmentation approach. Subsequently, we offer theoretical assurances regarding the effectiveness of CMCD.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully disclose all the information needed to reproduce the main experimental results in section 6Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
monotonic data augmentation: 
 In CD, there is a fundamental assumption called the monotonicity assumption [1] , which plays a vital role in ensuring interpretability. In this paper, we introduce a novel approach that integrates the monotonicity assumption with data augmentation. This combined technique allows us to generate more realistic student response data that conforms to educational principles. As a result, we can effectively address the challenge of data sparsity. The monotonicity assumption is defined as follows:Monotonicity Assumption The probability of correct response to the exercise is monotonically increasing at any dimension of the student's knowledge proficiency.Building upon the monotonicity assumption, suppose two students s i and s i ′ have answered exactly the same exercises, i.e., Q i = Q i ′ ≜ Q, we propose two specific hypotheses as follows:Hypothesis 1: If s i answered exercise e j correctly, while s i ′ answered e j incorrectly, and answers for other exercises are the same, then the proficiency level of s i should not be lower than s i ′ . Formally, for only onej ∈ Q, r i,j = 1, r i ′ ,j = 0, while for k ∈ Q, k ̸ = j, r i,k = r i ′ ,k , we have θ i ≥ θ i ′ .Hypothesis 2: If s i answered exercise e j incorrectly, while s i ′ answered e j correctly, and answers for other exercises are the same, then the proficiency level of s i should not be greater than s i ′ . Formally, for only one j ∈ Q, r i,j = 0,r i ′ ,j = 1, while for k ∈ Q, k ̸ = j, r i,k = r i ′ ,k , we have θ i ≤ θ i ′ .To leverage these hypotheses and enhance the training data to mitigate the data sparsity issue, we first need to construct suitable interaction records for students. Specifically, for student s i , we randomly select one of his answered exercise e j ∈ E i and flip the corresponding score on the interaction record (changing 1 to 0 or 0 to 1), and keep other records unchanged. By doing so, we generate the records for s i ′ . Assume the proficiency of s i ′ is θ i,j , the fitting function for student s i ′ can be expressed as:EQUATIONNext, we consider the partial order relationship between s i , s i ′ . Two scenarios need to be considered.According to Hypothesis 1, when θ i ≥ θ i,j should hold but θ i < θ i,j in fact, we need to add a regularization term (θ i,jθ i ) 2 to impose the constraint. Similarly, as for Hypothesis 2, when θ i ≤ θ i,j should hold but θ i > θ i,j in fact, we need to add a regularization term (θ iθ i,j ) 2 as penalty.As a result, we establish two regularization terms:EQUATION)where 1 is the indicator function. Then we summarize the final regularization term as:EQUATIONMoreover, to enhance the monotonicity assumption, for each student s i , we consider generating multiple fake students and comparing them with s i by randomly sample a subset C i ⊂ Q i . For each exercise in C i , we perform a flipping operation on the corresponding score while keeping the other records unchanged, resulting in |C i | versions of student s i .As a result, the final loss function in CMCD is:EQUATIONwhere c represents the strength of constraint.In this section, we provide theoretical guarantees of CMCD's effectiveness in terms of accuracy and convergence speed. For theoretical analysis, we will focus on the most classic cognitive diagnostic model, IRT (introduced in Section 3.2), which serves as the foundation for many other cognitive diagnostic models and has been widely implemented in GRE [30] . Please note that the following theoretical analysis can be easily extended to other CDMs.Firstly, we analyze the accuracy of CMCD. Given that the core objective of cognitive diagnostic tasks is to estimate the true proficiency θ i of student s i , to assess the accuracy of our CMCD, we theoretically analyze the Maximum Likelihood Estimation (MLE) of θ i through the optimization of L A (Eq. ( 6)). Let Θ = (θ 1 , θ 2 , • • • , θ m ), we assert that CMCD has accurate estimation stated as:Proposition 5.1 (Accuracy). CMCD can accurately estimate the student proficiency level Θ.To validate the above proposition, we prove that the MLE under CMCD is a consistent estimation:Theorem 5.2. Suppose a student s i with ability θ i has records R i on exercises indexed by Q i , andEQUATIONDetailed proofs for Theorem 5.2 can be found in Appendix A.1. The consistency estimation ensures the accuracy of CMCD. Next, we explore the superiority of CMCD in convergence speed compared to traditional CD models (the loss function L is introduced in Eq.( 1)).Proposition 5.3 (Convergence Speed). In terms of the estimation of proficiency level Θ, CMCD has a faster convergence speed than the original approach.To validate the above proposition, we first notice that L and L A are both convex. Then, in the specific zone that violates the monotonicity assumption, we prove that L A are strong-convex in terms of Θ.Finally we can analyze and compare the convergence speed in optimizations of L and L A .Theorem 5.4. L and L A are convex in terms of Θ. In addition, L A is strong-convex whenEQUATION) and, meanwhile, we can also conclude Detailed proofs for Theorem 5.4 are provided in Appendices A.2. When performing the gradient descent algorithm to optimize the loss function, it is widely known that strong-convex functions converge faster than convex functions [3] . Overall, this represents the primary distinction between L A and L. By examining the frequency of occurrences of this expression during the optimization process, we can analyze the convergence speeds of the two functions. The more frequent the occurrences, the faster the convergence of L A . Simultaneously, this expression also indicates a violation of our hypotheses, necessitating the incorporation of regularization. To validate the superior convergence speed, we calculated instances of hypothesis violation across different datasets, with the results illustrated in Figure 5 . It reveals a prevalent occurrence of strong convexity in L A across various datasets and backbones. Therefore, by optimizing L A , CMCD converges faster.∇ 2 L A > ∇ 2 L when Θ ∈ D.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification:The paper provides open access to the code in section 6Guidelines:• The answer NA means that paper does not include experiments requiring code. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The societal impacts are introduced in section 7 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
experiments: 
 In this section, we first introduce the dataset and experimental setup. Then, we conduct extensive experiments on real-world datasets to answer the following questions:• RQ1: Does CMCD learn fair and accurate cognitive diagnosis results? • RQ2: How does the constraint of data augmentation impact CMCD? • RQ3: Does CMCD have a faster convergence speed compared to other baseline models?• RQ4: How do different hyper-parameter settings (i.e. c, |C i |) affect the performance?The code is released at https://github.com/Mercidaiha/CMCD.Datasets In this paper, we conduct experiments in two diagnostic datasets, namely Math and AS-SIST. The Math dataset was collected from an online educational system that provides homework, and evaluation for students. It collected records of junior high school students on mathematical exercises. In this dataset, we take the region as the sensitive attribute. The ASSIST dataset (ASSISTments 2009-2010 "skill builder") is an open dataset collected by the ASSISTments online tutoring systems [12] , which includes student response logs and knowledge concepts 3 . Given that the ASSIST dataset does not provide sensitive attribute information, we follow [23] and categorize users into two groups based on whether the number of their responses exceeds 50. Regarding the dataset division, we allocate 80% of each student's response log for training and the remaining 20% for testing. The overview of basic statistics can be found in Appendix Table 2 . Evaluation. In this paper, we evaluate the results of user modeling by predicting scores. This evaluation can be divided into two aspects: utility and fairness evaluation. For utility evaluation, following previous works [39, 14] , we adopt different metrics from the perspectives of both regression (MAE, RMSE) and classification (AUC, ACC). For fairness evaluation, we adopt the Definition introduced in Section 3.2. For the M in Eq. ( 2), we adopt MAE, RMSE, AUC, ACC (denoted as ∆MAE, ∆RMSE, ∆AUC, ∆ACC in the experiment).Baseline approaches. CMCD is a versatile framework applicable to various CD models. To validate its generality, we employed CMCD in three classical CD backbones, IRT [29] , MIRT [5] , NCDM [39] . For the baselines, we compare our methods with two categories of methods. Firstly, we compare our methods with the data augmentation baseline. To the best of our knowledge, there are currently no data augmentation methods specifically applied to diagnostics. Therefore, following the approach [44] , we constructed the following baselines: CF-IRT, CF-MIRT, and CF-NCDM.Secondly, as our model enhances fairness, we compare it with several classical fairness baselines, such as CD+GF [23] , CD+EO [47] , CD+DP [47] . Further details regarding these baselines and the implementation of our methods can be found in Appendix A.3.In this section, we discuss whether CMCD framework can alleviate the issue of data sparsity, thereby enhancing both the accuracy and fairness of the model. Specifically, we conducted comparative experiments with baselines on the Assistment and Math datasets, and the results of Math datasets results are presented in Table 1 , the results of Assistment datasets can be found in Appendix A. 4 .From these results, we can draw the following findings:(1) From the perspective of utility, we observed a significant enhancement in CMCD across different datasets and various backbones. This underscores the effectiveness of our approach. Furthermore, we noted that, compared to backbone models like NCDM that rely on neural networks, CMCD demonstrates more pronounced improvements on traditional backbones (IRT). We attribute this to the fact that models dependent on neural network backbones, such as NCDM, exhibit superior data fitting capabilities, thereby alleviating issues related to data sparsity. Consequently, the gains achieved by CMCD on NCDM are less conspicuous. In contrast, traditional IRT models are more susceptible to the impact of data sparsity, making CMCD's influence more substantial in such cases.(2) From the perspective of fairness, we observed that CMCD can, to a certain extent, alleviate unfair phenomena in cognitive diagnostic models. Particularly noteworthy is its performance on the Math dataset, where our model achieved the best fairness outcomes across different backbones (IRT, MIRT). This underscores that CMCD, while mitigating data sparsity issues, concurrently promotes fairness in model outcomes.(3) From the perspective of the trade-off between fairness and utility, our model consistently surpasses the baselines with data augmentation in both performance and fairness outcomes. While compared to fairness-aware baselines, our model may not always outperform dedicated fairness-enhancing methods on specific datasets, it is important to note that these baselines often come at the expense of utility. In contrast, CMCD not only enhances fairness but also improves performance. This highlights that CMCD achieves the optimal trade-off between fairness and utility. To address the issue of data sparsity, we introduce two hypotheses for data augmentation. In this section, we validate the effectiveness of these two hypotheses. Specifically, we conduct ablation experiments on the Math dataset, where we systematically removed the corresponding hypothesis strategies. The results are illustrated in Figure 6 . It is evident that after removing different hypothesis strategies, both Fairness and Utility performances exhibited varying degrees of decline, providing evidence for the efficacy of each strategy. Additionally, we observed that these two hypotheses yield different effects on various backbones. In comparison to NCDM, a neural network-dependent model, hypotheses perform better on traditional models like IRT and MIRT. We attribute this to the powerful data-fitting capability of NCDM, which alleviates issues related to data sparsity. In contrast, models such as IRT and MIRT, constrained by data sparsity, exhibit limited capabilities. RQ3. In Section 5.2, we theoretically demonstrated that CMCD exhibits a faster convergence rate. In this section, we empirically validate the superiority of CMCD in terms of convergence speed. Specifically, we compared the convergence speeds (the epoch at which early stopping occurs) of different baselines on various backbones using the Math dataset, as depicted in Figure 7 . It is observed that in the early stages, CMCD shows minimal differences compared to other baselines. We attribute this to the fact that, initially, the effectiveness of the backbone models is not optimal, resulting in less precise enhancement of the monotonicity assumption. However, as the number of epochs increases, CMCD significantly outpaces other baselines, demonstrating a faster convergence.In CMCD, two hyperparameters control the effectiveness. Specifically, c controls the strength of the constraints, and |C i | regulates the number of generated students. In this section, we investigate the impact of adjusting these two hyperparameters on CMCD using the Math dataset. The specific results are presented in Figure 8 and Figure 9 . In Figure 8 , a consistent trend is observed across different backbones. Initially, with the strengthening of c, both fairness and utility show improvement, indicating the efficacy of our monotonicity enhancement. However, as c reaches a certain intensity, both fairness and utility experience a decline. We attribute this to the possibility that an excessively large c might interfere with the model's normal training, leading to a decrease in effectiveness. Moving to Figure 9 , diverse trends are identified across different backbones. Specifically, in the case of IRT, as the number of contrasts increases, both fairness and utility consistently improve. Conversely, for NCDM, as the quantity of contrasts grows, there is a continuous decline in utility. We attribute these observations to inherent differences in the nature of the models. IRT is more affected by data sparsity, so enhancing C i effectively mitigates data sparsity, yielding improved results. On the other hand, NCDM, benefiting from its neural network's fitting capacity, partially alleviates data sparsity issues. Consequently, an increase in contrasts may interfere with its intrinsic performance. 
conclusion and discussion: 
 In this paper, we conducted a focused study on addressing data sparsity in CD. Initially, we empirically validated that data sparsity leads to inaccurate and unfair diagnostic results. Subsequently, by integrating data augmentation and the monotonicity assumption, we introduced two constraints to alleviate the issue of data sparsity. Moreover, we provided theoretical guarantees regarding accuracy and convergence speed. The experiments on real-world datasets demonstrated the effectiveness of our approach in addressing the data sparsity issue, culminating in fair and accurate diagnosis results.CD is a highly significant task in intelligent education. The generated diagnostic results can be applied to various areas, e.g., personalized tutoring and intelligent question recommendations. This can reduce the burden on teachers and students, providing effective learning experiences for students. Our CMCD mitigates the unfairness and inaccuracy issues arising from data sparsity, thereby promoting educational equity to a certain extent. Simultaneously, we recognize that CD relies on students' response records that might be inaccessible due to privacy concerns. In the future, we will contemplate incorporating federated learning to enhance diagnostic services and ensures student privacy.when |Q i | ≥ n 0 holds, here α = max j α j . ConsideringEQUATIONwe can see when |Q i | ≥ n 0 , with probability less than δ,EQUATIONwhich means θi is a consistent estimation of θ i . Then, we denote θi,j = arg min θ l i,j (θ), and we haveEQUATION)If y i,j = 1, l ′ i,j ( θi ) = α j > 0 and it means θi is bigger than the zero point, i.e., θi > θi,j . If y i,j = 0, l ′ i,j ( θi ) = -α j < 0 and it means θi is smaller than the zero point, i.e., θi < θi,j . Overall, we can check that Ω(y i,j , θi , θi,j ) = 0.In this way, θi makes l i (θ) the smallest, θi,j makes l i,j (θ) the smallest, and with them the regularized terms are always 0. Therefore, such θi and θi,j are the minimum points of L A , and this fact results in θ * i = θi . Combining the conclusion that θi is a consistent estimation of θ i , we can conclude that θ * i is a consistent estimation of θ i .A.2 Proof of Theorem 5.4. Theorem A.2. L and L A are convex in terms of Θ. In addition, L A is strong-convex when (18) and, meanwhile, we can also conclude ∇ 2 L A > ∇ 2 L when Θ ∈ D.Θ ∈ D = {(θ 1 , θ 2 , . . . , θ m )|∀i ∈ [m], ∃j ∈ C i , y i,j 1 [θi<θi,j ] + (1 -y i,j )1 [θi>θi,j ] = 1},Proof. For the loss function L (Eq.( 1)), the mean negative log-likelihood function of student s i isEQUATIONWe can further validate thatEQUATION)For any i, j we have∂ 2 L ∂θ 2 i ≥ 0 and ∂ 2 L ∂θi∂θj = 0, hence ∇ 2 L ≥ 0 and L is convex in terms of Θ.In addition, we can see thatEQUATIONand ∂ 2 L ∂θi∂θj = 0 for all i, j. Hence ∇ 2 L A ≥ ∇ 2 L ≥ 0 and L A is also convex in terms of Θ. When Θ ∈ D, we can see thatEQUATION)Based on Eq.( 21), we can conclude that L A is strong-convex and ∇ 2 L A > ∇ 2 L when Θ ∈ D. In fact, we have ∇ 2 L ≥ 2c • min i { j∈Ci y i,j 1 [θi<θi,j ] + (1 -y i,j )1 [θi>θi,j ] } when Θ ∈ D. 
 
 Appendix: Baseline approaches. CMCD is a versatile framework applicable to various CD models. To validate its generality, we employed CMCD in three classical CD backbones, IRT [29] , MIRT [5] , NCDM [39] . The details of baselines are as follows:• IRT [29] is a cognitive diagnosis method that models the cognitive processes from students' exercise records with a logistic-like function. • MIRT [5] is a variant of the basic IRT model, which extends the unidimensional latent traits of students and exercises in IRT to multidimensional vectors. • NCDM [39] is a deep neural cognitive diagnosis framework that models the complex interaction from students' exercising records with a multilayer perceptron (MLP).CMCD can alleviate the issue of data sparsity in diagnostic models, enhancing both accuracy and fairness. To validate the effectiveness of our approach, we compare it with two categories of methods. The first category is data augmentation methods. To the best of our knowledge, there are currently no data augmentation methods specifically applied to diagnostics. Therefore, we adapt the data augmentation concept "What would ... if ...?" from traditional recommendation systems to the cognitive diagnostic field. Specifically, following the approach in [44] , we apply a causal rule: "What would the student's response be if they encountered a new question?" Additionally, considering the characteristics of CD domain, we make appropriate adaptations by using classic IRT, MIRT, and NCDM models to predict responses to new questions. Ultimately, we extend the data augmentation baseline to CF-IRT, CF-MIRT, and CF-NCDM. The second category is fairness-aware methods. Since our model enhances fairness, we compare it with several classical fairness baselines, CD+GF [23] , CD+EO [47] , CD+DP [47] . The details of baselines are as follows:• CD+GF: a well-known fairness improvement strategy that considers the fairness metric as a regularization for the loss and has been used in prior fairness works [47, 23] . In our work, we use Eq. ( 4) as a regularization for Eq. ( 2); • CD+EO: a method regards Equal opportunity (EO) as a regularization [16] .• CD+DP: a method regards Demographic Parity (DP) as a regularization [16] .Implementation detail. In terms of model parameter configuration, NCDM adheres to the settings outlined in [39] . The fully connected layers have dimensions of 512, 256, and 1. The sigmoid function serves as the activation function for all layers. MIRT's dimension of student proficiency parameters matches those in NCDM. For all models, we set the learning rate to 0.001 and the dropout rate to 0.2. We apply Adam as the optimization algorithm to update the model parameters. We implement all models with PyTorch and conduct all experiments on four 2.0GHz Intel Xeon E5-2620 CPUs and a Tesla K20m GPU.The results for the Assistment datasets can be found in Table 3 . They exhibit similar patterns as those introduced in RQ1.Table 3 : The utility (U) and fairness (F) results on Assistment datasets. The best results on each backbone are highlighted in bold. ↓ indicates the lower, the better. ↑ indicates the higher, the better. We compare our CMCD with the Origin and get the p-value in the t-test. ( * * * : p < 0.001, * * : p < 0.01, * : p < 0.05.) IRTThe checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.• Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.IMPORTANT, please:• Delete this instruction block, but keep the section heading "NeurIPS paper checklist",• Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers.Here we adopt two-parameter logistic IRT model.https://sites.google.com/site/assistmentsdata 


Decision: The submitted paper was reviewed by 3 knowledgeable reviewers all of whom recommended acceptance of the paper. The reviewers appreciate the importance of the considered problem for intelligent education, leveraging the monotonicity assumption for improved data augmentation, and the theoretical as well as extensive experimental evaluation demonstrating the benefits of the proposed approach. Thus, in line with the reviewers' unanimous recommendation, I recommend acceptance of the paper. 
The authors are asked to prepare the camera-ready version by considering the reviewers' comments and in particular include the clarifications made during the discussion-phase as well as new results in the final paper.

Reviews: review: paper_summary: paper_summary: This paper discusses the issues of fairness and accuracy in cognitive diagnostic tasks, which hold significant societal value and impact the fairness of education. Through an experimental perspective, the paper analyzes how data sparsity can lead to unfairness and inaccuracy in cognitive diagnostic tasks. It also leverages the specific assumption of monotonicity in the field of education to alleviate the challenges posed by data sparsity. Notably, the authors theoretically prove the convergence and efficacy of the proposed method. Finally, an analysis is conducted on real-world datasets, accompanied by thorough validation, providing substantial evidence for the efficacy of the approach.
strength_weakness: strength_weakness: - W1: I noticed that the experimental results at the end show an improvement in both the fairness and accuracy of diagnostics simultaneously, which seems to contradict the prevailing trade-off between fairness and accuracy. It would be beneficial for the paper to delve deeper into this issue, exploring whether this approach can be extended to other domains such as the field of recommender systems.
- W2: In the experimental section, I observed that the utility and fairness metrics are aligned. While I understand that fairness analysis is based on utility outcomes, this alignment could potentially mislead readers to some extent. Clarity on this relationship would enhance the understanding for the readers.
questions: questions: See above weaknesses.
limitations: limitations: The authors have extensively discussed the limitations of their approach.
paper_summary: paper_summary: This paper focuses on fairness and accuracy issues in Cognitive Diagnosis. Unlike model-based methods, the approach in this paper tackles these issues from a data-driven perspective. Leveraging the unique monotonicity assumption in cognitive diagnosis, the authors propose a general monotonic data augmentation method that can be applied to all cognitive diagnosis models. This method comes with both theoretical guarantees and empirical validation.
strength_weakness: strength_weakness: This paper proposes two constraints for data augmentation, which are actually quite similar in nature. I am curious about the relationship between these two augmentation methods. The authors need to engage in further discussion and analysis on this aspect.
By leveraging the unique monotonicity assumption in the field of cognitive diagnosis, this paper applies data augmentation techniques that have shown promising results in both accuracy and fairness. Cognitive diagnosis falls under the subfield of user modeling. Is it possible to extend these methods to other user modeling tasks? How do these methods differ from data augmentation techniques used in other user modeling tasks? I recommend that the authors delve into further discussions on these points.
questions: questions: What is the relationship between the two proposed constraints for data augmentation? 
Can the data augmentation methods suggested be applied to other domains, thereby enhancing the generalizability of the proposed approach?
limitations: limitations: This paper discusses the privacy concerns related to user data in its limitations section and suggests federated learning as a potential future research direction. Given this, is it feasible to integrate the proposed method in this paper with federated learning?
paper_summary: paper_summary: The paper "Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation" introduces a framework called CMCD to address data sparsity in cognitive diagnosis, which affects accuracy and fairness. The authors integrate the monotonicity assumption to ensure data augmentation remains interpretable. They provide theoretical guarantees for accuracy and convergence speed and validate their method with extensive experiments on real-world datasets, showing improved performance in both accuracy and fairness. However, the paper has notable methodological flaws, limited novelty, and practical application challenges, making it less relevant for NeurIPS.
strength_weakness: strength_weakness: The paper has significant methodological flaws, including the reliance on small, homogeneous datasets, limiting the generalizability of the findings. The theoretical guarantees for accuracy and convergence speed are not convincingly demonstrated, and the practical implementation of the proposed framework is inadequately discussed. Additionally, the novelty of the contribution is limited, with the method being an incremental improvement over existing approaches rather than a groundbreaking innovation. The lack of detailed ethical considerations and practical deployment strategies further diminishes the paper's overall impact.
questions: questions: 1. Validation on Diverse Datasets: Have you considered validating your model on more diverse and larger datasets to enhance the robustness and generalizability of your findings?

2. Practical Implementation: Can you provide more details on the practical challenges and solutions for implementing CMCD in real-world educational environments?

3. Theoretical Guarantees: How do you plan to empirically demonstrate the theoretical guarantees of accuracy and convergence speed provided in the paper?

4. Ethical Considerations: What measures have you considered to address data privacy and informed consent issues in future studies?
limitations: limitations: The paper lacks a detailed discussion on the potential negative societal impact, such as bias in cognitive diagnosis.

Comments: title: Thanks for your encouragement and support!
comment: We appreciate your recognition of the value of our work! We are pleased to have addressed your concerns and will revise the paper in accordance with your suggestions.
Thanks for your prompt feedback and for acknowledging the value of our work! We will revise the paper in line with your suggestions. Once again, we appreciate your support and constructive feedback.
Thanks for the clarification which addresses most of my concerns. I will keep my score.
Thank you for your timely response and for recognizing the importance of our work! We are pleased to have clarified any uncertainties you had and will incorporate all suggested changes into the revised version as per your feedback. If you have any other questions, please feel free to ask. We will do our best to address any concerns you may have. Once again, we sincerely appreciate your encouragement and support.
Thanks a lot for the authors' great effort. I have carefully read their response.
These responses satisfy my questions and reinforce my rating.
The authors provided a comprehensive rebuttal addressing key concerns. While the responses are adequate, incorporating more detailed empirical evidence, practical implementation strategies, and ethical considerations into the paper would significantly improve its quality and relevance. 

Q1: Validation on Diverse Datasets
The authors clarified that the datasets used are highly representative and validated their model on the ASSISTments2012 dataset, demonstrating improved accuracy and fairness.
The additional dataset validation strengthens the paper.

Q2: Practical Implementation
The authors acknowledge privacy concerns and plan to integrate federated learning and differential privacy.
This approach is reasonable.

Q3: Theoretical Guarantees
The authors empirically validated accuracy and convergence speed through experiments on different datasets and backbones.
While the explanation helps, including more detailed empirical results and comparisons in the paper would clarify these guarantees.

Q4: Ethical Considerations
The authors plan to use encrypted communication, differential privacy, and federated learning to address privacy issues.
These measures are appropriate.

I will modify my review.