SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiangzhi Chen
Kun Zhang
Le  $\mathbf{W}\mathbf{u}^{1,3,*}$
Richang Hong<sup>1</sup>, 3
Fei Liu<sup>1</sup>
iu1 Lei Chen2
Meng Wang<sup>1, 4</sup>
<sup>1</sup> Hefei University of Technology
2 Tsinghua University
<sup>3</sup> Institute of Dataspace, Hefei Comprehensive National Science Center
$^{4}$  Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
{cxz.hfut, lewu.ustc}@gmail.com feiliu@mail.hfut.edu.cn
{chenlei.hfut, zhang1028kun, hongrc.hfut, eric.mengwang}@gmail.com
Cognitive diagnosis is an important task in intelligence education, which aims at measuring students' proficiency on specific knowledge concepts. Given a fully labeled exercise-concept matrix, most existing models focused on mining students' response records for cognitive diagnosis. Despite their success, due to the huge cost of labeling exercises, a more practical scenario is that limited exercises are labeled with concepts. Performing cognitive diagnosis with limited exercise labels is under-explored and remains pretty much open. In this paper, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges of limited exercise labels. Specifically, we utilize students' response records to model student proficiency, exercise difficulty and exercise label distribution. Then, we introduce two novel modules - group-based disentanglement and limited-labeled alignment modules - to disentangle the factors relevant to concepts and align them with real limited labels. Particularly, we introduce the tree-like structure of concepts with negligible cost for group-based disentangling, as concepts of different levels exhibit different independence relationships. Extensive experiments on widely used benchmarks demonstrate the superiority of our proposed model.
In the field of intelligent education systems, cognitive diagnosis (CD) [25, 26] is a fundamental and essential task, which aims at measuring students' proficiency on specific knowledge concepts through the student performance prediction process. CD can serve various applications, such as computerized adaptive testing [2], targeted training [20], and exercise recommendation [24, 14]. We show a toy example of CD in Fig. 1-(a). The exercise-concept matrix (known as Q-matrix) is labeled to indicate which concepts are tested in each exercise. For example, concepts  $k_{3}$  and  $k_{5}$  are tested in exercise  $v_{3}$ . The CD model diagnosed that student  $u_{2}$  has a higher proficiency on the two concepts than that of  $u_{1}$  because  $u_{2}$  answered correctly on the exercise while  $u_{1}$  answered incorrectly. It's obvious that fully labeled Q-matrix plays an essential role in interpretability (i.e., diagnostic report) for CD models.
Most existing CD models focus on enhancing the mining process of response records to achieve better diagnosis results. These models typically use a fully labeled Q-matrix that has been annotated by domain experts to train their models. For example, the classical DINA model [8] assumes that students must master all knowledge concepts associated with an exercise to answer it correctly,
Figure 1: Example of cognitive diagnosis. (a) We show an example to demonstrate the CD process, using a fully-labeled Q-matrix annotated by domain experts. (b) Tree-like structure of concepts is easily annotated with negligible cost as the number of concepts is far fewer than that of exercises.
except in cases where the student guesses correctly. Representative neural-based CD models, such as NCDM [38], KaNCD [39] and KSCD [30], explicitly model the explicit relationship between exercise and knowledge concepts to constrain student proficiency during diagnostic processes. Despite the success of these models, they share a common assumption that the knowledge concepts of each exercise are fully labeled. However, the annotation of exercises is a labor-intensive task that requires the expertise of professionals. There are always many exercises involved in intelligent education systems, and the workload for annotation significantly increases with the number of exercises. Thus, a more practical scenario is cognitive diagnosis with limited exercise labels.
Performing cognitive diagnosis using limited exercise labels is indeed an under-explored area in current research. When faced with the few labeled exercise, a straightforward approach is to prefetch the Q-matrix in order to provide a complete Q-matrix as input to an existing CD model. One limitation of this method is that its effectiveness largely depends on the accuracy of the pre-filling algorithm. If the pre-filling algorithm fails to identify the knowledge concepts associated with each exercise correctly, the resulting filled Q-matrix may be inaccurate or incomplete, which can negatively impact the model's diagnostic performance. Therefore, how to achieve interpretable diagnosis on specific knowledge concepts with limited exercise labels remains a challenge.
In this work, we propose Disentanglement based Cognitive Diagnosis (DCD) to address the challenges posed by limited exercise labels. Specifically, DCD is a semi-supervised disentanglement method to model the students' proficiency with limited exercise labels. To achieve this, we utilize historical practice records to model student proficiency, exercise difficulty, and exercise relevance on each knowledge concept. Then, we introduce two novel modules - the group-based disentanglement module and the limited-labeled alignment module - to disentangle factors relevant to knowledge concepts and align them with real limited exercise labels. Particularly, we introduce the tree-like structure of concepts (shown in Fig. 1-(b)) for group-based disentangling, as concepts of different levels exhibit different independence relationships. The knowledge concept tree reduces the workload required for annotating knowledge concepts, as there are far fewer concepts than exercises to annotate. These modules work together can deal with the limited exercise labels and diagnose the student's proficiency on each concept with interpretability. The main contributions of this work are summarized as follows:
- Our work represents one of the few attempts to focus on the problem of limited exercise labels in cognitive diagnosis, which is common in real-world practice.
- We propose a semi-supervised disentanglement approach to address the challenge of cognitive diagnosis with limited exercise labels.
- We conduct extensive experiments under different few-labeled settings on three real-world datasets, which demonstrates the effectiveness of our proposed DCD model.
2.1 Cognitive Diagnosis in Intelligent Education Systems
The early CD works like IRT [10] and MIRT [35] focus on modeling students' answering process by predicting the probability of a student answering a question correctly, which utilizes latent factors as
the student's ability. IRT and MIRT are also known as Latent Factor Models (LFM). The obvious limitation of the above models is the lack of interpretability, i.e., the inability to obtain an explicit multidimensional diagnostic report on each knowledge concept. To achieve better interpretability, later diagnostic models focus on incorporating knowledge concepts of exercises to diagnose students' proficiency in all knowledge concepts [8, 38, 6, 47, 21]. Representative NCDM [38] adopts neural networks to model non-linear interactions instead of handcrafted interaction functions in previous works [10, 35]. However, researchers find that the free student proficiency vector learning paradigm in NCDM is not capable of tackling weak-knowledge-coverage scenario and model relation among knowledge concepts implicitly to address the problem [39, 30].
Cognitive Diagnosis in Few-labeled Scenarios. Most existing CD models are unable to directly deal with the few-labeled scenarios, and assume that the exercises are fully labeled. NCDM+ [39] fills the missing knowledge concepts by TextCNN [18] from the exercise's text information, which regards it as a previous task for CD. Therefore, the performance of CD lies in the accuracy of pre-filled knowledge concepts. In this work, we focus on a more challenging scenario of missing exercise textual information and infer missing knowledge concepts from response records only.
2.2 Disentangled Representation Learning
Disentangled Representation Learning (DRL) is an important learning paradigm that aims to disentangle the underlying generative factors hidden in the observed data into the latent variables in the representation [1]. One notable benefit attributed to DRL is its ability to extract latent factors that embody semantic meanings, thus enhancing the interpretability of machine learning models. Both VAE [19] and GAN [12] based DRL are widely adopted techniques. Considering that the training of VAE is more stable than that of GAN [17, 45], in this work, we extend the VAE framework and focus on learning interpretable disentangled representations to help cognitive diagnosis.
VAE-based Disentangled Representation Learning.  $\beta$ -VAE [13] modifies the Evidence Lower Bound (ELBO) of VAE [19] by adding a hyperparameter  $\beta$  on KL term and find that a larger  $\beta$  corresponds to the better disentanglement of latent code. However, the increasing  $\beta$  would also lead to the worse reconstruction error dramatically [17, 36]. To analyze the trade-off between reconstruction accuracy and the quality of disentangled representation, both FactorVAE [17] and  $\beta$ -TCVAE [5] show that the success of  $\beta$ -VAE in learning disentangled representations can be attributed to penalizing the Total Correlation (TC) term.  $\beta$ -TCVAE decomposes the expected KL term in ELBO into index-code MI term, TC term, and dimension-wise KL term. The TC term would decrease the mutual information among factors and improve independence among latent factors, which results in better disentanglement. However, some researchers claim that the above unsupervised learning of disentangled representations is fundamentally impossible without inductive biases as sometimes the target dataset is not semantically clear and well-structured to be disentangled [27]. Hence, more works focus on weak-supervised approaches [28, 37] and semi-supervised [29, 16]. A theoretical framework is provided to assist in analyzing the disentanglement guarantees by weak supervision methods (e.g. restricted labeling, match pairing, and rank pairing) [37]. Some researchers validate that with little and imprecise supervision (e.g. manual labeling of factors) it is possible to reliably learn disentangled representations [29]. In this work, we regard limited exercise labels as a semi-supervision to help disentangle knowledge concepts-related factors.
Disentangled Representation Learning for User Modeling. User modeling [22] aims to capture a number of attributes of each user, with the help of items, item features and/or user-item response matrix [43], etc. DRL has a wide range of applications in user modeling to disentangle attributes. For example, recommendation with several aspects of users' interests [23, 31, 46, 40], fair user representation to disentangle sensitive attributes [7, 34].
Let  $\mathcal{U} = \{u_1, u_2, \ldots, u_N\}$ ,  $\mathcal{V} = \{v_1, v_2, \ldots, v_M\}$ , and  $\mathcal{K} = \{k_1, k_2, \ldots, k_K\}$  denote the sets of students, exercises, and knowledge concepts, respectively, where  $N, M$ , and  $K$  represent the size of each set. The response records are denoted as  $\mathbf{X} = \{X_{ij}\}_{N \times M}$ , where  $X_{ij}$  equals 1, 0 or -1 representing that the student answered the exercise correctly, incorrectly, or did not answer the exercise, respectively. The relationship between exercises and knowledge concepts is represented by the Q-matrix  $\mathbf{Q} = \{Q_{ij}\}_{K \times M}$ . Typically, each exercise is related to multiple knowledge concepts.
$Q_{ij}$  equals 1 or 0 representing that exercise  $v_{j}$  is related to the knowledge concept  $k_{i}$  or not. In this work, we explore the few-labeled situation, that is, there are only a few exercises that are labeled. Thus, we split exercise set  $\mathcal{V}$  into the labeled set  $\mathcal{V}_1$  and the unlabeled set  $\mathcal{V}_2$ .
There are correlations and independence among different knowledge concepts. In our work, we introduce the knowledge concept tree [41] to model their relations at different levels. By utilizing this tree structure, we can effectively capture knowledge concepts' interrelationships. For convenience, we transform the knowledge concept tree into a standard tree (the height of all subtrees of any node in the tree is equal). For more details about knowledge concept tree, please see Appendix C. Suppose we construct a standard tree  $G$  with  $L$  levels, wherein its leaf nodes are denoted by the set  $\mathcal{K} = \{k_1, k_2, \ldots, k_K\}$ . Each level of knowledge concept can be regarded as a grouping of the last level of knowledge concepts. The deeper the level, the finer the granularity of the grouping. Let  $G^i = \{G_1^i, G_2^i, \ldots, G_{|G^i|}^i\}$  denote grouping method of the  $i$ -th level knowledge concepts, where  $i = 1, 2, \ldots, L$  and  $|G^{i+1}| \geq |G^i|$ . Therefore,  $G^1 = \{G_1^1 = \mathcal{K}\}$  and  $G^L = \{G_1^L = \{k_1\}, G_2^L = \{k_2\}, \ldots, G_K^L = \{k_K\}\}$ .
Problem Definition. In an intelligent education system, given student set  $U$ , exercise set  $\mathcal{V} = \mathcal{V}_1 \cup \mathcal{V}_2$ , knowledge concepts  $\mathcal{K}$ , response records  $\mathbf{X}$ , knowledge concept tree  $G$  and incomplete Q-matrix, our goal is to diagnose students' cognitive states on knowledge concept set  $\mathcal{K}$  with few-labeled exercises, and predict the scores of students doing exercises.
4 The Proposed Model
In cognitive diagnosis, it is crucial to determine whether a student has the proficiency in knowledge concepts required to answer an exercise correctly. This becomes even more challenging when there are limited exercise labels available. To model students' proficiency with few-labeled exercises, we design disentanglement and alignment modules that can effectively utilize the weak supervision of a few labeled exercises.
The overall framework of DCD is illustrated in Fig. 2, including input, encoder, disentanglement, alignment, and decoder. The input of DCD contains the student-exercise interaction matrix  $\mathbf{X}$  and the few-labeled matrix  $\mathbf{Q}$ . For diagnosing students' cognitive states, we first utilize the encoder module to model three important components from  $\mathbf{X}$ : student proficiency, exercise difficulty, and exercise relevance. Then, we leverage the group-based disentanglement module to separate the factors related to knowledge concepts from the three components mentioned above. Next, the limited-labeled alignment module associates each decoupled factor with a knowledge concept semantic with the missing matrix  $\mathbf{Q}$ . Finally, the decoder module predicts the scores based on the student proficiency, exercise difficulty, and exercise relevance. Overall, DCD provides an effective approach for cognitive diagnosis under limited exercise labels.
4.1 Encoder Module
The interaction matrix  $\mathbf{X}$  provides valuable information regarding student proficiency, exercise difficulty, and exercise relevance from different perspectives. Specifically, 1) From a student-centric (row-wise) perspective, the response records of a student reflect his proficiency on all knowledge concepts. 2) From an exercise-centric (column-wise) perspective, the response records of an exercise reflect exercise difficulty and exercise relevance on all knowledge concepts. On the one hand, to effectively model from two perspectives, we design the student proficiency encoder, exercise difficulty encoder, and exercise relevance encoder modules. On the other hand, following existing cognitive models, an interaction function (corresponding to our decoder) to predict the answering result with student traits and exercise traits as input (corresponding to our encoders), which ensures the interpretability of student proficiency representation. Traditional single encoder is not applicable for cognitive diagnosis.
Student Encoder. The student encoder module is intended to infer student's proficiency on each knowledge concept from  $\mathbf{X}$ , which models true posterior distribution  $p(\mathbf{z}_u|\mathbf{x}_u)$  by constructing an inference network  $f_{\phi_u}(\mathbf{x}_u)$  corresponding to approximate posterior distribution  $q_{\phi_u}(\mathbf{z}_u|\mathbf{x}_u)$ , parameterized by  $\phi_u$ . We assume student's proficiency on all knowledge concepts  $\mathbf{z}_u$  follows a multivariate standard Gaussian distribution prior, i.e.,  $p(\mathbf{z}_u)\sim \mathcal{N}(\mathbf{0},\mathbf{I})$ . The approximate posterior
Figure 2: The overall framework of the proposed DCD. We first encode student proficiency, exercise difficulty, and relevance vectors from students' historical response records. Then, we devise two novel modules ( i.e., the group-based disentanglement and the limited-labeled alignment modules) to disentangle factors relevant to knowledge concepts and align them with real limited exercise labels. Finally, the decoder module predicts the scores based on the student proficiency, exercise difficulty, and exercise relevance.
for  $K$ -dimensional latent code  $\mathbf{z}_u \in \mathbb{R}^K$  is expressed as follows:
$$
q _ {\phi_ {u}} \left(\mathbf {z} _ {u} \mid \mathbf {x} _ {u}\right) = \prod_ {k = 1} ^ {K} \mathcal {N} \left(\mathbf {z} _ {u} [ k ]; \boldsymbol {\mu} _ {u} [ k ], \boldsymbol {\sigma} _ {u} [ k ]\right), \tag {1}
$$
where  $\pmb{\mu}_u\in \mathbb{R}^K$  and  $\sigma_u\in \mathbb{R}_{> = 0}^K$  are generated from student encoder  $f_{\phi_u}(\mathbf{x}_u)$ . The  $[\cdot ]$  indicates the index selection operation.
Exercise Encoders. We design two encoders at exercise side: exercise difficulty and exercise relevance encoders. The two encoders aim to infer exercise's difficulty and exercise's relevance on each knowledge concept from  $\mathbf{X}$ , respectively. For exercise difficulty encoder, we assume exercise's difficulty on all knowledge concepts  $\mathbf{z}_v^d$  follows a multivariate standard Gaussian distribution prior similar to the student encoder, i.e.,  $p(\mathbf{z}_v^d) \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ .
We assume exercise's relevance on all knowledge concepts  $\mathbf{z}_v^r$  follows a multivariate Bernoulli prior, i.e.  $p(\mathbf{z}_v^r) \sim \text{Bernoulli}(\mathbf{p})$ , where  $\mathbf{p}$  is a hyperparameter. The exercise relevance encoder models true posterior distribution  $p(\mathbf{z}_v^r | \mathbf{x}_v)$  by constructing an inference network  $f_{\phi_v^r}(\mathbf{x}_v)$  corresponding to approximate posterior distribution  $q_{\phi_v^r}(\mathbf{z}_v^r | \mathbf{x}_v)$ , parameterized by  $\phi_v^r$ . The approximate posterior for K-dimensional latent code  $\mathbf{z}_v^r \in \{0, 1\}^K$  is expressed as:
$$
q _ {\phi_ {v} ^ {r}} \left(\mathbf {z} _ {v} ^ {r} \mid \mathbf {x} _ {v}\right) = \prod_ {k = 1} ^ {K} \operatorname {B e r n o u l l i} \left(\mathbf {z} _ {v} ^ {r} [ k ]; \boldsymbol {\mu} _ {v} ^ {r} [ k ]\right), \tag {2}
$$
where  $\pmb{\mu}_v^r\in [0,1]^K$  are generated from exercise encoder  $f_{\phi_v}(\mathbf{x}_v)$
4.2 Group-based Disentanglement Module
The representations obtained by the encoder module do not relates to the knowledge concepts, which makes it difficult to perform an accurate cognitive diagnosis. To address this issue, we assume the entire response records as evolving from three groups of generative factors: student proficiency factors on each knowledge concept, exercise difficulty factors on each knowledge concept, and exercise relevance factors on each knowledge concept. We desire to disentangle these factors following existing DRL methods.
Inspired by the outstanding performance of  $\beta$ -TCVAE [5] in disentanglement, we introduce the structure of  $\beta$ -TCVAE to obtain a better trade-off between the reconstruction accuracy and the quality of disentangled representation. We find the index-code term of  $\beta$ -TCVAE plays a side effect on performance and drop it during optimization like [32, 7]. Take student proficiency factors as an example, the eventual Totoal Correlation (TC) constraint on  $\mathbf{z}_u$  can be expressed as follows:
$$
\mathcal {L} _ {d} \left(\mathbf {z} _ {u}\right) = D _ {K L} \left(q \left(\mathbf {z} _ {u}\right) \mid \mid \prod_ {i = 1} ^ {K} q \left(\mathbf {z} _ {u} [ i ]\right), \right. \tag {3}
$$
where  $q(\mathbf{z}_u) = \sum_{i=1}^{N} q(\mathbf{z}_u | \mathbf{x}_{u_i}) p(\mathbf{x}_{u_i})$  is the aggregated posterior of  $\mathbf{z}_u$ . The TC term would decrease the mutual information among latent factors and improve independence among latent factors. The dimension-wise KL constraint on  $\mathbf{z}_u$  can be written as follows:
$$
\mathcal {L} _ {p} \left(\mathbf {z} _ {u}\right) = \sum_ {i = 1} ^ {K} D _ {K L} \left(q \left(\mathbf {z} _ {u} [ i ]\right) \mid \mid p \left(\mathbf {z} _ {u} [ i ]\right)\right), \tag {4}
$$
where  $p(\mathbf{z}_u[i])$  follows the prior standard Gaussian distribution. The dimension-wise KL term prevents each latent variable from deviating too far from specified priors.
However, some CD works argue that the knowledge concepts are not independent and employ relation among knowledge concepts to improve CD [11, 39]. The direct independence constraint for each knowledge concept may be not applicable for cognitive diagnosis, which is also verified in our experiment. To address the challenge, we propose to utilize the easy-labeled knowledge concept tree to capture underlying independence among knowledge concepts. We assume there is less independence among groups of knowledge concepts grouped by their parent nodes in the knowledge concept tree. Corresponding to the real world, knowledge concepts in the same chapter would be related more than knowledge concepts in different chapters. Therefore, each level concept in the tree is a kind of grouping method for the last level concepts. The deeper the level, the finer the granularity of the grouping. Considering the grouping methods at the  $i$ -th level, the Eq. (3) can be rewritten as follows:
$$
\mathcal {L} _ {d} ^ {i} (\mathbf {z} _ {u}) = D _ {K L} (q (\mathbf {z} _ {u}) | | \prod_ {j = 1} ^ {| G ^ {i} |} q \left(\mathbf {z} _ {u} \left[ G _ {j} ^ {i} \right]\right)), \tag {5}
$$
where  $G_{j}^{i}$  is the knowledge concepts that belong to  $j$ -th group according to the grouping method of  $i$ -th level in the tree. The Eq. (5) allows dependence in intra-group knowledge concepts and maintains independence among inter-group knowledge concepts.
4.3 Limited-Labeled Alignment Module
After applying the group-based disentanglement module, it is important to correspond each latent factor with an actual knowledge concept to ensure the interpretability. However, there are only a few labeled exercises with knowledge concepts, making it challenging to perform the accurate alignment. Inspired by semi-supervised DRL, to address this issue, we devised an alignment module that aligns both few labeled and numerous unlabeled exercises separately. For convenience, we use  $\mathbf{q}_j\in \{0,1\} ^K$  to denote  $j$ -th column of matrix  $\mathbf{Q}$  corresponding to exercise  $v_{j}$  that belongs to  $\nu_{1}$ .
Few Labeled Exercises. For an exercise  $v_{j} \in \mathcal{V}_{1}$ , we utilize the Mean Square Error (MSE) loss to constraint similarity between  $\mathbf{q}_j$  and  $\pmb{\mu}_{v_j}^r$ , which can be defined as follows:
$$
\mathcal {L} _ {l} = \sum_ {v _ {j} \in \mathcal {V} _ {1}} \| \mathbf {q} _ {j} - \boldsymbol {\mu} _ {v _ {j}} ^ {r} \| _ {2} ^ {2}, \tag {6}
$$