SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liya Hu $^{1}$ , Zhiang Dong $^{1}$ , Jingyuan Chen $^{1,2*}$ , Guifeng Wang $^{1}$ , Zhihua Wang $^{2}$ , Zhou Zhao $^{1}$  and Fei Wu $^{1,2*}$
$^{1}$ Zhejiang University  
 $^{2}$ Shanghai Institute for Advanced Study of Zhejiang University  
{liyahu, dongza, jingyuanchen, wangguifeng, zhihua.wang, zhaozhou, wufei}@zju.edu.cn
The focus of our work is on diagnostic tasks in personalized learning, such as cognitive diagnosis and knowledge tracing. The goal of these tasks is to assess students' latent proficiency on knowledge concepts through analyzing their historical learning records. However, existing research has been limited to single-course scenarios; cross-course studies have not been explored due to a lack of dataset. We address this issue by constructing PTADisc, a Diverse, Immense, Student-centered dataset that emphasizes its sufficient Cross-course information for personalized learning. PTADisc includes 74 courses, 1, 530, 100 students, 4, 054 concepts, 225, 615 problems, and over 680 million student response logs. Based on PTADisc, we developed a model-agnostic Cross-Course Learner Modeling Framework (CCLMF) which utilizes relationships between students' proficiency across courses to alleviate the difficulty of diagnosing student knowledge state in cold-start scenarios. CCLMF uses a meta network to generate personalized mapping functions between courses. The experimental results on PTADisc verify the effectiveness of CCLMF with an average improvement of  $4.2\%$  on AUC. We also report the performance of baseline models for cognitive diagnosis and knowledge tracing over PTADisc, demonstrating that our dataset supports a wide scope of research in personalized learning. Additionally, PTADisc contains valuable programming logs and student-group information that are worth exploring in the future.
Personalized learning aims to provide students with customized learning services that align with their specific goals and abilities, which is facilitated by the vast amount of data accumulated on thriving online learning platforms. The focus of personalized learning is on diagnosing students' knowledge state and involves two key research tasks: (1) Cognitive Diagnosis (CD) [23, 5, 24, 3, 27, 8, 13], which assesses students' static latent proficiency on concepts using their learning records, and (2) Knowledge Tracing (KT) [21, 32, 20, 9, 30, 29, 16, 25, 15], which evaluates students' dynamic latent proficiency during various study phases based on their past sequential learning records. Based on the diagnostic results of students' knowledge states, several applications such as Computerized Adaptive Testing [1, 14] and Personalized Educational Planning [11] can be conducted, as illustrated in Figure 1(a).
To support the above research, several educational datasets are constructed [7, 12, 4, 31, 33, 22, 28]. However, existing datasets still face the following challenges: (1) Insufficient Data Coverage: Many
Figure 1: (a) Illustration of the personalized learning process. (b) Illustration of cold-start problem in personalized learning. (c) Overview of PTADisc.
existing datasets are limited in their application scope due to being built for specific tasks. For instance, KDD Cup 2010 [12] lacks exercise content text, making it unsuitable for supporting content-based knowledge tracing. (2) Lack of Concept Annotation: Existing datasets reveal two problems including incomplete relation annotations between concepts and problems, and missing dependency relations among concepts, as seen in Table 5. (3) Poor Cross-course Information: Most existing datasets are constructed within a single course (e.g., ASSIST [7] in Math and EdNet [4] in English). Although MOOCCubeX [31] consists of 4, 216 courses, there exist no students with learning records in multiple courses. Cross-course studies are not supported by existing datasets.
In our work, we first address the above challenges by constructing a Diverse, Immense, Student-centered and Cross-course dataset licensed by Programming Teaching Assistant (PTA<sup>1</sup>) platform, namely PTADisc. Specifically, PTADisc features in: (1) Diverse. As shown in Figure 1(c), it contains various information, comprising four key entities and relationships among them. The entities include course, student, problem, and concept. The relationship among entities is reflected in student behavior, concept-problem correlation and concept dependency. (2) Immense. It includes 74 courses, 1,530, 100 students, 4,504 concepts, 225, 615 problems, and over 680 million student exercising response logs (i.e., answer correctly or not). (3) Student-centered. The entire dataset is organized around student behaviors (i.e., response logs), providing valuable insight into personalized learning. (4) Cross-course. A subdataset with 29,454 students simultaneously taking 5 courses is extracted. This makes PTADisc the first dataset to support cross-course analysis.
Furthermore, we proposed a model-agnostic Cross-Course Learner Modeling Framework (CCLMF) based on the cross-course subdataset of PTADisc. As shown in Figure 1(b), when a student enrolls in a new course and has few response logs, it's difficult to predict the student's latent proficiency on concepts and future performance. To address this cold-start problem, CCLMF leverages student latent proficiency relationships between courses to transfer knowledge from courses with sufficient response logs, thereby improving performance in the target course. The experimental results demonstrate the advancement of CCLMF in cold-start scenarios with an average improvement of  $4.2\%$  on AUC. Our code and datasets are available at https://github.com/wahr0411/PTADisc.git.
The contributions of this paper include: (1) Construct PTADisc, a diverse, immense, student-centered and cross-course dataset, supporting various studies in personalized learning. (2) Construct a crosscourse subdataset with a significant amount of students enrolled in multiple courses, making up for the lack of existing datasets that cannot support cross-course analysis. (3) Propose CCLMF, a model-agnostic Cross-Course Learner Modeling Framework which can improve the performance of diagnostic tasks in cold-start scenarios.
'https://pintia.cn/
This study focuses on two tasks: cognitive diagnosis (CD) and knowledge tracing (KT). CD aims to analyze students' latent proficiency on concepts, assuming that they are in a stable learning state. Alternatively, KT focuses on dynamically assessing the knowledge proficiency of students as they progress through the learning process. Figure 2 visually represents the differences.
2.1 Cognitive Diagnosis
Suppose there is a course with a total of  $N$  students,  $M$  problems, and  $K$  knowledge concepts, which can be denoted as  $\mathcal{S} = \{s_1,s_2,\dots ,s_N\}$ $\mathcal{P} = \{p_1,p_2,\ldots ,p_M\}$  and  $\mathcal{C} = \{c_1,c_2,\dots ,c_K\}$  respectively. The response logs of student  $s$  are denoted by  $\mathcal{R}_s$  , a set of tuple  $(p,r)$  , with  $p\in \mathcal{P}$  and  $r$  indicating the score obtained by student  $s$  on problem  $p$  . In addition, we have  $Q = \{Q_{ij}\}_{M\times K}$  ,with  $Q_{ij} = 1$  if
(a)
Figure 2: Illustration of CD (a) and KT (b).
problem  $p_i$  is related to knowledge concept  $c_j$ , and  $Q_{ij} = 0$  in all other cases. The goal of CD is to assess student's level of proficiency on different knowledge concepts through the prediction process of student performance, given students' response logs  $\mathcal{R}$  and the Q-matrix  $Q$  [27].
2.2 Knowledge Tracing
The presentations of problems and knowledge concepts in KT are the same as in CD. For each student  $s$ , their response logs are represented by  $\mathcal{R}_s = \{e_j\}_{j=1}^l$ , where  $e_j$  denotes the  $j$ -th response log and  $l$  is the total number of logs. Each  $e$  is denoted by a triplet  $(p, r, t)$  where  $p \in \mathcal{P}$ ,  $r$  is the score and  $t$  is the timestamp of the student's response. The goal of KT is to predict students' proficiency levels of concepts at different study phases, given their response logs  $\mathcal{R}$  and the Q-matrix  $Q$ .
PTADisc is sourced from PTA, an online learning platform developed by Hangzhou PAT Education Technology Co., Ltd. PTA is an automatic program evaluation and open teaching assistance platform for universities and society. Given the close collaboration between PTA and universities, it is common for students to concurrently pursue a series of courses that align with their training program. Up to July 2023, PTA has attracted over 1,000 organizations, 9,000 teachers and 3,900,000 users and provides a problem bank of over 290,000 problems referenced by course problem sets and exams. The highlight of PTA is that it covers a significant amount of students enrolled in multiple courses. This feature perfectly meets the need to conduct cross-course research and mine student characteristics between courses.
3.1 Privacy Protection
To prevent privacy disclosure, we have excluded personal and sensitive data such as student names and email addresses, retaining only the unique student IDs as individual identifiers, with anonymization employed. Additionally, we confirmed that the user-generated data was strictly authorized during the registration process, as specified in the terms of service and privacy statement of the PTA platform.
3.2.1 Raw Data Processing
First, we manually selected 74 courses. Then as shown in Figure 3, we applied privacy protection, data curation, data filtering
Figure 3: Construction pipeline of PTADisc.
and data extraction to generate intermediate structured data. To eliminate redundancy, we created a unified problem bank that captured each problem's full score, content, and relevant concepts across all courses. Additionally, we filtered out problems without concept annotations or those with full scores of 0. For non-programming data, we selected only those students who had 5 or more response logs for each course. For programming data, we performed extraction to include relevant details such as codes submitted, programming language, as well as time and memory consumption.
3.2.2 Structured Data
The data is organized by course, and we provide examples of course, problem, and response logs in Tables 1, 2 and 3, respectively. These structured data include the following content.
Knowledge Concepts. Knowledge concepts, also known as knowledge skills, are structured into a tree format that illustrates the hierarchical dependencies among concepts, as depicted in Figure 1(c). This hierarchical structure is manually annotated using the textbook catalog, and each relation is denoted as a tuples  $(c_i, c_j)$  where  $c_j$  is the sub-concept of  $c_i$ . Each problem in the dataset is associated with one or more leaf concepts within the tree, while the course name is the root node.
Problems. On PTA platform, registered teachers can create and publish problems containing content, title, problem difficulty, problem type, full score, and other specialized configurations. In general, PTADisc contains two broad categories of problems, which are non-programming and programming problems. All the problems are stored in a problem bank and assigned a unique problem ID.
Student Groups. Student groups can be regarded as classes. Each student group is given a student group ID and contains multiple students taking the same course.
Problem Sets. Problem sets are published by teachers as homework or quiz. All the problems in problem sets are selected from the problem bank. Once a problem is selected into a problem set, it is given a problem_set probleM ID (psp ID) and a specified full score. Each problem set has an opening and closing time, and students are only allowed to complete the problem set during this period.
Student Behaviors. Response logs are provided to represent student behaviors, including submission time, problem type, test score, psp ID, and judge status. It is important to note that PTADisc provides a variety of judge-related information for the programming problems, such as code, language, running time and memory consumption, which are well worth exploring in the future.
Table 1: An example of a course in PTADisc.
Course ID	Course Name	Concept ID	Concept Name	Concept Parent ID	Problem ID	Student Group ID	Student ID
C_9088	Python Programming	C_1568	Loop	C_9088	P_3122	G_2144	S_e28d
C_9472	Function	C_9088	P_3120	G_2144	S_369e
C_2592	Break	C_1568	P_2600	G_5952	S_1c3a
C_7488	Continue	C_1568	P_3143	G_5952	S_6f58
Table 2: Examples of two problems in PTADisc.
Problem ID	Concept ID	Difficulty	Reference Count	Problem Type	Problem Set Problem ID	Full Score	Problem Set ID	Start Time	End Time
P_9600	C_3408	3	1479	Programming	PSP_1617	10	PS_8480	2018/6/7 01:08	2018/7/6 23:59
PSP_3168	15	PS_4512	2018/7/3 11:50	2018/7/8 23:59
P_4480	C_5696, C_6400	1	464	Multiple choice	PSP_7537	2	PS_2112	2020/11/3 08:24	2020/11/4 10:24
PSP_5952	1	PS_2448	2021/3/26 16:00	2021/3/31 23:59
Table 3: Examples of three response logs in PTADisc.
Problem Type	Submission ID	Student ID	Submit Time	Score	Problem Set Problem ID	Status	Language	Code	Judge Logs
True or false	Sub_4736	S_9059	2018/12/28 09:05	0	PSP_4731	NO_answer	\	\	\
Multiple choice	Sub_1520	S_9059	2018/12/28 08:58	2	PSP_4750	ACCEPTED	\	\	\
Programming	Sub_5088	S_be7f	2018/6/8 19:18	75	PSP_6944	PARTIAL_ACCEPTED	PYTHON	# Code s=input() ......	time: 0.025, memory: 3260416, result: ACCEPTED; ......
(b)
Figure 4: PTADisc statistics. (a) Correlation coefficients of student performance between five courses. (b) The distribution of response logs and enrolled students.
3.2.3 Task-specific Datasets
Cognitive Diagnosis Dataset. Following the generation of structured data, we performed index mapping to create the CD datasets, which consist of a Q-matrix and students' response logs. The Q-matrix stores the relationships between problems and con
Table 4: Cross-course datasets in PTADisc.
Course Name	Java	C++	Python	DS2	C
#Students	29,454	29,454	29,454	29,454	29,454
#Problems	16,752	15,172	17,787	21,952	26,056
#Response Logs	4,750,970	6,587,356	6,454,336	7,789,280	11,378,017
#Concepts	773	547	685	767	847
cepts. The response logs of each student were split into training, validation, and testing datasets in a ratio of  $70\% /10\% /20\%$ , respectively.
Knowledge Tracing Dataset. The process for generating the KT dataset is similar to the CD dataset, except that each log in the KT dataset includes an additional submitting time.
Cross-course Dataset. We analyzed students taking multiple courses and identified the top five frequent course sets. Furthermore, we eliminated students with less than 10 response logs for any of the five courses and were left with a total of 29,454 students for the study, as presented in Table 4.
PTADisc includes 74 courses, 1, 530, 100 students, 4, 504 concepts, 225, 615 problems, and over 680 million student response logs. The response logs were partitioned according to problem type into programming and non-programming logs. Figure 4(b) presents the distribution of response logs and enrolled students for each course. Additional detailed statistics can be found in Appendix B.
3.4 Dataset Characteristics
The comparison of PTADisc with other open-access educational datasets is shown in Table 5. We divide these datasets into three categories: (1) student-centered datasets including ASSIST09[7] $^3$ , Junyi $^4$ , KDD Cup 2010 [12], EdNet [4], which focus on student behaviors and are frequently used for diagnostic tasks in personalized learning; (2) knowledge-centered datasets including MOOC-CubeX [31]; and (3) programming datasets including BePKT [33] and CodeNet [22]. We illustrate the characteristics of PTADisc in four aspects.
Diverse. As illustrated in Table 5, PTADisc contains extensive concept-related information and detailed records of student behaviors. PTADisc offers coverage of concept dependencies which are not found in ASSIST, EdNet and CodeNet. Moreover, PTADisc is the only dataset that evaluates student responses using a scoring ratio system rather than binary values [17]. Besides, PTADisc includes student group information, enabling group-level analysis.
Immense. As Table 5 illustrates, PTADisc is currently the largest educational dataset, featuring diverse data scales across multiple courses. This range of data scales presents researchers with a multitude of options for personalized learning studies.
$^{2}$ DS stands for Data Structure and Algorithm Analysis.
<sup>3</sup>https://sites.google.com/site/assistmentsdata/datasets
4https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=1198
Table 5: Comparison between PTADisc and existing open datasets.
Field	Dataset Name	PTADisc	ASSIST	Junyi	KDD.	EdNet	MOOCCubeX	BePKT	CodeNet
Overview	Number of LogsCross-course Info	6.81e8	4.02e5	2.59e7	2.27e6	1.31e8	2.96e8	6.79e4	1.39e7
✓	✘	✘	✘	✘	✘	✘	✘
Concepts Variety	Concepts Annotations	✓	✓	✓	✓	✓	incomplete5	✓	✘
Concepts Dependencies	✓	✘	✓	✘	✘	✓	✘	✘
Problems Variety	Programming Problems	✓	✘	✘	✘	✘	✘	✓	✓
Problem Type	✓	✓	✓	✘	✘	✓	✘	✘
Logs Variety	Detailed Programming Logs	✓	✘	✘	✘	✘	✘	✓	✓
Non-binary Response Results	✓	✘	✘	✘	✘	✘	✘	✘
Other Info	Problem Set	✓	✓	✘	✓	✓	✓	✘	✘
Student Group	✓	✓	✓	✘	✘	✘	✘	✘
Student-centered. PTADisc is a student-centered dataset focusing on student behaviors. By validating problems and knowledge concepts through analysis of student response logs, the dataset improves consistency and maintainability, making it well-suited for diagnostic tasks.
Cross-course. PTADisc is the first dataset with cross-course information. For the cross-course subset in PTADisc, we further analyzed the correlation coefficient between these courses based on the performance of each student. As shown in Figure 4 (a), there is a positive correlation among the five courses, providing a statistical basis for cross-course learner modeling.
3.5 Dataset Applications
PTADisc provides support for CD and KT tasks, as mentioned in Section 2. Section 5 presents the CD and KT baseline experimental results on several existing methods. Additionally, PTADisc can also support the following tasks in the field of AI for education: (1) Prerequisite discovery [19], which identifies the sequence in which concepts or topics should be learned, ensuring foundational concepts are understood before advanced ones. (2) Computerized adaptive testing [1], which aims to rapidly and accurately diagnose a candidate's level of knowledge mastery through personalized test items. (3) Educational recommendation [11], which provides appropriate learning suggestions to students based on their interactions with problems. (4) Cross-course research. PTADisc provides cross-course information, enabling the study of how students perform in different classes. We conducted a cross-course study addressing the cold-start issue in Section 4.
With various information provided, PTADisc also has the potential to support the following research directions: (1) It provides non-binary performance data. Most existing CD and KT methods tackle this as binary classification (wrong/right answer). Non-binary grades allow for regression-focused investigations. (2) It provides information on problem types, aiding research of CD and KT. (3) It provides information on problem difficulty, enabling to study how the level of difficulty of exercises relates to students' final learning outcomes. (4) It provides problem set specifics, like submission time, enabling modeling of students' learning habits based on when they submit their work. (5) It provides data on student groups, facilitating the assessment of teaching quality within classes and group-level educational analysis. (6) It provides information on programming exercises, including detailed code submissions and records, enabling in-depth research into programming-related studies.
4 CCLMF: Cross-course Learner Modeling Framework
By utilizing the cross-course subdataset of PTADisc, we can leverage the relationships between courses to mitigate the challenges associated with the cold-start problem. When a student begins a new course (i.e., target course) and has limited data, it can be challenging to diagnose his proficiency level through CD or KT. To address this problem, we present the Cross-Course Learner Modeling Framework (CCLMF) inspired by cross-domain recommendation [18, 34], which utilizes auxiliary information from another course (i.e., source course) taken by the student and has enough data. By identifying connections between the student's proficiency in the target and source courses, CCLMF enhances the model's performance in low-data scenarios. CCLMF incorporates a meta-learner to predict network parameters, leveraging the power of meta-learning to improve performance in the target course by harnessing the knowledge acquired from source courses. Please note that CCLMF
<sup>5</sup>Concept annotations are not available for all problems, since the annotation process is similar to keyword matching.
Figure 5: CCLMF architecture consists of two stages. (a) In the pre-training stage, we train a CDM in the source course and get student  $i$ 's proficiency representation  $\boldsymbol{u}_i^s$ . (b) In the meta stage, we utilize the meta network to generate the mapping function  $F_{\boldsymbol{u}_i}$  for each student  $i$ , and the proficiency representation in the target course is obtained by  $\boldsymbol{u}_i^t = F_{\boldsymbol{u}_i}(\boldsymbol{u}_i^s; \boldsymbol{w}_{\boldsymbol{u}_i})$ .
Algorithm 1 CCLMF: Cross-course Learner Modeling Framework
Input: shared students: S   
other inputs:  $\mathcal{P}^s,\mathcal{C}^s,\mathcal{Q}^s,\mathcal{R}^s,\mathcal{P}^t,\mathcal{C}^t,\mathcal{Q}^t,\mathcal{R}^t$    
Pre-training Stage: 1. Train a CDM in the source course.   
2. Get student proficiency representation in the source course:  $\pmb {u}_i^s = H_{\mathrm{CDM}}(s_i;\pmb {w}_h)$    
Meta Stage: 1. Use a meta network  $G(\cdot)$  to generate mapping function  $F_{u}$    
2. Get the student proficiency representation in the target course by  $\hat{\pmb{u}}_i^t = F_{\pmb{u}_i}(\pmb {u}_i^s;\pmb {w}_i$    
3. Train  $G(\cdot)$  and  $F_{u}$  by optimizing losscc-CDM  $= \text{loss}_{\text{CDM}}(\hat{\pmb{u}}_i^t$  , other inputs).   
Inference: For a cold-start student  $S_{j}$  in the target course, his level of proficiency is  $F_{u_j}(u_j^s;w_u$
is a model-agnostic framework that can be applied to various CD or KT models. To explain it in a simpler way, we will illustrate CCLMF in the context of cognitive diagnosis.
We assumed that  $N$  students have enrolled in both the source and target courses, denoted as  $\mathcal{S}$ . The source course has  $M^s$  problems and  $K^s$  knowledge concepts, denoted as  $\mathcal{P}^s$  and  $\mathcal{C}^s$ . And the target course has  $M^t$  problems and  $K^t$  knowledge concepts, denoted as  $\mathcal{P}^t$  and  $\mathcal{C}^t$ . The response logs in the source and target course are denoted as  $\mathcal{R}^s$  and  $\mathcal{R}^t$ . For both courses, the Q-matrices are denoted as  $\mathbf{Q}^s = \{Q_{ij}^s\}_{M^s\times K^s}$  and  $\mathbf{Q}^t = \{Q_{ij}^t\}_{M^t\times K^t}$ . The goal of the CCLMF is to accurately measure students' level of proficiency on various knowledge concepts in the target course, which incorporates student response logs and Q-matrices from both the target and source courses.
4.2 CCLMF Architecture
We illustrate the architecture of CCLMF in Figure 5. CCLMF mainly consists of two stages: a pre-training stage and a meta stage. In the pre-training stage, a cognitive diagnosis model (CDM) such as NCD is trained using data from the source course, yielding a representation  $\boldsymbol{u}_i^s$  for student  $s_i$ . In the meta stage, instead of directly mapping  $\boldsymbol{u}_i^s$  to student representation of the target course  $\boldsymbol{u}_i^t$ , we applied the idea of meta-learning to learn a personalized cross-course transformation function  $F_u$  for each student. Moreover, this meta learning procedure is task-oriented, with  $F_u$  being trained by minimizing Equation (4) for specific tasks using data in the target course. After completing the meta stage, a personalized transformation function is obtained for each student. Thus, given any student  $s_j$  who is new to the target course, their proficiency representation can be determined by their transformation function as  $F_{u_j}(u_j^s; w_{u_j})$ .