PTADisc: A Cross-Course Dataset Supporting Personalized Learning in Cold-Start Scenarios
Authors: Liya Hu
Zhiang Dong
Jingyuan Chen
Guifeng Wang
Zhihua Wang
Zhou Zhao
Fei Wu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Personalized Learning
Cognitive Diagnosis
Knowlege Tracing
Open-Access Datasets
Cross-Course Learner Modeling

Abstract: The focus of our work is on diagnostic tasks in personalized learning, such as cognitive diagnosis and knowledge tracing. The goal of these tasks is to assess students' latent proficiency on knowledge concepts through analyzing their historical learning records. However, existing research has been limited to single-course scenarios; cross-course studies have not been explored due to a lack of dataset. We address this issue by constructing PTADisc, a Diverse, Immense, Student-centered dataset that emphasizes its sufficient Cross-course information for personalized learning. PTADisc includes 74 courses, 1,530,100 students, 4,054 concepts, 225,615 problems, and over 680 million student response logs. Based on PTADisc, we developed a model-agnostic Cross-Course Learner Modeling Framework (CCLMF) which utilizes relationships between students' proficiency across courses to alleviate the difficulty of diagnosing student knowledge state in cold-start scenarios. CCLMF uses a meta network to generate personalized mapping functions between courses. The experimental results on PTADisc verify the effectiveness of CCLMF with an average improvement of 4.2% on AUC. We also report the performance of baseline models for cognitive diagnosis and knowledge tracing over PTADisc, demonstrating that our dataset supports a wide scope of research in personalized learning. Additionally, PTADisc contains valuable programming logs and student-group information that are worth exploring in the future.

PTADisc: A Cross-Course Dataset Supporting Personalized Learning in Cold-Start Scenarios
Liya Hu', Zhiang Dong', Jingyuan Chen Guifeng Wang'. Zhihua Wang", Zhou Zhao' and Fei Wu
'Zhejiang University
Shanghai Institute for Advanced Study of Zhejiang University (liyahu, dongza, jingyuanchen, wangguifeng, zhihua. waihu wufei]@zzu. edw.cc..edu.c
wang, zhaozhou,
Abstract
The focus our work on diagnostic tasks in personalized learning, such as cogmi- tive diagnosis and knowledge tracing. The goal of these tasks is to assess students' latent proficiency on knowledge concepts through analyzing their historical learn- ing records. However, existing research has been limited to single-course scenarios; cross-course studies have not been explored due to a lack of dataset. We address this issue by constructing PTADisc Diverse, Immense, Student-centered dataset that emphasizes its sufficient Cross-course information for personalized learning. PTADisc includes 74 courses, 1, 530, 100 students, 054 concepts, 225, 615 prob- lems, and over 680 million student response logs. Based on PTADisc, we developed model-agnostic Cross-Course Learner Modeling Framework (CCLMF) which utilizes relationships between students' proficiency across courses to alleviate the difficulty of diagnosing student knowledge state cold-stare scenarios. CCLMF uses a meta network to generate personalized mapping functions between courses. The experimental results on PTADisc verify the effectiveness of CCLMF with an average improvement of 4.2% on AUC. We also report the performance of baseline models for cognitive diagnosis and knowledge tracing over P'TADisc, demonstrat- ing that our dataset supports wide scope of research in personalized learning. Additionally, PTADisc contains valuable programming logs and student-group information that are worth exploring in the future.
Introduction
Personalized learning aims to provide students with customized learning services that align with their specific goals and abilities, which is facilitated by the amount of data accumulan ated thriving online learning platforms. The focus of personalized learning is on diagnosing students' knowledge
state and involves two key research tasks: (1) Cognitive Diagnosis (CD) 23 24
27
g [3].
which assesses students' static latent proficiency on concepts using their learning records, and (2)
Knowledge Tracing (KT) 21 32
30
29
16 25 15), which evaluates students' dynamic
latent proficiency during various study phases based on their past sequential learning records. Based on the diagnostic results of students' knowledge states, several applications such as Computerized Adaptive Testing [1] and Personalized Educational Planning [[] can be conducted, as illustrated in Figure aa).
To support the above research, several educational datasets are constructed 12 However, existing datasets still face the following challenges: (1) Insufficient Data Coverage: Many
Corresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(a)
(b)
start taking
2i
Online Course
Massive Dala
Online
Insufficient Data
Insocurate Analysis
Computerized Adaptive Testing Personalized Educational Planning
1. Cognitive Diagnosis
2. Knowledge Tracing
Diegnostic Resul
(c)
Course
Student
Problem
Concept
probiom ID:
problem ID:
B
8
difizilly
diliaily:
hull soore:
tut score: 10
coroest ID: name: inverse manty
concept ID:2 name matth transpostion
concapt ID: name: makik
course
group group
kpp: siglle
#reference:
ope: progranviing
#reference:
Student Behavior
Concept-Problem Correlation
Concept Dependency
Response Loge
course
correct
Problem
wrongl
377, commect
atudent course
MNDICSEEEE CONCREL
Figure 1: (a) Illustration of the personalized learning process. (b) Illustration of cold-start problem in personalized learning. (c) Overview of PTADisc.
existing datasets are limited in their application scope due to being built for specific tasks. For instance, KDD Cup 2010 [12 lacks exercise content text, making it unsuitable for supporting content- based knowledge tracing. (2) Lack of Concept Annotation: Existing datasets reveal two problems including incomplete relation annotations between concepts and problems, and missing dependency relations among concepts, as seen in Table []] (3) Poor Cross-course Information: Most existing datasets constructed within single course (e.g., ASSIST (7] in Math and EdNet 4 in English). Although MOOCCubeX (31) consists of 4, 216 courses, there exist no students with learning records in multiple courses. Cross-course studies are not supported by existing datasets.
In our work, we first address the above challenges by constructing a Diverse, Immense. Student- centered and Cross-course dataset licensed by Programming Teaching Assistant (PTA platform, namely PTADisc. Specifically, PTADisc features in: (1) Diverse. As shown in Figure [c]. contains various information, comprising four key entities and relationships among them. The entities include course, student, problem, and concept. The relationship among entities is reflected in student behavior, concept-problent correlation and concept dependency. (2) Immense. It includes 74 courses, 1, 530, 100 students, 4, 504 concepts, 225, 615 problems, and over 680 million student exercising response logs (i.e., answer correctly or not). (3) Student-centered. The entire dataset organized around student behaviors (i.e., response logs), providing valuable insight into personalized learning. (4) Cross-course. A subdataset with 29, 454 students simultaneously taking 5 courses is extracted. This makes PTADisc the first dataset to support cross-course analysis.
Furthermore, proposed model-agnostic Cross-Course Learner Modeling Framework (CCLMF) based on the cross-course subdataset of PTADisc. As shown in Figure when a student enrolls in course and has few response logs, it's diffieult to predict the student's latent proficiency concepts and future performance. To address this cold-start problem, CCLMF leverages student latent proficiency relationships between courses to transfer knowledge from courses with sufficient response logs, thereby improving performance in the target course. The experimental results demonstrate the advancement of CCLMF in cold-start scenarios with average improvement of 4.2% on AUC. Our git
code and datasets are available at https::///// /tt///ittuub... www.///aww///////Aa.A..
The contributions of this paper include: (1) Construct PTADisc, diverse, immense. student-centered
and cross-coure dataset, supporting various studies in personalized learning. (2) Construct a cross- course subdataset with significant amount of students enrolled in multiple courses, making up for the lack of existing datasets that cannot support cross-course analysis. (3) Propose CCLMF, a model-agnostie
Learne Modeling Framework which can improve the performance of diagnostic tasks in cold-start scenarios.
https://pinta..cm

---

2
Problem Definition
This study focuses two tasks: cognitive diagnosis (CD) and knowledge tracing (KT). CD aims to analyze students' latent proficiency on concepts, assuming that they are in a stable learning state. Alternatively, KT focuses dynamically assessing the knowledge proficiency of students they progress through the learning process. Figure 2
visually represents the differences.
2.1 Cognitive Diagnosis
(8)
Fuecton
Protheema
Concapte
Suppose there is a course with to- tal of N students, M problems, and K knowledge concepts, which can be denoted as S {81, S2,...,8
Pleciton
lina
Plaamwtrr
Duta type
Probiom Bank
Response Logs
Diagnostic Result
P (pt.P.....:.P PMM and C fer, C2....... CK) respectively. The re- sponse logs of student are denoted by Ra, set of tuple rp, with p P and r indicating the score obtained by student on problem p. addition, we have Q (Qis) M with Qú 1 problem p. related to knowledge concept cj, and Qi = 0 in all other cases. The goal CD is to assess student's is level of proficiency different knowledge concepts through the prediction process of student performance, given students' response logs R and the Q-matrix Q [27].
(b)
Problem
Response Loge
Diagnostic Result
?
Figure 2: Illustration of CD (a) and KT (b).
2.2 Knowledge Tracing
The presentations of problems and knowledge concepts in KT are the same as in CD. For each student their response logs represented by R. {eyli=z where ej denotes the j-th response log and 7 is the total number of logs. Each is denoted by triplet (p, r,t) where is the timestamp of the student's response. The goal of KT is to predict students' proficiency levels of concepts different study phases, given their response logs and the Q-matrix Q.
P, r is the score and t
3 Dataset
PTADise is sourced from PTA, an online learning platform developed by Hangzhou PAT Education Technology Co., Ltd. PTA is an automatic program evaluation and open teaching assistance platform for universities and society. Given the close collaboration between PTA and universities, it common for students to concurrently pursue series of courses that align with their training program. Up to July 2023, PTA has attracted over 1, 0.000 organizations, 9, 000 teachers and 3, 900, 000 users and provides problem bank of over 290, 000 problems referenced by course problem sets and exams. The highlight of PTA that it covers a significant amount of students enrolled in multiple courses. This feature perfeetly meets the need to conduct cross-course research and mine student characteristics between courses.
3.1 Privacy Protection
To prevent privacy disclosure, we have excluded personal and sensitive data such as student names and email addresses, retaining only the unique student IDs individual identifiers, with anonymization employed. Additionally, we confirmed that the user-generated data was strictly authorized during the registration process, as specified in the terms of service and privacy statement of the PTA platform.
3.2 Dataset Construction
Pro--rornniiiggg
3.2.1 Raw Data Processing
Privacy Protection
Data Curation
Data Flitering
Index Mapping
Structured
speottl
First, we manually selected 74 courses. Then as shown in Fig- ure(] we applied privacy protec- tion, data curation, data filtering
Programming Protiens
Data Extraction
Figure 3: Construction pipeline of PTADisc.
3

---

and data extraction to generate intermediate structured data. To eliminate redundancy, we created unified problem bank that captured each problem's 's full score. content, and relevant concepts across all courses. Additionally, we filtered out problems without concept annotations or those with full scores O. For non-programming data, we selected only those students who had 5 or more response logs for each course. For programming data, we performed extraction to include relevant details such as codes submitted, programming language, as well as time and memory consumption.
3.2.2 Structured Data
The data is organized by course, and we provide examples of course, problem, and response logs in Tables T and respectively. These structured data include the following content.
Knowledge Concepts. Knowledge concepts, also known as knowledge skills, are structured into a tree format that illustrates the hierarchical dependencies among concepts, depicted in Figure [c). This hierarchical structure is manually annotated using the textbook catalog, and each relation is denoted tuples (ci, oj) where Gj the sub-concept of cz. Each problem in the dataset is associated with one or more leaf concepts within the tree, while the course name is the root node.
Problems. On PTA platform, registered teachers can create and publish problems containing content, title, problem difficulty, problem type, full score, and other specialized contigurations. In general, PTADisc contains broad categories of problems, which are non- non-programming and programming problems. All the problems are stored in a problem bank and assigned a unique problem ID.
Student Groups. Student groups can be regarded as classes. Each student group is given student group and contains multiple students taking the same course.
Problem Sets. Problem sets are published by teachers as homework or quiz. All the problems in problem sets are selected from the problem bank. Once problem is selected into problem set, is given a problem_ La___see_________________ peobleerppemm___eeeeeeee ID (psp ID) and a specified full score. Each problem set has an opening and closing time, and students are only allowed complete the problem during this period.
Student Behaviors. Response logs are provided to represent student behaviors, including submission time, problem type, test score, psp ID, and judge status. It is important note that PTADisc provides a variety of judge-related information for the programming problems, such as code, language, running time and memory consumption, which are well worth exploring in the future.
Table 1: An example of a course in PTADisc.
Concept
Concept
Student Group ID
Course ID
Course Name
Concept ID
Problem ID
Student ID
Name
Parent ID
C. 1,1568
Loop
C. C_9088
P P_3122
G 2144
S c28d
Python
C 0.9222
Function
C 9088
P_3120
G G.2144
$_369c
C_9088
Programming
C 2592
Break
1568
P 2600
G G_5952
S. Ic3a
7488
Continuc
1568
P_31
G_5952
S. 6158
Table 2: Examples of two problems in PTADisc.
Problem ID
Concept ID
Difficulty
Reference Count
Problem Type
Problem Set
Full
Problem
Start
End
Problem ID
Score
Time
Time
P_9600
C_3408
1479
Programming
PSP_1617
10
PS. 8480
2018/6/7
01:08
2018/7/6
23:50
PSP_3168
15
PS_4512
2013/733
2018/788
11:50
23:59
P_4480
C_5696,
C46000
464
Multiple choice
PSP_5952
PSP_7537
2
PS 2448
PS PS_2111
2021/3/26
2020V11/5
08:24
2021/3/31
202011/4
Table Examples of three response logs PTADisc.
Problem Type
Submission
Student ID
Submit
Problem
ID
Time
Score
Problem ID
Status
Language
Code
Judge Logs
Truc or false
Sub_4736
$_9059
2018/12/28 09:05
PSP_4731
NNSWER
\
Multiple choice
Sub 1520
$_9059
PSP_4750
ACCEPTED
0W:58
Programming
Sub_5088 5088
b_
2018/6/8
75
PSP_6944
PARTIAL ACCEPTED
PYTHON
Couee sminput)
time: 0.025,
memeary: 3260416,
19:18
result: ACCEPTED:

---

C++ DSPython
0.64
10"
10'
10
#Students
#Non Programming Logs
10
1007
Progranmmnng Logs
0.54
0.65 0.62
0.64
10"
10
10"
0.64 0.69
0.79
0.67
10a
10F
Python DS C++ Java C
10.
Comp. DR.
Einux
Probabfiity
10
Course sorted by #ton-Programming Logs
(a)
(h)
Figure 4: PTADisc statistics. (a) Correlation coefficients of student performance between five courses. (b) The distribution of response logs and enrolled students.
3.2.3 Task-specific Datasets
Table 4: Cross-course datasets in PTADisc.
Cognitive Diagnosis Dataset. Following the generation of structured data, per- formed index mapping to create the CD datasets, which consist of Q-matrix and students' response logs. The Q-matrix stores the relationships between problems and con-
Course Name
Java
C++
Python
D
C
#Students
29,454
29,454
29.454
29,454
29.454
#Problmss
16.752
15,172
17.787
21,952
26,056
#Response Logs #Concepts
4,750,970
6,587,356
6,454,336
7,789,280
11.378,017
773
547
685
767
847
cepts. The response logs of each student were split into training, validation, and testing datasets in a ratio of 70%/10%//20%, respectively.
Knowledge Tracing Dataset. The process for generating the KT dataset similar to the CD dataset, except that each log in the KT dataset includes additional submitting time.
Cross-course Dataset. We analyzed students taking multiple courses and identified the top five frequent course sets. Furthermore, we eliminated students with less than 10 response logs for any of the five courses and were left with a total of 29, 454 students for the study, as presented Table
3.3 Dataset Statistics
PTADisc includes 74 courses, 1, 530, 100 students, 4, 504 concepts, 225, 615 problems, and over 680 million student response logs. The response logs were partitioned according to problem type into programming and non-programming logs. (igure4[h) presents the distribution response logs and enrolled students for each course. Additional detailed statistics can be found in Appendix B.
3.4 Dataset Characteristics
The comparison of PTADisc with other open-access educational datasets is shown in Table We divide these datasets into three categories: (1) student-centered datasets including ASSISTO9 7 Juny(" KDD Cup 2010 [12] EdNet 4 which focus on student behaviors and are frequently used for diagnostic tasks personalized learning; (2) knowledge-centered datasets including MOOC-
CubeX [3I] and (3) programming datasets including BePKT [33] and CodeNet [22].
We illustrate
the characteristics of PTADisc in four aspects.
Diverse. As illustrated Table 5 PTADisc contains extensive concept-related information and detailed records of student behaviors. PTADisc offers coverage of concept dependencies which are found ASSIST, EdNet and CodeNet. Moreover, PTADisc is the only dataset that evaluates student responses using scoring ratio system rather than binary values 17 Besides, PTADisc includes student group information, enabling gevup-level analysis.
Immense. As Table 5 illustrates, PTADisc is currently the largest educational dataset, featuring diverse data scales across multiple courses. This range of data scales presents researchers with multitude of options for personalized learning studies.
2DS stands for Data Structure and Algoritium Analysis. htapp:://sittes.oogge www.//tteaaasiiieenaaaaaaaaaaee
https://ppslcdatasshpp web. cmu. htt..aa..d//ootteeeettttaaaaaaaa

---

Table 5: Comparison between PTADisc and existing open datasets.
Field
Dataset Name
PTADise
ASSIST
Junyi
KDD.
EdNer
MOOCCubeX
BePKT
CodeNet
Overview
Number of Logs
6.81e8
4.02c5
2.59c7
2.27c6
131c8
2.9608
6.7964
L.397
Cross-curre Info
x
x
x
X
Cuncepts Variety
Concepts Annotarions
/
incomples
Concepts Dependencies
x
x
x
Problems Variety
Problem Type
Programinin Probloms
x
x
Logs Variety
Non. binary Response Results
Detailed
Logs
x
x
X
V
Other Info
Problem Set
3
x
V
x
Student Group
Student-centered. PTADise is a student-centered dataset focusing on student behaviors. By validat- ing problems and knowledge concepts through analysis of student response logs, the dataset improves consistency and maintainability, making it well-suited for diagnostic tasks.
Cross-course. PTADisc is the first dataset with cross-course information. For the cross-course subset in PTADisc, we further analyzed the correlation coefficient between these courses based on the performance of each student. As shown in Figure (a), there is a positive correlation among the five courses, providing a statistical basis for cross-course learner modeling.
3.5 Dataset Applications
PTADisc provides support for CD and KT tasks, as mentioned in Section 2 Section [ presents the CD and KT baseline experimental results on several existing methods. Additionally, PTADisc can also support the following tasks in the field of AT for education: (1) Prerequisite discovery [9]] which identifies the sequence in which concepts or topics should be learned, ensuring foundational concepts are understood before advanced ones. (2) Computerized adaptive testing [. which aims to rapidly and accurately diagnose a candidate' 's level of knowledge mastery through personalized test items. (3) Educational recommendation (II. which provides appropriate learning suggestions to students based on their interactions with problems. (4) Cross-course research. PTADisc provides cross-course information, enabling the study of how students perform in different classes. We conducted cross-course study addressing the cold-start issue in Section 4
With various information provided, PTADisc also has the potential support the following research directions: (1) It provides non-binary performance data. Most existing CD and KT methods tackle this as binary classification (wrong/right answer). Non-binary grades allow for regression- focusee investigations. (2) It provides information on problem types, aiding research of CD and KT. (3) It provides information on problem difficulty, enabling to study how the level of difficulty of exercises relates to students' final learning outcomes. (4) It provides problem set specifics, like submission time, enabling modeling of students' learning habits based on when they submit their work. (5) provides data student groups, facilitating the assessment of teaching quality within classes and group-level educational analysis. (6) It provides information on programming exercises, including detailed code submissions and records, enabling in-depth research into programming-relaan related studies.
4 CCLMF: Cross-course Learner Modeling Framework
By utilizing the cross-course subdataset of PTADise, we can leverage the relationships between courses to mitigate the challenges associated with the cold-start problem. When a student begins a new course (i.e., target course) and has limited data, it can be challenging to diagnose his proficiency level through CD or KT. To address this problem, we present the Cross-Courts Learne Modeling Framework (CCLMF) inspired by cross-domain recommendation [8 34 which utilizes auxiliary information from another course (i.e., source course) taken by the student and has enough data. By identifying connections between the student's proficiency in the target and source courses, CCLMF enhances the model's 's performance in low-data scenarios. CCLMF incorporates a meta-learner to predict network parameters, leveraging the power of meta-learning to improve performance in the target course by harnessing the knowledge acquired from source courses. Please note that CCLMF
'Concept annotations are not available for all problems, since the annotation process is similar to keyword matching.

---

(a) Pre-training Stage
Student Input
(b) Meta Stage
Course
Target Course
CDM
HeDD
Mepping Function
300
Fu((M]
300
Ms.
NCD
+
Lreee Lever
Mota Network
Limar Levers
Figure 5: CCLMF architecture consists of two stages. (a) In the pre-training stage, we train CDM in the source course and get student i's proficiency representation u²- (b) In the meta stage, we utilize the meta network to generate the mapping function Fu,, for each student i, and the proficiency representation in the target course is obtained by u Fu, (uf; W.:)-
Algorithm 1 CCLMF: Cross-course Learner Modeling Framework
Input: shared students: S
other inputs: P=, _C*QQ*
Q", RR, RR,,P,,C ,,, Q' R
Pre-training Stage:
I. Train a CDM in the source course.
2. Get student proficiency representation in the source course: ut HCDM(s;: wa).
Meta Stage:
1. Use meta network G(-) to generate mapping function Fu.
2. Get the student proficiency representation in the target course by ai = Fu, (ut; wan)-
3. Train G(.) and F., by optimizing lossCC-CDM lossCDM other_ (ithhrr_ Inference:
For a cold-start student 8j in the target course, his level of proficiency is Fu, (uf; w.,).
is a model-agnostic framework that can be applied to various CD or KT models. To explain it in simpler way, we will illustrate CCLMF in the context of cognitive diagnosis.
4.1 Problem Definition
We assumed that N students have enrolled in both the source and target courses, denoted as S. The source course has M problems and K* knowledge concepts, denoted as pp' and C*. And the target course has M problems and Kt knowledge concepts, denoted as Pd and C The response logs in the source and target course are denoted as RD and R'. For both courses, the Q-matrices are denoted as Q' {Qr } J x and Q' {Qr } xI.. The goal of the CCLMF is to accurately measure students' level of proficiency on various knowledge concepts in the target course, which incorporates student response logs and Q-matrices from both the target and source courses.
4.2 CCLMF Architecture
We illustrate the architecture of CCLMF in Figure 5 CCLMF mainly consists of two stages:
pre-training stage and a meta stage. In the pre-training stage, a cognitive diagnosis model (CDM) such as NCD is trained using data from the source course, yielding representation ui for student s; In the meta stage, instead of directly mapping u to student representation of the target course u/. we applied the idea of meta-learning to learn a personalized cross-course transformation function Fu for each student. Moreover, this meta learning procedure is task-ori, with Fu being trained by minimizing Equation (4) for specific tasks using data in the target course. After completing the meta stage, a personalized transformation function is obtained for each student. Thus, given any student 8, who is new to the target course, their proficiency representation can determined by their transformation function as Fu, (uz: way).
7

---

Table 6: CCLMF results MIRT and NCD.
Metries
Model
no dropout
10% dropout
20% dropout
30% dropout
40% dropout
50% dropout
AUC
CC-MIRT
MIRT
0.6379
0.7089 (+0.0680)
0,6412
0,7025 (+0.0612)
0.6398
0.6998 (40 (40,0600)
0,6399
0,6998 (+0.0599)
0.6342
0.6957 (40.0615)
0.6363
0.6918 (+0.0555)
ACC
CC-MIRT
MIRT
0.6822
0.7854440.102
0.7037
0.7834 (+0.0797)
0.6869
0.0866(1000958)
0,6896
0.783
0.6889
0.7822 (10.0933)
0.6991
0.7797 (+0.0606)
KMSE
CC-MIRT
MIRT
0.3948 (.0.0954)
0.4902
0.4821
0.7965 (-0.0856)
0.3973 (-0.091)
0:4884
0.4916
0.3973 (-0.0943)
0,5013
0.3983 (-0.1029)
0.3999 (-0.0950) (-0.999-)
0.4949
AUC
NCD
0.6885
0,6846
0.6797
0,67877
0,6736
0.6675
CC-NCD
0,7106 (+0.0221)
0.7061 (+0.0215)
0.7028 (+0.0231)
0,7007 (+0.0221)
0.6953 (+0.0216)
0.6895 (+0.0221)
ACC
CC.NCD
NCD
0.7613
(+0.0246)
0,7662
0.7834 (+0.0172)
0,7682
(+0.01377
0.7606
(+0.0208)
0.7812 (+0.0177)
0,7635
0,7640
0.7817 (+0.0177)
RMSE
NCD
0,4109
0,4095
0.4081
0.4127
0.4121
0.4133
CC-NCD
0.3966 (-0.0143)
0,3973 (40.0122)
0.3981 (-0.0100)
0.3991 (-0.0136)
0,3998 (-0.0123)
0,4009 (-0.0124)
Pre-training stage. CCLMF aims to leverage valuable insights gleaned from the vast data in the source course. In the pre-training stage, a CDM trained in the source course, generating an informative student representation u². Specifically, our CCLMF can be employed on CDMs which characterize student proficiency as:
- HCDM wp),
where Wh, denotes the parameters of HCDM. and HCDM is a function abstracted from CDM which can calculate each student's latent proficiency representation u. The dimensionality of vector uş is d which is usually related to the number of knowledge concepts and the number of problems. The generated u contains personalized and auxiliary information that can be utilized in the meta stage.
Meta stage. Due to individual differences, the relationships between student proficiency the source and target courses can vary significantly from one student to another. Therefore we need to create personalized mapping functions for each student in order to retain students' individual characteristics. In the meta stage, utilized a meta network to learn personalized mapping functions for each student. The meta network G(-) is formulated as:
Wu, = =Guf, w.y)
(2)
where w, is the parameters of G(-) and wn; is used as the parameters of the mapping function. The personalized mapping function (- w.) then produces personalized transformed student's representation in the target course as:
u Fu, (u,; W..).
Instead of mapping-oriented optimization used by Man et al. 18 we directly used the performance of diagnostic tasks as our optimization goal to train the meta network. This task-oriented training procedure advances in making full use of the ground truth values rather than approximate intermediate results. Therefore, the meta network G(+) as well as the mapping function Fu w.) are trained together using data in the target course.
Given the ground truth value from R' and the CC-CDM's final output i which is generated based on u the task-oriented loss can be formulated as:
lossCC-CDM = Σ log + +11 r.)) log (1 - r.))
Inference. The goal of CCLMF is to accurately measure students' level of proficiency in the target course. During inference, for any student sj who is new to the target course, their level of proficiency can be determined by their personalized transformation function ui = Fu, w.,).
The whole procedure of CCLMF is summarized in Algorithm| T Detailed implementation of CCLMF on NCD [27 can be found in Appendix C.
4.3 Experiment Settings
We conducted CCLMF on a traditional cognitive diagnosis model MIRT 24 and a deep-learning- based model NCD (27,, called CC-MIRT and CC-NCD respectively.
8

---

Datasets. We constructed the cross-courss course datasets based on the datasets shown in Table 4 As depicted in Figure aa), the correlation coefficient between students' performance in Python Pro- gramming (Python) and Java Programming (Java) is 0.65, indicating relatively high correlation. This high correlation makes these two courses well-suited for addressing cold-start problems. Python Programming was chosen as the source course. To simulate cold-start scenarios, 332, 568 response logs (7% of each student's 's response logs) from Java Programming were selected to form the target course. We reserved 20% of response logs in the target course the test set and then conducted experiments on sub-datasets with varying degrees of sparsity. Specifically, we randomly dropped 10%, 20%, 30%, 40%, and 50% of the remaining data and split the resulting data into 7/1 as train/valid set. To reduce the influence of randomness, we repeated the dropout process 10 times for each dropout ratio, then reported the average results across the 10 sets of data.
Settings and Metrics. The meta network was implemented as a two-layer perceptron with input-size K output-size K* x K" and hidden-size 100, where each linear layer was followed by RELU activation function. The weights were initialized by Xavier [10]. The batch size and learning rate are 128 and 1 x 10-3 All experiments were executed on Linux server with two GeForce RTX 3090s. The evaluation metrics include Area Under the ROC Curve (AUC) (2), Prediction Accuracy (ACC) and Root Mean Square Error (RMSE).
4.4 Experimental Results and Analysis
Table 6 shows the superiority of CCLMF over baseline models. CCLMF has better performance on both NCD and MIRT models with different dropout ratios on all metrics. Specifically, CCLMF achieves average improvement of 4.2% AUC, 5.5% on Accuracy, and 5.3% on RMSE.
To investigate the performance of our model over students of different sparsity levels, we show the performance with respect to the number of response logs student has in Figure O Note that we did not re- re-train the model with different sets of students, instead we divided the test set into different groups by the number of logs per student. We observe that the performance improvement of CCLMF is more significant for students with fewer response logs, highlighting the advantage of our model in cold-start scenarios.
drapost ratio=6.0
dropout ratlo=6.1
drapout ratio-0.22
0.75
0.75
5.75
RCD
MCD
NCD
0.70
0.70
5.70
0.83
0.88
5.11
32
52
53
54
56
59
69
52
53
24
25
57
58
59
51
52
53
54
65
96
10
drapout ration0.3
dropout ratlon0.4
dropost ratlono.5
0.75
0.75
6.75
RCD
MCD
NCO
0.70
0.70
5.70
0.5
0.88
0.51
16
50
32
52
14
59
51
52
groups of students with different number of logs
Figure 6: AUC on NCD and CC-NCD w.r.t. student groups with varying degrees of sparsity.
5 Experiments on CD and KT Tasks
Table 7: Datasets for CD and KT.
To demonstrate PTADisc's support for CD and KT we specifically selected four courses with varying response log scales, namely Probability and Statistics (Probability), Linux System (Linux), Database Technology and Ap- plication (DB), and Computational
Course Name
Probability
Linux
Comp
#Students #Problems #Concepts #Response Logs
557
4.398
12,646
45,329
1,054
2,678
3,616
8,399
247
284
325
477
46,
391,434
2,363,206
6,504,414
Logs per Student
82.78
89.00
186.87
143.49
Thinking (Comp), as seen in Table 7
which were also marked in Figure (b) with vertical dashes.
5.1 Baseline Models
For CD, we considered the following baseline methods: three traditional methods DINA 5
26
IRT (6), and MIRT [24], deep-learning method NCD [27] and graph based method RCD 8 For

---

KT, we considered the following baseline methods: deep sequential method DKT [21]. memory augmented method DKVMN [32], attention- based methods AKT [9] and SAKT 20], graph based methods GIKT 30 and SGKT [29]. and pre-training based method PEBG [66] Due to the intensive
computing resources required by RCD, we did not conduct RCD on two courses with relatively
large-scale data, which are DB and Comp. Additional information of these baseline models can be found in Appendix D.
5.2 Experiment Settings and Metrics
For each baseline model, we employed the same parameter settings and optimization methods as described their respective papers to ensure fairness in the comparison. The evaluation metrics used for CD included AUC, ACC, and RMSE, while for KT, the evaluation metrics were AUC and ACC, which are consistent with commonly used metrics in the literature.
5.3 Experimental Results and Analysis
The results of the CD) and KT tasks have yielded notable findings, as shown in Table 8 Firstly, we observed that the dataset size has a significant influence on model performance, with higher prediction accuracy observed for larger datasets. Secondly, we noted in Table that the MIRT model performance was generally higher than that of the NCD model; this contrasts with the findings in T We attribute this discrepancy to the cross-course dataset's lower average number of student response logs (i.e. approximately 10 logs per student), compared to the larger dataset with over 80 logs per student, as shown in Table 7 These results suggest that the number of logs per student significantly influences model performance, and that the NCD model may outperform the traditional MIRT model specifically in cases where there are fewer student logs on average. Thirdly, PEBG achieved the best AUC performance across all datasets in the KT task, which demonstrates the effectiveness of employing pre-training techniques capture the heterogeneous educational data.
Table 8: Baselines of existing CD and KT models on PTADisc.
Dataset Name
Metric
DINA
IRT
MIRT
NCD
RCD
DKT
DKVMN
SAKT
AKT
GIKT
SGKT
PEBG
AUC
0.6569
0,7092
0.7485
0.6815
0.7086
0.7079
Probability
ACC
0.6121
0,7097
0.7112
0.6894
0.7196
0.7025
0.6961
0.6721
0.6978
0.6883
0.6938
0.6977
RMSE
0,5141
0.449)
0.4730
0.4590
0.4346
AUC
0.7577
0.8171
0.7856
0.7756
0.8156
0.8379
Linux
ACC
0.7053
0,7802
0.7799
0.7755
0,7860
0.7726
0.7713
0.7665
0.7849
0.7791
0.7801
0,8040
RMSE
0.4493
0,3900
0.4023
0.3934
0
0.7141
0.8121
0.7901
0.7813
0,7635
0.7562
0.8101
0.7996
0.8381
DB
ACC
0.7856
0.8322
0.8424
0,8299
0.8312
0.8281
0.8181
0.8383
0.8353
0.8299
0.8373
RMSE
0,3934
0,3495
0.3407
0,3504
AUC
0.7137
0.7819
0.809
0.7734
0.7978
0,7811
0.7717
0.8091
0.8091
0.8172
0,8281
Comp
ACC
0.7303
0.7929
0.8018
0.7880
0.8276
0,8234
0,8012
0.8283
0.8337
0.8274
0.8194
RMSE
0.4335
0.3808
0.3749
0.3849
6 Conclusion
Outlook: We describe concrete ongoing and future work based PTTADicc. Specifically,
We will conduct research about adaptive learning and personalized educational planning, and incorporate them into personalized learning system alongside the CCLMF model shown in Figure I a). We will also analyze the group-level student learning behaviors.
We will explore programming knowledge tracing based on PTADisc which contains large amount of multi-round programming problem submission records and rich evaluation information.
Limitation: The original content text of the problems cannot be shared currently due to copyright constraints. We intend to disclose the content by extracting text features in the future.
Conclusion: PTADisc is a diverse, immense, cendent-centered and cross-course dataset that enables researchers to conduct previously infeasible cross-course studies. Based on PTADise, we developed CCLMF to alleviate the difficulty of diagnosing student knowledge states in the cold-start scenario. Furthermore, we demonstrate the broad range of applications of our dataset by reporting on the performance of baseline models for CD and KT tasks over PTADisc.
10

---

Acknowledgments and Disclosure of Funding
The authors wish to acknowledge the generous support and contributions of Hangzhou PAT Education Technology Co. Ltd. for this research. This work was supported by the National Natural Science Foundation China (No.62037001, No.62307032, No. No.62293550), Shanghai Rising-Star Program (23QAI National Key R&D Program of China (No. 20222D0117106 and the Starry Night Science Fund at Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010)
References
[1] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In ICDM, pages 42-51. IEEE, 2020.
[2] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Puttern recognition, 30(7):1 145-1159, 1997.
[3] Song Cheng, Qi Liu, Enhong Chen, Zai Huang, Zhenya Huang, Yiying Chen, Haiping Ma, and Guoping Hu. Dirt: Deep learning enhanced item response theory for cognitive diagnosis. In CIKM, pages 2397-2400, 2019.
[4] Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. Ednet: A large-scale hierarchical dataset in education. In Artificial Intelligence in Education, pages 69-73. Springer, 2020.
[5] Jimmy De La Torre. Dina model and parameter estimation: didactic. Journal of educational and behavioral statistics, 34(1):115-130, 15-130, 2009.
[6] Susan E Embretson and Steven P Reise. Item response theory. Psychology Press, 2013.
[7] Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with online system that tutors as it assesses. User modeling and user-adapted interaction, 19:243-266, 2009.
[8] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Su. Red: Relation map driven cognitive diagnosis for intelligent education systems. In SIGIR, pages 501-510, 2021.
[9] Aritra Ghosh, Neil Heffernan, and Andrew S Lan. Context-aware attentive knowledge tracing. In SIGKDD, pages 2-309 2020.
[10] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In AISTATS, pages 249-2549 JMLR Workshop and Conference Proceedings, 2010.
[11] Zhenya Huang, Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In CIKM, pages 1261-1270, 2019.
[12] Stamper J., Niculescu-Mizii A., Ritter S., Gordon G.J., and Koedinger K.R. Challenge data set from kdd cup 2010 educational data mining challenge. Find it at athttp://psscdatassh web. cmu, edu/ www//dDDoo /tDC//DDC/p/pppp hwtp://wwwwwdd.o.//m/mmmmmmmmm.m jsp/ 2010.
[13] Jiatong Li. Fei Wang, Qi Liu. Mengxiao Zhu, Wei Huang, Zhenya Huang. Enhong Chen, Yu Su, and Shijin Wang. Hieredf: A bayesian network-bar hierarchical cognitive diagnosis framework. In SIGKDD, pages 904-913, 2022.
[14] Qi Liu. Towards a new generation of cognitive diagnosis. In IJCAI, pages 4961-4964, 2021.
[15] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Exercise-aware knowledge tracing for student performance prediction. TKDE, 3301):000-115, 2019.

---

[16] Yunfei Liu, Yang Yang, Xianyu Chen, Jian Shen, Haifeng Zhang, and Yong Yu. Improving knowledge tracing via pre-training question embeddings. arXiv preprint ar5502201..0003, 2020.
[17] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang, and Weiqi Luo. pykt: A python library to benchmark deep learning based knowledge tracing models. arXiv preprini arXiv:2206.
1460, 2022.
[18] Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. Cross-domain recommendation: An embedding and mapping approach. In ICCAI, volume 17, pages 2464-2470, 2017.
[19] Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. Prerequisite relation learning for con- cepts in mooes. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1447-1456, 2017.
[20] Shalini Pandey and George Karypis. self-attentive model for knowledge tracing. 12th In- ternational Conference on Educational Data Mining. EDM 2019, pages Educational Data Mining Society, 2019.
International
[21] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. NIPS, 28, 2015.
[22] Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105./200 2021.
[23] Georg Rasch. Studies in mathematical psychology: I. probabilistic models for some intelligence and attainment tests. 1960.
[24] Mark D Reckase and Mark fl) Reckase. Multidimensional item response theory models. Springer, 2009.
(25] Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, and Shijin Wang. Convolutional knowledge tracing: Modeling individualization in student learning process. In SIGIR, pages 1857-1860, 2020.
[26] Matthias Von Davier. The dina model constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67(1)499-7,, 2014.
[27] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang. Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In AAAI, volume 34, pages 6153-6161, 2020.
[28) Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobate Lobato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Instructions and guide for diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007. 12061, 2020.
[29] Zhengyang Wu, Li Huang, Qionghao Huang, Changqin Huang, and Yong Tang. Sgkt: Session graph-based knowledge tracing for student performance prediction. Expert Systems with Applications, 206: 17681, 2022.
[30] Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang and Yong Yu. Gikt: gasph interaction model for knowledge tracing. In PKDD, pages 299-315. Springer, 2021.
[31] Jifan Yu, Yuquan Wang, Qingyang Zhong, Gan Luo, Yiming Mao, Kai Sun, Wenzheng Feng, Wei Xu, Shulin Cao, Kaisheng Zeng, et al. Mooccubex: a large knowledge-centered repository for adaptive learning in mooes. In CIKM, pages 4643-4652, 2021.
[32] Jiani Zhang, Xingjian Shi, Irwin King, and Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In WWW, pages 765-774, 2017.
12

---

[33] Renyu Zhu, Dongxiang Zhang, Chengcheng Han, Ming Gaol, Xuesong Lu, Weining Qian, and Aoying Zhou, Programming knowledge tracing: A comprehensive dataset and new model. In ICDMW, pages 298-307. IEEE, 2022.
[34] Yongchun Zhu, Zhenwei Tang, Yudan Liu, Fuzhen Zhuang, Ruobing Xie, Xu Zhang, Leyu Lin, and Qing He. Personalized transfer of user preferences for cross dromind recommendation. In WSDM, pages 1507-1515, 2022.
13

---

PTADisc: A Cross-Course Dataset Supporting Personalized Learning in Cold-Start Scenarios
(Appendix)
Contents
A
Related Work
A.I
Cognitive Diagnosis and Knowledge Tracing
A.2 Cross-Domain Recommendation
A.3
Other educational applications
Dataset Statictics
Implementation Details
4
Baseline Model Details
Supplementary Experiment Results

---

Our code, dataset, and detailed description for the dataset are available at httph::t//giiii com/ wahr0411/PTADisc.ee
A
Related Work
A.1 Cognitive Diagnosis and Knowledge Tracing
The goal of cognitive diagnosis (CD) is to assess students' level of proficiency of different knowledge concepts through the prediction process of student performance, given students' exercise records, aka response logs. Traditional psychometric-based methods include Item Response Theory (IRT) [8].
MIRT [9] and DINA 4 22)- These methods depend manually designed functions and the effectiveness requires a large number of psychological experiments to verify, which are labor-intensive and lack of ability to capture complex relationships between students, problems, and knowledge concepts. Recent years, with the development of artificial intelligence [10]. deep learning- based cognitive diagnosis models have been developed 23
28 Specifically, NCD (23] incorporates
neural networks to learn the complex exercising interactions. And RCD [ models the interactive and structural relations via a multi -laye--llrrrrrrr student-problee pronlem- relation map.
The goal of knowledge tracing (KT) is to dynamically model students' knowledge proficiency through her historical learning records, so to predict her performance new problems. Traditional proba- bilistic KT models assume students' knowledge state as a set of binary variables where each variable represents whether a student masters an individual concept or not, such as Bayesian Knowledge Tracing (BKT) [3] Recent years, deep tearning-basee KT models are proposed for learning valid rep- resentations especially when large amounts of data are available, such as DKT (21 and DK VKM [26). Further, from the perspective of model structure, a few methods based on transformers (SAKT 15
GNNs (SGKT ), and pre- pre-training frameworks (PEBG are proposed.
A.2 Cross-Domain Recommendation
Cross-domain recommendation is promising method to alleviate data sparsity and the cold-start problem [30 27 2 Several models have been proposed, including CMF (20]. which uses shared parameters for all domains, and CST [4], which transfers knowledge about users and items from auxiliary data sources. Mapping-based methods have been shown to be effective in solving cold-start recommendation problems [1]. by learning a mapping function from the source domain to the target domain. However, these methods have limited generalization ability for cold-start items users. To address this issue, TMCDR (29) introduces meta learning to improve the generalization ability and PTUPCDR (30 further improves TMCDR by learning personalized bridges for each user. While the cross-domain problem has been widely explored in the recommendation domain, there is limited research cross-course learner modeling in personalized learning.
A.3 Other educational applications
Prerequisite discovery refers to the task of identifying and establishing the sequence or order in which concepts or topics should be learned or presented, ensuring that foundational concepts are understood before more advanced ones (13). Suppose a MOOC corpus is composed by n courses in the same subject area, denoted as D = (D,,- D,, ... D. where D; signifies an individual course. Course concepts are subjects taught in the course, i.e., the concepts not only mentioned but also discussed and taught in the course. Let us denote the course concept set of D as == Ci U... U Cn, where C, representing the concepts intrinsic D; Prerequisite relation learning in MOOCs is formally defined as follows. Given a MOOC corpus D and its corresponding course concepts C, the objective to learn a function P C² + {0, }} that maps a concept pair (a, b). where b E C, to binary class that predicts whether serves as a foundational prerequisite for concept b.
Computerized adaptive testing (CAT) is an emerging testing format in many standardized examina- tions, aiming to rapidly and accurately diagnose candidate's 's level of knowledge mastery through personalized test items M. Let's conceptualize a set of students represented by S {s $2, .2,.... ... S, a problem set represented by P {pI, [pi.P2... .......... Pas) } and a set of knowledge concepts represented by C= {c1, C2..... , CK related the problems. We denote the record of student s; answering problem Pj as a triplet rıj = (s;, Pj. aij), where ai equals if s; answers Pj correctly, and 0 otherwise. Problem set P is divided into tested set Pr and an untested set Pt. When introduced to a novel student 8; E S, problem pool P with knowledge concepts C, the challenge is to architect a strategy A to select a X- -ize question set Pr = (pi, P2: ...,. Px } step by step that has the maximum quality and diversity. Prior to the testing phase, we set up an abstract cognitive diagnosis model M with
2

---

parameters 0 capturing knowledge states. During testing, at step t(1 - t X), we select one question př A (Pu,M), then observe new interaction test record r (st, P? aig) and update the knowledge states, i.e., 0. in M instantly. After testing, we measure the effectiveness of A by computing (AA) and Cov(A), where (A) denotes the measurement of quality and Cov(A) denotes the measurement of diversity.
Educational recommendation lies in constructing recommend system that can process the inter- actions between students and questions. This system should be capable of making appropriate learning suggestions students [8]. In the context of a digital educational platform, assume there are N students and P problems. We record the exercising process of certain student E N. where pt E represents the problems that stu- dent practices at her time step t, and denotes the corresponding score. Conventionally. a correct response to problem pt is denoted by ra equals to I, and an incorrect response by ri equals to O. Each problem p E P is characterized by a triplet its text content as word sequence p = [W1, W'2, ..... ww ). e E describes its knowledge concept coming from all K concepts. And d means its difficulty factor.
{(11 r1). (P2, 12) .... (pe.re)
{w, d). Specifically, the element w represents
B Dataset Staticties
30
26
30
27
23
#course
21
#course
20
17
20
12
12
10
10
10
(0,
1e4] (le4,
1e2]
(le2, 1e3] [1e3, (5e3,
#student
#problem
(a)
(b)
34
30
28
29
35
30
#course
25
# #course
25
25
20
15
15
10
7
(100, 500] (500,
1e5] (1es, 1e6] (1e6, (1e?, #non-programming log (d)
#concept
(c)
Figure 1: Distribution of the number of students, problems, concepts and non-programming logs courses in PTADisc.
3

---

Table 1: Detailed statistics of 68 courses in PTADisc.
Astudeni
Eprobleem
Reoncept
knom programming
Aprogramming log
Ct+ progtamminng
20,786
110,641,195
4,058.000
Camputer App. foundation
37,250
7,M34
527
110,200,87
216.146
programming
1,04,901
43,140
1,009
97,385 5636
164,459,382
DS. and algotihh anilysis
294,236
29,914
897
11 01.667.66
Python programming
21,8272997
Java peogramming
198,335
24,524
23,5422208
8,461, 9/30
Computational thinking foundatioo
34,519
4,501
401
16,571 A472
2,212
Informarion technology
36,700
6,499
516
9,935 894
159
Computational thinking
45,329
8,399
6,504,414
19,085
Datahase principle
33/031
10,428
5,219.955
1.3
Information processing technology and App.
17.152
2,428
221
3,127,821
7.536
Computer netwodk
20,333
7,572
609
2,451,211
1,023
Database technology and App.
12,645
3,616
325
2.363,200 206
Introduction computer science
Operating system
18.237
8.563
726
1,777,233
64,9477
Objecct orionted po programming jaci
7,552
2,993
315
944,474
195.830
Principles of compater composition
11,099
5,474
475
755,90 982
34,456
Web froes-end technology
6.612
4,247
572.799
876
Compilation principle
Thinking by data
2.597
1,027
115,636
Multivarate statistiaal analysas
1,704
790
83
416,427
62,993
Lizux
4,398
2,672
284
491,344
1,026
Computer problem solving
14,411
Sallware ongineering
4.197
2,815
270,692
Assembly longuage programming
2,330
1,301
2066
200.231
26
Machine learning
2,073
1,984
331
197 A85
500
Csharp programming
2,134
76,053
Java
Big data processing technology
1.211
1,055
165,573
VB programming
2,032
1,273
113
1573360
$.829
Discrete muthematics and
3,516
1,278
123
147,522
17,855
Digital image processing
142.197
English
1,952
660
19
328
Saltware preject management
850
1,753
133
133,874
M713
Scala programming
1,378
616
124
127. 382
9,367
Literature history
2,306
886
31
126,748
mathematics
92
Fortran programming
1.398
858
123,198
248 698
Intro algocithn competition
2,699
1,403
229
19.565
190,772
Data warchouse and data mining
689
V7
117,717
Practice staristies
35
Principles infomatioo security
33,303
Sallware design and architocture
496
360
43
82, 141
16,658
Single chip microcomputer principle and App.
784
SQS
88
80,977
Network programming technology
1,236
362
Digital logic
Introduction computer
1,301
546
99
63,359
Numerical analysis
610
1,111
241
$4.558
17,348
Big data maragement
217
640
65
$1,724
Psychology
Probubility
1,054
1,413
Problem solving fundaaiio
1,037
373
63
38.440
323
Artificial intelligence
582
353
77
28,088
Intro artificial intelligence
740
301
58
24,399
Software testing quality assurancc
Lincur algebra
194
420
17.341
8.81L
PHIP peogramte argrainnnng
165
632
153
14,62
3110
Objece oriented amilysi and design
265
178
47
11,572
Introduction
things
14
Microcomputer principle interface
Signals and systems
378
38
12
8,502
Calculus
154
357
120
7.024
9.085
Malab simulation
219
76
10
6,030
6,089
5,375
Computer system Sundamenials
67
25
3,643
Introduction cloud computing
101
59
39
1,571
Wircless network
60
S4
26
2,242
Swilt programming
14
2,170
Data
2,105
Fundomenstas of analogy electron technique
36
19
1,476
Polities
56
29
LA50
Tourism
22
1,320
requirement analysis design
20
Huskell programming
302
C Implementation Details
CCLMF model-agnostic framework that can be applied to various CD or KT models. Here, we take NCD as an example and showcase the implementation details of CCLMF based on NCD, namely CC-NCD. After pre-training the NCD model in the source course, the student's proficiency representation in the source course be obtained by extracting the corresponding from the

---

matrix A given the student ID i:
u NCD [1]
where A is the student representation matrix learned by NCD.
In the meta stage, we used a two-layer perceptron (MLP) as the meta network. This meta network then generates transformation matrix for each student the personalized mapping function:
T K (LP(uu;:)) 8),
where 0 is the parameters of MLP, and T x is the transformation matrix. K and K denote the dimensionality of the student proficiency representation in the source and target course respec- tively. Specifically, the dimensionality of the student representation is determined by the number of knowledge concepts considered.
The transformation matrix T K is then used to map student proficiency representation to the target course using matrix multiplication:
m = un TR*XK.
(3)
The final output é CC-NCD formulated as:
f L(Q' (u) - 1 1oiff) x didiee, 0t),
(4)
where O E {0, 10 is the concept relevancy of the problem p in the target course. diff (0, 10,1)xxk, diiio (0,1) denotes concept difficulty and problem discrimination learned from the NCD model using data of the target source. L(-) denotes the Linear Layers in NCD which is shown full paper Figure 5 and 0; the parameters of L(-).
Given the ground truth value r from R', all learnable parameters are trained together with the meta network and mapping function by optimizing the cross-entropy loss function as:
lossCC-NCD Σ (re log + rr:) log (1 fz)).
During the inference stage, given cold-start student 8j in the target course, we can get the latent proficiency representation in the target course as:
ut ARc
(6)
which can be utilized to predict the student's performance in the target course via Equation (4).
D Baseline Model Details
Cognitive diagnosis models:
DINA 44, [2] is a traditional method that is well-suited for binary scoring items, and it can effectively account for student errors due to guessing slipping.
IRT 5 is an important psychological and educational theory rooted in psychometrics, which employs linear function model the features of both students and problems.
MIRT 19 is a multidimensional extension of IRT, modeling multiple knowledge proficiency.
NCD X
is the first attempt to introduce neural networks for Cognitive Diagnosis, which can model high-order and complex sudent-problen interaction.
RCD 66 models the interactive and structural relations via a multi-layer guudent-problem-comee relation map and infers students' proficiency through the representations from this map.
Knowledge tracing models:
DKT [17] is the first approach applying deep learning to knowledge tracing tasks, making use the recurrent neural network in the process of modeling students' behavior.
DKVMN 26 makes use of memory network, static matrix to store all concepts and dynamic matrix to update students' knowledge states of those concepts.

---

SAKT [16]
employs a self-attention mechanism to capture the connections between exercises and student responses.
AKT 7 utilizes an attention mechanism to analyze the temporal gap between questions and students' prior interactions better understand their past engagement.
GIKT [25] makes use of a bipartite graph to model the input information, namely problems and concepts, and uses graph convolutional neural network (GCN) to process the data. Then the results were sent to LSTM and get the final prediction.
SGKT [24] uses a session graph and during the process of students' answering, a gated graph neural network was set to handle the problem-coneeee graph.
PEBG [ utilizes pre-training model to get the low-dimensional problem embeddings and models the relation between problems and concepts as a bipartite graph.
E Supplementary Experiment Results
We conducted experiments between the selected five courses in Figure 4(a) of the paper. The experimental results are presented in Table 2 due to its wide range of correlation coefficients with other courses. The source courses are marked within brackets and are ranked from lowest to highest correlation coefficient with C++ Programming: 0.54 for Python Programming (Python), 0.59 for Data Structure and Algorithm Analysis (DS), 0.64 for i (Java), 0.79 for Programming (C). To simulate cold-start scenarios, we sampled 5% of each student's s response logs in C++ Programming to form the target course.
We chose C++ Programming as the target course
From Table we can observe that CCLMF achieves certain improvement of the two models in all source courses. Notably, the results of the NCD model reveal that the extent of model improvement is related to the correlation coefficient between the source course and the target course. The source course with the highest correlation coefficient (0.79 for C Programming) exhibits the most significant improvement, while the source course with the weakest correlation coefficient (0.54 for Python Programming) demonstrates relatively less improvement.
Table 2: CCLMF results on MIRT and NCD, taking C++ Programming the target course. Source
courses are marked within brackets.
Metries
Model
dropout
10% dropout
20% dropout
30% dropout
40% dropout
50% dropost
0.6272
0.6218
0.6051
CC-MIRT (Python)
0,7150 (+0.0878)
0.7123 (+0.0905)
0.7102 (+0.0945)
0.6864 (+0.0780)
0.6801 (+0.0750)
0.6716 (+0.0659)
AUC
DC-MIRT (DS)
0.6912 (40.06-0)
0.6933 (-0.0715)
0.6901 (+0.0747)
0,7012 (+0.0888)
0.7042 (+0.0991)
0.6990 (+0.0933)
OC-MIRT ((asa)
0,7132 08600)
0. 0,703 (+0.0885)
0.7065 (40.0908)
0,7004 (40) (ISK0)
0.6075
(824)
0.6890 (83)0833)
CC-MIRT (C)
0.6996 (+0.0724)
0.7021 (+0.0803)
0.6935 (+0.077S)
0.6912 (+0.0788)
0.6570 (+0.0819)
0.6768 (+0.0711)
MIRT
0.7493
0.7479
0.7479
0,7471
0.6825
0.6860
OC-MIRT (Python)
0.7738 (+0.0245)
0.7721 (+8.0242)
0.7714 (+0.0235)
0.7657 (.0.0186)
0.7637 (+0.0812)
0.7606 (10.07466)
ACC
OC-MIRT (DS)
0,7668 (40.0175)
0,7681 (=0.0200))
0,7665 (40.0186)
0,7694 (+4,0223)
0.7706 (+1.188T)
0.7685 (+0.0825)
CC-MIRT (Javs)
0.7719 (40.0226)
0.7707
(-4.80228)
0.7706
(40.0227)
(40.0001)
0.7671 (-0.18466)
(40.0781)
OC-MIRT (C)
0.7703 (+0.0210)
0.7719 (+0.0240)
0.7695 (10.0216)
0.7683(000022)
0.7671 (s0.0846)
0.7643 (10.0783)
MIKT
0.4919
0.4935
0.4935
0,4931
0.511
0.5106
CC-MRRT (Python)
0,4009 (-0.0909)
0.4016 (-0.0919)
0.4022 (-0.0913)
0,4104 (.0.0827)
0.4118 (-0.099))
0.4151 (-0.0955)
RMSE
CC-MIRT (DS)
0.4089 (-0.0830)
0,4074 (-0.0861)
0.4086 (-0.8849)
0,4052 (-0.0879)
0.4035 (-0.1075)
0.4054 (-0.1052)
CC-MIRT (Java)
0,4021 (-0.0898)
(-0:0908)
0.4035 (-0.9900)
(-0.0876)
0.4062 1018)
(-0. 1021)
OC-MIRT (C)
0,4050 (-0.086))
0,4039 (-0.0896)
0,4062 (-0.0873)
0.4071 (-0.0660)
0.4085 1025)
0,4112 (-0.0994)
NCD OC-NCD (Python)
0.6981
0.6960
0.6926
0.6873
0.6846
0.6791
0,7008 (40.0028)
0,6979 (40.9819)
0.66463 (40.0017)
0.6931 (40.0058)
06683 (40.0027)
0.6807 (40.0016)
ALCC
CC-NCD (DSS
0.7225 (+0.0244)
0.7189 (=0.0029))
0.7164 (40.0236)
0.7128 (.0.0255)
0.7078 (+0.0222)
(40.0206)
CC-NCD (Java)
0.7154 (10.0133)
0.7123 (+0.0163)
0.7079 (+0.0153)
0,7068 (+0.0185)
0.6989 (+0.0142)
0.6907 (40.01
CC-NCD (C)
0.7663 (+0.0682)
0.7627 (+0.0667)
0.7598 (+0.0672)
0.7541 (+0.0668)
0.7486 (+0.0640)
0.7423 (+0.0632)
NCD CC-NCD (Python)
0.7619
0,7602
0.7616
0.7552
0.7556
0.7558
0.7693 (+0.0074)
0,7675 (+0.0073)
0,7666 (+0.0050)
0,7674 (+0.0122)
0.7668 (+0.0112)
0.7668 (+0.01 J0)
ACC
CC-NCD (DS)
0,7747 (+0.0128)
77002 (+0.0100)
0.7677 (+0.0061)
(+0.0121)
0.7675 (+0.0119)
(+0.0113)
OC-NCD (Java)
0.7661 (40.004)
0. 76755 (+0.8093)
76767 (+0.0071)
0.7667 (.0.0111)
0.7668 (+0.0112)
0.7660 (+000102)
OC-NCD (C)
(+0.023))
0.7875 (+4.0273)
0.7833 (+0.0217)
0.7759 (+4.01207)
0.7734 (+1101788)
0,7710 (+0.0152)
NCD CC-NCD (Python)
0.4116
D.4124
0,4155
0.4174
0.4164
0.4156
0.4102 (-0.0014)
0.4094 (-0:0030)
0.4111 (-0.0004)
0.4115 (-0.0059)
0.4223 (-0.0041)
0.4130 (-0.0026)
RMSE
CC-NCD (DS)
0,4018 (-0.0098)
0.4059 (-0:0065)
0,4081 (-0.0034)
(-0.0041)
0.4123 (-0:0041)
(0.0025)
CC-NCD (Java)
0,4088 (-0.0028)
0.4066
(-0.0058)
0.4075 (-0.0040)
0.4097 (-0.0077)
0.4104 (-0.0060)
0.4130 (0.0026)
OC-NCD (C)
0.3877 (-0.0239)
0.3881 (-0.0243)
0,3913 (-0.0202)
0.3956 (-0.0218)
0.3967 (-1.0197)
0.4028 (-0.0128)

---

References
[1] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In ICDM, pages 42-51. IEFE, 2020.
[2] Ye Bi, Liqiang Song, Mengqiu Yao, Zhenyu Wu, Jianming Wang, and Jing Xiao. Dedir: A deep cross-domain recommendation system for cold start users in insurance domain. SIGIR, pages 1661-1664, 2020.
[3] Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4):253-278, 1994.
[4] Jimmy De La Torre Dina model and parameter estimation: A didactic. Journal of educational and behavioral statistics, 34(1):1 1530 2009.
[5] Susan E Embretson and Steven Reise. Item response theory. Psychology Press, 2013.
[6] Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. Red: Relation map driven cognitive diagnosis for intelligent education systems. In SIGIR, pages 501-510, 2021.
[7] Aritra Ghosh, Neil Heffernan, and Andrew $ Lan. Context-aware attentive knowledge tracing. In SIGKDD, pages 2330-2339, 2020.
[8] Zhenya Huang, Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In CIKM, pages 1261-1270, 2019.
[9] Jiatong Li, Fei Wang, Qi Liu, Mengxiao Zhu, Wei Huang, Zhenya Huang, Enhong Chen, Yu Su, and Shijin Wang. Hieredf: A bayesian network-based hierarchical cognitive diagnosis framework. In SIGKDD, pages 904-913, 2022.
[10] Mengze Li, Tianbao Wang. Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Wenqiao Zhang, Jiaxu Miao, Shiliang Pu, and Fei Wu. Hero: Hierarchical spatio-temporal reasoning with contrastive action correspondence for end-to-end video object grounding. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3801-3810, 2022.
[11) Yunfei Liu, Yang Yang, Xianyu Chen, Jian Shen, Haifeng Zhang, and Yong Yu. Improving knowledge tracing via pre-training question embeddings. arXiv preprint arXiv:2012.05031, 2020.
[12] Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. Cross-domain rocommendation: An embedding and mapping approach. In ICAAI, volume 17, pages 2464-2470, 2017.
[13] Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. Prerequisite relation learning for con- cepts moocs. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1447-1456, 2017.
[14] Weike Pan, Evan Xiang, Nathan Liu, and Qiang Yang. Transfer learning in collaborative filtering for sparsity reduction. In AAAI, volume 24, pages 230-235, 2010.
[15] Shalini Pandey and George Karypis. A self-attentive model for knowledge tracing. arXiv preprint 1rXiv:1907.066 2019.
[16] Shalini Pandey and George Karypis. self-attentive model for knowledge tracing. 12th In- ternational Conference on Educational Data Mining, EDM 2019, pages 384-389. International Educational Data Mining Society, 2019.
[17] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. NIPS, 28, 2015.
[18] Georg Rasch. Studies in mathematical psychology: probabilistic models for some intelligence and attainment tests. 1960.
7

---

[19] Mark D Reckase and Mark fD Reckase. Multidinensional item response theory models. Springer, 2009.
[20] Ajit P Singh and Geoffrey Gordon. Relational learning via collective matrix factorization. In SIGKDD, pages 650-658, 2008.
[21] K Sohn, H Lee, X Yan, C Cortes, N Lawrence, and D Lee. Advances in neural information processing systems. NIPS, pages 3483-3491, 2015.
[22] Matthias Von Davier. The dina model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology. 67(1)449-7,, 2014.
[23] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. AAAI, volume 34, pages 6153-6161, 2020.
[24] Zhengyang Wu, Li Huang, Qionghao Huang, Changqin Huang, and Yong Tang. Sgkt: Session graph-based knowledge tracing for student performance prediction. Expert Systems with Applications. 206: 17681, 2022.
[25] Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang, and Yong Yu. Gikt: graph-based interaction model for knowledge tracing. In PKDD, pages 299-315. Springer, 2021.
[26] Jiani Zhang, Xingjian Shi, Irwin King, and Dit- Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In WWW, pages 765-774, 2017.
[27] Cheng Zhao, Chenliang Li, Rong Xiao, Hongbo Deng, and Aixin Sun. Catn: Cross-domain recommendation for cold-start users via aspect transfer network. In SIGIR, pages 229-238,
2020.
[28] Yuqiang Zhou, Qi Liu, Jinze Wu, Fei Wang, Zhenya Huang, Wei Tong, Hui Xiong, Enhong Chen, and Jianhui Ma. Modeling context-aware features for cognitive diagnosis in student learning. In SIGKDD, pages 2420-2422222 2021.
[29] Yongchun Zhu, Kaikai Ge, Fuzhen Zhuang, Ruobing Xie, Dongbo Xi, Xu Zhang, Leyu Lin, and Qing He. Transfer-meta framework for cross-domain recommendation to cold-start users. In SIGIR, pages 1813-1817, 2021.
[30] Yongchun Zhu, Zhenwei Tang. Yudan Liu, Fuzhen Zhuang, Ruobing Xie, Xu Zhang, Leyu Lin, and Qing He. Personalized transfer of user preferences for cross cross-domain recommendation. In WSDM, pages 1507-1515, 2022.


Reviews:
[{"review_id": "JwA3nKzMjx", "reviewer": "Reviewer_T6X2", "summary": "", "questions": "", "limitations": "1. The authors claim that the proposed dataset is student-centered. From lines 48-49, the word \"student-centered\" means student response logs. Every KT or CD related datasets have student responses. What's the speciality in this dataset? Furthermore, what's the real difference between \"diverse\" and \"immense\"?\n2. After checking the datasets, it seems all the datasets are CS related data, even for the non-programming datasets. I think the \"dataset diversity\" is over claimed.\n3. Section CCLMF is very confusing. In previous sections, the paper introduce a datasets contains lots of features. All of sudden, it comes to CCLMF which seems to have no directly relations with previous section. It's not clear how the CCLMF utilizes the rich information provided in the proposed dataset. The NCD model used in CCLMF is a bit old and only relies on very standard ID features.\n4. In Section 5.3, the authors reported several baselines of KT and CD. It seems these baselines don't consider the diverse and immense features in the proposed dataset. These results are disconnected with the features of PTADisc.\n\nMinors:\n1. Footnotes of PTA are duplicated at both page 2 and page 3.\n2. The courses in Figure 4(a) are Python, DS, C++, Java and C while courses in Figure 4(b) are compDB, Linux and Probability. I am a bit confused about this. Why they are different?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well motivated and easy to read and follow. \n2. The authors provide detailed data descriptions on the proposed dataset and show a modeling framework on this dataset. \n3. Several experiments are conducted and results are evaluated via different metrics. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0432fuD37p", "reviewer": "Reviewer_ToYw", "summary": "", "questions": "", "limitations": "The authors list the limitation of their work in the paper, which is that currently the original content text of the problems cannot be shared currently due to copyright constraints. They will address them by extracting text features in the future.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Reasonable data construction and solid experiment results for benchmarking.\n2. The authors investigated a new task of cross-course diagnostic task, which has never been explored in the area of AI for education.\n3. PTADisc is a unique dataset that evaluates student responses using a scoring ratio system rather than binary values.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LiovnsgdFB", "reviewer": "Reviewer_ieLM", "summary": "", "questions": "", "limitations": "The author has declared the limitation of the dataset in conclusion section. Hope they can release the text features in the future, which is important for knowledge tracing and cognitive diagnosis tasks.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed dataset is immense and valuable, benefiting  data mining community and the pedagogy community.\n2. PTADisc is the first dataset with cross-course information. The analysis of student latent proficiency relationships between courses makes a contribution to personalized learning.\n3. PTADisc is currently the largest educational dataset. It contains extensive concept-related information and detailed records of student behaviors, which is well-suited for diagnostic tasks.\n4. Experimental results demonstrate the effectiveness of the proposed method and the generalizability of the dataset. The paper is well-written and addresses important concerns, such as ethics issue.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Atzx9DuKIu", "reviewer": "Reviewer_SZNR", "summary": "", "questions": "", "limitations": "Lack of textual content of the problems in the dataset – as stated by the authors.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "First, it is a very big new dataset, with cross-course information. \nSecond, a model is presented for utilizing information across courses for proficiency prediction, which is can be useful in 'cold-start' cases, i.e. when not much is known yet about student performance in the target course.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3lSxLGF3WF", "reviewer": "Reviewer_G8A9", "summary": "", "questions": "", "limitations": "- From Figure 3, I can't infer what's the raw data and thus the final form of data. There needs more explanation on the raw data they sourced from the PTA platform\n- In section 3.2.3, if knowledge tracing and cognitive diagnosis dataset only differ on submitting time, why make them as two separate tasks?", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper has stated their motivation clearly and gives detailed argument of why adding cross-course component is necessary\n- They have tackled quite a few challenges in this domain, such as data coverage, concept annotations, and cross-course information. The efforts to get rid of these problem in a large scale data is appreciated", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviews are positive, although some reviewers struggled with fully understanding the paper. This seems to be a function of how the paper is written and presented. Reviewers have provided help on how to better explain to readers what the work is doing. Notably, the authors' main point that the dataset is focused on supporting "cross-course analysis" – and does not offer rich, diverse content outside of this paradigm – should be made more clear in the paper.  After discussions and clarifications, the reviews support the following ideas:

Strengths:
- Paper provides a new large, unique dataset.
- The dataset provides a novel type of information: detailed cross-course information.
- Dataset can be valuable to several communities (data mining, education).
- Dataset can be useful for several tasks.
- Authors demonstrate its utility for proficiency prediction well.

Opportunities For Improvement:
- Outline potential applications of the dataset across various tasks.
- Provide some analysis and examples on students’ behavior patterns across difference courses.
- Address issues with label definitions and their implications.

Limitations:
- Addressed well by authors.

Correctness:
- Reasonable.

Clarity:
- Reasonable, although there are suggestions throughout on what to improve in a final version.

Relation To Prior Work:
- Done well.

Documentation:
- Done well.
Decision: Accept (Poster)