Consistency and Monotonicity Regularization for Neural Knowledge Tracing
Authors: Seewoo Lee
Youngduck Choi
Juneyoung Park
Byungsoo Kim
Jinwoo Shin
Venue:  (2021)
Keywords: knowledge tracing
data augmentation
regularization

Abstract: Knowledge Tracing (KT), tracking a human's knowledge acquisition, is a central component in online learning and AI in Education. In this paper, we present a simple, yet effective strategy to improve the generalization ability of KT models: we propose three types of novel data augmentation, coined replacement, insertion, and deletion, along with corresponding regularization losses that impose certain consistency or monotonicity bias on model's predictions for the original and augmented sequence. Extensive experiments on various KT benchmarks show that our regularization scheme significantly improve the prediction performances, under 3 widely-used neural networks and 4 public benchmarks for KT, e.g., it yields 6.3% improvement in AUC under the DKT model and the ASSISTmentsChall dataset. 

Under review as a conference paper at ICLR 2021
CONSISTENCY AND MONOTONICITY REGULARIZA- TION FOR NEURAL KNOWLEDGE TRACING
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge Tracing (KT), tracking human's knowledge acquisition, is central component in online learning and AI Education. In this paper, we present a simple, yet effective strategy to improve the generalization ability of KT models: we propose three types of novel data augmentation, coined replacement, insertion, and deletion, along with corresponding regularization losses that impose certain consistency or monotonicity biases on model's predictions for the original and augmented sequence. Extensive experiments on various KT benchmarks show that our regularization scheme consistently improves the model performances, un- der 3 widely-use neural networks and 4 public benchmarks, e.g.. yields 6.3% improvement in AUC under the DKT model and the ASSIST menttsCall dataset.
INTRODUCTION
In recent years, Artificial Intelligence in Education (AIEd) has gained much attention as one of the currently emerging fields in educational technology. In particular, the recent COVID-19 pandemic has transformed the setting of education from classroom learning to online learning. As a result, AIEd has become more prominent because of its ability to diagnose students automatically and pro- vide personalized learning paths. High-quality diagnosis and educational content recommendation require good understanding of students' current knowledge status, and is essential model their learning behavior precisely. Due to this, Knowledge Tracing (KT), a task of modeling student's s evolution of knowledge over time, has become one of the most central tasks in AlEd research.
Since the work of Piech et al. (2015), deep neural networks have been widely used for the KT modeling- Current research trends in the KT literature concentrate on building more sophisticated, complex and large-scale models, inspired by model architectures from Natural Language Processing (NLP), such as LSTM (Hochreiter & Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) ar- chitectures, along with additional components that extract question textual information students' forgetting behaviors (Huang et al., 2019; Pu et al., 2020; Ghosh et al., 2020). However, as the number of parameters of these models increases, they may easily overfit small datasets and hurt model's ' generalizabiliy. Such an issue has been under-explored in the literature.
To address the issue, propose simple, yet effective data augmentation strategies for improving the generalization ability KT models, along with novel regularization losses for each strategy. In particular, we suggest three types of data augmentation, coined (skill-based) replacement, inser- tion, and deletion. Specifically, we generate augmented (training) samples by randomly replacing questions that student solved with similar questions or inserting/dellttng interactions with fixed responses. Then, during training, we impose certain consistency (for replacement) and monotonic- ity (for insertion/deletion) bias on a model's predictions by optimizing corresponding regularization losses that compares the original and the augmented interaction sequences. Here, our intuition be- hind the proposed consistency regularization is that the model's output for two interaction sequences with same response logs for similar questions should be close. Next, the proposed monotonicity reg- ularization is designed to enforce the model's prediction to be monotone with respect to the number of questions that correctly (or incorrectly) answered, i.e., student is more likely to answer correctly (or incorrectly) if the student did the same more in the past. By analyzing distribution of the previ- correctness rates of interaction sequences, datasets indeed have monotonicity properties- see Figure I and Section A.2 for details. The overall augmentation and regularization strategies sketched in Figure 2. Such regularization strategies
observe that the existing student interaction

---

Under review as a conference paper at ICLR 2021
TOTNO
1910
0.8
05
0.7
Prvioos correctness 190s
Presious
Presicun cormecteess rale
Previzze
(a) ASSISTments2015
(b) AeShSTmentsChall
(c) STATICS2011
(d) EdNet-KT1
Figure 1: Distribution of the correctness rate of past interactions when the response correctness of current interaction fixed, for knowledge tracing benchmark datasets. Orange (resp. blue) represents the distribution of correctness rate (of past interactions) where current interaction's sponse is correct (resp. incorrect), axis represents previous interactions' correctness rates (values in 00, IJ. The orange distribution lean more to the right than the blue distribution, which shows the monotonicity nature of the interaction datasets. See Section A.2 for details.
model's prediction
replacement
(02.
(Qg. 1)
(Qs.
(Q1'.
(O2.
(Qg'.
(O4. 0)
sequence
correct insertion
correct deletion
(Qr, 1)
(Q.,
(Oz. 0)
(Oa, 1)
(Qr.1)
(Oa, 0)
(O2. 0)
(Qg. 1)
(O4. 0)
Figure 2: Augmentation strategies and corresponding bias on model's predictions (predicted cor- rectness probabilities), Each tuple represents question id and response of the student's interaction (1 means correct). Replacing interactions with similar questions (Q1, Q2 to Q Q/3) does not change model's predictions drastically. Introducing interactions with correct responses (Q', Q") in- creases model's estimation, but deleting such interaction (Q1, 1) decreases model's estimation.
motivated from observation that existing knowledge tracing models' prediction often fails to satisfy the consistency and monotonicity condition, e.g., see Figure Section 3.
We demonstrate the effectiveness of the proposed method with 3 widely used neural knowledge tracing models DKT (Piech et al., 2015), DKVMN (Zhang et al., 2017b), and SAINT (Choi et al., 2020a) 4 public benchmark datasets ASSISTments2015, ASSISTmentsChal STATICS201 1, and MaNeetKKTI Extensive experiments show that, regardless of dataset or model architecture, our scheme remarkably increases the prediction performance 6.2% gain Area Under Curve (AUC) for DKT on the ASSISTmentsChall dataset. In particular, ours is much more effective under smaller datasets: by using only 25% of the CesSstllntthalll dataset, we improve AUC of the DKT model from 69.68% to 75.44% which even surpasses the baseline performance 74.4% with the full training set. We further provide various ablation studies for the selected design choices, e.g., AUC of DKT model on the ASSISTments2011 dataset is dropped from 72.44% to 66.48% when we impose 'reversed' (wrong) monotonicity regularization. We believe that our work can be a strong guideline for other researchers attempting improve the generalization ability of KT models.
1.1 RELATED WORKS AND PRELIMINARIES
Data augmentation is arguably the most trustworthy technique to prevent overfitting or improve the generalizability machine learning models. In particular, has been developed as an effective way to impose domain-specifee inductive bias to model. For example, for computer vision models, simple image warpings such as flip, rotation, distortion, color shifting, blur, and random erasing are the most popular data augmentation methods (Shorten & Khoshgoftaar, 2019). More advanced tech- niques, e.g., augmenting images by interpolation (Zhang et al., 201 7a; Yun et al., 2019) or by using generative adversarial networks (Huang al., 2018), have been also investigated. For NLP models,
2

---

Under review as a conference paper at ICLR 2021
it is popular to augment texts by replacing words with synonyms (Zhang al., 2015) or words with similar (contextualizee)) embeddings (Wang & Yang, 2015; Kobayashi, 2018). As an alternative method, back translation (Sennrich et al., 2016; Yu et al., 2018) generates augmented sentence by translating a given sentence into a different language domain and translate it back to the original domain with machine translation models. Recently, Wei & Zou (2019) show that even simple meth- ods like random insertion/swapddletion could improve text classification performances. In the area of speech recognition, vocal tract length normalization (Jaidl & Hinton, 2013), synthesizing noisy audio (Hannun et al., 2014), perturbing speed (Ko et al., 2015), and augmenting spectrogram (Park et al., 2019) are popular as data augmentation methods.
The aformentioned data augmentation techniques have been used not only for standard supervised learning setups, but also for various unsupervised and semi-supervised learning frameworks, by imposing certain inductive biases to models. For example, consistency learning (Sajjadi et al., 2016; Xie al., 2019; Berthelot et al., 2019; Sohn et al., 2020) impose a consistency bias to model so that the model's output is invariant under the augmentations, by means of training the model with consistency regularization loss (e.g. L--loss between outputs). Abu-Mostafa (1992: 1990) suggested general theory for imposing such inductive biases (which are stated as hints) via additional regularization losses. Their successes highlight the importance of domain specific knowledge for designing appropriate data augmentation strategies, but such results are rare in the domain of AlEd, especially for Knowledge Tracing.
Knowledge tracing (KT)) is the task of modeling student knowledge over time based on the student's
learning history. Formally, for given student interaction sequence (41,.... IT), where each 4 = (Qr, RR) is pair of question id Qt and the student's response correctness Rg E {0, 13 (1 means correct), KT aims to estimate the following probability
(1) i... the probability that the student answers correctly the question Qe at t-th step. Corbett & Anderson (1994) proposed Bayesian Knowledge Tracing (BKT) that models student's knowledge as latent variable in a Hidden Markov Model. Also, various seq2seq architectures including LSTM (Hochreiter & Schmidhuber, 1997). MANN (Graves et al., 2016), and Transformer (Vaswani et al., 2017) are used in the context of KT and showed their efficacy. Deep Knowledge Tracing (DKT) is the first deep learning based model that models student's knowledge states as LSTM's 's hidden state vectors (Piech et al., 2015). Dynamic Key- Value Memory Network and its variation can exploit rela- tionships between questions/skills with concept vectors and concept-state vectors with key and value matrices, which is more interpretable than DKT (Zhang et al., 90; Abdelrahman & Wang. 2019). 9). Transformer based models (Pandey & Karypis, 2019; Choi et al., 2020a; Ghosh et al., 2020; Pu et al., 2020) are able to learn long-range dependencies with their self-attention mechanisms and be trained in parallel. Utilizing additional features of interactions, such as question texts (Huang al., 2019: Pandey & Srivastava, 2020). prerequisite relations (Chen et al., 2018) and time information (Nagatani et al., 2019 Choi al., 2020a; Pu et al., 2020) another way improve performances. Recent works try to use graph neural networks (Nakagawa et al., 2019; Liu et al.; Tong et al., 2020; Yang et al., 2020b) and convolutional networks (Yang et al., 2020a; Shen et al., 2020) to model relations between questions and skills or extract individualized prior knowledge.
PlR,- = 1141, 12, 14.1::11.......1.....
2 CONSISTENCY AND MONOTONICITY REGULARIZATION FOR KT
For a given set of data augmentations A, we train KT models with the following loss:
Liot
Lort +
Σ
MaugCauu + Areg-suu Creg-sug);
augEA
where Lori is the commonly used binary cross-entropy (BCE) loss for original training sequences and Cang are the same BCE losses for augmented sequences generated by applying augmentation strategies aug E A.' Lreg-aug are the regularization losses that impose consistency and monotonicity bias on model's predictions for the original and augmented sequence, which are going to be defined in the following sections. Finally, Aaug. Areg-aug > are hyperparameters to control the trade-off among Cori, Laug, and Lreg-aug.
For replacement and insertion, do include outputs for augmented interactions in Laug.
3

---

Under review as a conference paper at ICLR 2021
In the following sections, we introduce our three simple augmentation strategies, replacement, inser- tion and deletion with corresponding consistency and monotonicity regularization losses, freg-reps Lreg-cor ins (or Lrog-incan incog ins) and Lreg-cor del (or Lrog-imcor del), respectively.
2.1 REPLACEMENT
Replacement, which is motivated by the synonym replacement in NLP, is an augmentation strat- egy that replaces questions in the original interaction sequence with other similar questions without changing their responses, where similar questions are defined as the questions that have overlapping skills attached to. Our hypothesis is that the predicted correctness probabilities for questions in an augmented interaction sequence will not change a lot from those in the original interaction sequence. Formally, for each interaction in the original interaction sequence (.,...., Ir), randomly decide whether the interaction will be replaced or not, following the Bernoulli distribution with the proba- bility Crep- If an interaction 4, = -Q², Rτ) with set of skills S, associated with the question Qu is set to be replaced, we determine | = (QueP R₂) by selecting a question Q with its associated set of skills StTP that satisfies S, n STTP + (. The resulting augmented sequence (ITeP ..... I, is generated by replacing I, with 11- for i E R C [T] {1, 2, 2....
........,, T, where R is set of indices to replace. Then we consider the following consistency regularization loss:
Lreg-rep E Pt [
where Pt and Pt de are model's predicted correctness probabilities for t-th question of the original and augmented sequences, respectively. We do not include the output for the replaced interactions in the loss computation. For the replacement strategy itself consider several variants. For instance, randomly selecting a question for Q, from the question pool an alternative strategy a skill set for each question is not available. It is also possible only consider outputs for interactions that replaced or consider outputs for all interactions in the augmented sequence for the loss computation. We investigate the effectiveness of each strategy Section 3.
2.2 INSERTION
When a student answers more questions correctly (resp. incorrectly), the predicted correctness probabilities of the KT models for the remaining questions should increase (resp. decrease).
Based on this intuition, we introduce a monotonicity constraint by inserting new interactions into the original interaction sequence. Formally, generate augmented interaction sequence (I1 ; (resp. 1::00 = (Qin,,) into the original interaction sequence (41...., .......IT IT) for t E I c [T'. where the question Qi is randomly selected from the question pool and I with the size Qins proportion of the original sequence is a set of indices of inserted interactions. Then our hypothesis is formulated as Pt S Patt) (resp. pt 2 pins where pt and p!"s are model's predicted correctness probabilities for t-th question of the original and augmented sequences, respectively. Here, a [T] - [T') I is the order-preserving bijection which satisfies I, Line for 1 < t < T. (For instance, in Figure 2,
...., 1 by inserting a correctly (resp. incorrectly) answered interaction 1 = (Q .1)
sends 21 3, 4) to (2.3, 4, 63) We impose our hypothesis through the following losses:
ins Greg-ror.ins BEE[T] [max((, (0,pr po(t))): )}
[max(0, freg-incor,ing [ ins
p))
(4)
where Creg-corinns and Creg-incor are losses for augmented interaction sequences of inserting cor- rectly and incorrectly answered interactions, respectively.
2.3 DELETION
Similar to the insertion augmentation strategy, we propose another monotonicity constraint by re- moving some interactions in the original interaction sequence based on the following hypothesis: a student's response records contain less correct (resp. incorrect) answers, the correctness prob- abilities for the remaining questions would become decrease (resp. increase). Formally, from the original interaction sequence (Iı,..., Ir), we randomly sample a set of indices D C (7). where R2 = (resp. R2 = 0) for i. E D. based on the Bernoulli distribution with the probability O'de-- We remove the index t E and impose the hypothesis Pt 2 ddl (resp. Pt S &Pol)) where Pt and Po(()
del model's predicted correctness probabilities for (-th question of the original and augmented
4

---

Under review as a conference paper at ICLR 2021
insertion deletion
insertion + deletion
dataset
model
augmentation
replacement
DKT
72.01
0.05
72.46 1 0.06
72.39 1 0.07
ASSIST2015
DKVMN
71.21 - 0.00
72.00
1 0.18
72.23
1 0.09
SAINT
72.13
72.78
+
72.81
DKT
74.40 + 0.36
75.98 +
79.07 + 0.08
ASSISTChall
DKVMN
74.46
0.11
- 75.06
78.21 - 0.05
SAINT
77.01
0.18
78.02 - 0.09
80.18
0.05
DKT
86.43
87.18 +
87.27 =
STATICS2011
DKVMN
84.89
0.17
85.65
0.94
87.17
0.14
SAINT
85.82 = 0.50
86.53 + 0.30
87.56
= 0.06
DKT
72.75 0.00
74.04 + 0.04
74.28 + 0.06
ETT
DKVMN
73.58
0.08
73.94 + 0.05
74.16
+ 0.11
SAINT
74.78 0.05
75.32
1: 0.05
75.26
+ 0.02
Table 1: Performances (AUCs) of DKT, DKVMN, and SAINT models on public benchmark datasets. The results show the mean and standard deviation averaged over 5 runs and the best result for each dataset and model is indicated bold.
15
ASSISTmentsChall
STATICS2011
EdNet-KTI
72
76
86
72
82
72
70
64
64
Figure 3: Performances with various sizes of training data under the DKT model. x axis stands for the portion of the training set use for training (relative the full train set) and y axis is the AUC. Blue line represents the AUCs of the vanilla DKT model, and red line represents the AUCs of the DKT model trained with augmentations and regularizations.
sequences, respectively. Here, a [7] D (T') is the order preserving bijection with /del for E [T] D. D. We impose the hypothesis through the following losses:
a(())
Creg-cor.del EgED [max (0, Pa(() pt)l, Lreg-incor.del EgED (0. Po(t)) (+)
del
del
(5)
where Creg-cor.del and Creg-incor.de losses for augmented interaction sequences of deleting rectly and incorrectly answered interactions, respectively.
EXPERIMENTS
We demonstrated the effecctiveness of the proposed method on widely used benchmark datasets: ASSISTTennt20055, ASSISTmentsChall, STATICS2011, and EdNet-KT1. ASSISTments datasets the most widely used benchmark for Knowledge Tracing, which is provided by ASSISTments online tutoring platform² (Feng et al., 2009). There are several versions of dataset depend on the years they collected, and used 25 and ASSISTmentsChall! ASSIST- mentsChall dataset is provided by the 2017 ASSISTments data mining competition. STATICS201 consists of the interaction logs from an engineering statics course, which is available on the PSLC datashop EdNet-KTI is the largest publicly available interaction dataset consists of TOEIC (Test of English for Interational Communication) problem solving logs collected by Santa (Choi al.,
httpp://nww..ssssttensssss
'htps://ssttes: bhtpps//sies.ggogll.ccmm.m htt.::/ww......iaaoaaoaoaaaaaaaa
"nuts:://stes.goee gowgle.oooooooooooooooooo......o wow////ww//ssssttnnnaaaaannnnnnn
hhupps//ainuoorsaata.comm

---

Under review as a conference paper at ICLR 2021
(800 00
(300 00
(000 00
(752 00
(752 0322.00
(752 00
(752 002.00
(754
(800
(000
(800
(800
(800
(000
(800
(800
(800
(752
(752
vanilla DKT
0.8
(960 00
03102 33
0900 00
0000 00
(9000 00
(800 00
129t 3)
(3783 3)
000n 00
113500 33
(409 3)
11302 33
(850 00
(1375 3)
coon 00
(3087 33
(900 00
0792 00
0992 00
(297 30
052 00
0732 00
0233 33
1122 00
(284
(0000
(994
11540
0860
(059
1757
(000 4)
0000 40
0000, 0)
0190 000000
00
(732, 4)
(75). 00
(755. 00
(755. 00
(755, 01
with
0800
0000
(800
(100
(800,
02
monotonicity
regularization
11
(8C0 01
(3122 11
(800 01
(060) 01
(aco, o)
(960 01
(2036) 11
(1PER, 11
(1380, 11
(ocu x1
(1226 11
11ss 11
19c0, on
(1375, 11
(G19, 11
(800 o)
(3087 11
(850, 0)
(8c0, 11
1732, o
(207 31
(244, 11
(752, 00
(752, 00
(8C0,
(880 (2775,
loco
(792
(752
Figure 4: Response correctness prediction for a student in the CSSisllentsShall dataset. We randomly insert interactions with correct responses (interactions with yellow boundaries). In case of the vanilla DKT model, the predictions for the original interactions (especially the interactions with green boundaries) such problem is resolved when we train the model with monotonicity regularization (with the loss foot Lori feco jis 100 freg-corr ins). Unlike the vanilla DKT model, predicted correctness probabilities for the original interactions are increased after insertion.
decreased, even if the student answered more questions correctly. However,
2020b). We reduce the size of the EdNet-KTI dataset by sampling 6000 users among 600K users. Detailed statistics and pre-pros prepprcesssng methods for these datasets are described in Appendix. With the exception of the EdNet-KT1 dataset, we used 80% of the students as training set and the remaining 20% as test set.
We test DKT (Piech et al., 2015), DKVMN (Zhang al., 2017b), and SAINT (Choi al., 2020a) models. For DKT, we set the embedding dimension and the hidden dimension 256. For DKVMN, key, value, and summary dimension are all set to 256, and we set the number of latent concepts as 64. SAINT has layers with hidden dimension 256, 8 heads, and feed-forwart dimension 1024. All the models do not use any additional features of interactions except question ids and responses as an input, and the model weights initialized with Xavier distribution (Glorot & Bengio, 2010). They are trained from scratch with batch size 64, and we use the Adam optimizer (Kingma Ba, 2015) with learning rate 0.001 which is scheduled by Noam scheme with warm-up step 4000 Vaswani et al. (2017) suggest. We set each model's maximum sequence size as 100 on 2SSISTments2015 & EdNet-KTI dataset and 200 on ASSISTmentsChall & STATICS2011 dataset. Hyperparameters for augmentations, O'aug, Areg-ang. and Aaug are searched over Gaug E {0.11 0.3, 0.5). Areg-aug E {1, 10, 50, 100), and Xaug E {0, 1}. For all dataset, we evaluate our results using 5-fold cross validation and use Area Under Curve (AUC) as evaluation metric.
3.1 MAIN RESULTS
The results (AUCs) are shown in Table 1 that compares models without and with augmentations, and we report the best results for each strategy. (The detailed hyperparameters for these results are given in Appendix.) The 4th column represents results using both insertion and deletion, and the last column shows the results with all augmentations. Since there's no big difference on perfor- mance gain between insertion and deletion, we only report the performance that uses one or both of them together. We use skill-based replacement if skill information for each question in the dataset is available, and use question-rant muestoon-random replacement that that selects new questions among all questios if not (c.g. ASSIST ASSISTments2015) As one can see, the models trained with consistency and mono- tonicity regularizations outperforms the models without augmentations a large margin, regardless of model's architectures or datasets. Using all three augmentations gives the best performances for most of the cases. For instance, there exists 6.3% gain in AUC on ASSISTmentsChall dataset un- der the DKT model. Furthermore, not only enhancing the prediction performances, our training scheme also resolves the vanilla model's 'ss issue where the monotonicity condition on the predictions of original and augmented sequences is violated. As in Figure 4, the predictions of the model trained with monotonicity regularization (correct insertion) are increased after insertion, which contrasts to the vanilla DKT model's outputs.

---

Under review as a conference paper at ICLR 2021
dataset
loss
replacement
correct insertion
incorrect insertion
correct deletion
incorrect deletion
ASSIST2015
(6)
72.03 + 0.06
71.98 + 0.06
71.98 + 005
72.05 = 0.04
72.04 + 0.02
(72.01 + 0.5)
(2)
72.06 + 0.03
72.09 + 0.06
72.35 0.11
72.53 = 0.08
72.26 = 0.01
ASSISTChall
75.13 + 004
74.61 +: 0.17
74.57 0.14
74.92 +: 0.12
74.42 0.20
(74.40 + Q16)
78.85 +
75.98 +
75.64 0.12
75.60 + 74.77 0.11
STATICS2011
86.89 + 0.23
86.45 + 0.26
86.40 + 0.22
86.53 + 86.55 + 0.25
(86.43 + 0.99
87.27
+ 0.11
86.72 + 0.23
87.18 + 0.12
87.07 + 0.33
86.97 + 0,26
EdNet-KTI
(6)
73.04
1 0.10
72.81 +: 0.08
72.88 0.09
72.99
1: 0.007
73.28 0.04
(72.75 a0s)
73.89 0.06
73.73 1: 0.06
73.52 0.06
74.04
+ 0.04
73.76
0.04
Table 2: Comparison of the performances (AUCs) of the DKT model, trained with only data aug- mentation (i.e., using the loss (6)) and with consistency and monotonicity regularizations (i.e., using the loss (2)). AUCs of the vanilla DKT model are given in parentheses below the dataset names.
augmentation
direction
ASSIST2015
ASSISTChall
STATICS2011
EdNet-KT1
72.01 + 0.05
74.40 0.16
86.43 = 0.29
72.75 + 0.09
correct insertion
increase
72.08 +
75.98 + 0.06
86.72
73.70 + 0.08
incorrect insertion
decrease
72.31 + 0.1
75.34
0.16
87.18 = 0.12
73.40 + 0.06
correct deletion
decrease
72.44 + o.s
75.60
+ 0.6
87.07 = 0.33
74.01 + 0.5
incorrect deletion
increase
72.26 + 0.1
74.77
+ 0.11
86.68 = 0.29
73.71 + 0.04
correct insertion
decrease
71.79 + o06
75.42 + 0.17
86.58 = 0.50
69.67 + 0.66
incorrect insertion
increase
70.73 00000
74.92
+ 0.11
86.22
71.95 + 0.15
correct delétion"
increase
66.48" 1'100
74.68
+ 0.13
86.76
71.23 +
incorrect delction
decrease
67.34
73.91 + 0.14
86.58 = 0.28
69.99 + 0.11
Table 3: Ablation test on the directions of monotonicity regularizations with the DKT model. 2nd to 5th rows show the results with the original regularization losses, and the last rows show the results with the reversed regularization losses.
Since overfitting is expected to be more severe when using smaller dataset, we conduct experiments using various fractions of the existing training datasets (5%, 10%, 25%, 50%) and show that our augmentations yield more significant improvements for smaller training datasets. Figure shows performances DKT model on various datasets, with and without augmentations. For example, on ASSISTmentsChill dataset, using 100% of the training data gives AUC 74.4%, while the same model trained with augmentations achieved AUC 75.44% with only 25% of the training dataset.
3.2 ABLATION STUDY
Are constraint losses necessary? One might think that data augmentations are enough for boosting up the performance, and imposing consistency and monotonicity are not necessary. However, we found that including such regularization losses for training is essential for further performance gain. To see this, compare the performances of the model trained only with KT losses for both original and augmented sequences
Liot Lori +
Σ NangLaus
(6)
AugEA
(where Aang 1) and with consistency and monotonicity regularization losses (2) where A is set that contains a single augmentation. Training a model with the loss (6) can be thought as using augmentations without imposing any consistency monotonicity biases.
Table 2 shows results under the DKT model. Using only data augmentation (training the model with worse performances. However, training
the loss (6)) gives marginal gain in performances with both data augmentation and consistency or monotonicity regularization losses (2) give signif- icantly higher performance gain. Under ASSlSTmentsChal dataset, using replacement along with

---

Under review as a conference paper at ICLR 2021
dataset
augmentation
replaced inters
remaining inters
full inters
gDKT
ASSIST2015
72.01 + ons
70.53 + 0.7
72.07 + 0.2
71.39 + am
ASSISTChall
74.40 + 0.16
74.68 +
78.45
0.08
75.91 + our
75.17 +
STATICS2011
86.43
0.9
82.97 0.27
87.17
0.15
83.49
0.10
86.51 0.10
EdNet-KTI
72.75 1 0.09
65.52 1 0.07
73.87
+ 0.10
68.77
+ 0.11
64.49 + 0.09
Table 4: Performances (AUCs) of the DKT model with variations of replacements and gDKT with Lapacian regularization. Best result for each dataset is indicated in bold.
dataset
no augmentation
question- random
ceeraction-randonn
skill-set
skill
ASSIST2015
72.01 0.8
72.07 + 0.02
71.77 0.05
ASSISTChall
74.40 0.16
78.39 1 0.09
74.27
0.38
77.57 ou8
78.45 1 0.8
STATICS2011
86.43
0.99
86.35 1 0.06
84.50
1 0.28
87.17
1 0.15
EdNet-KTI
72.75 1 0.09
73.84 1 0.05
72.62 1: 0.17
73.80 1: our
73.87
+ 0.10
Table 5: Performances (AUCs) of the DKT model with different type of replacements question- random replacement, interaction-random replacement, skill-set-based replacement, and skill-based replacement. Best result for each dataset is indicated in bold.
consistency regularization improves AUC by 6%, which much higher than the I 1% improvement only using data augmentation.
Ablation on monotonicity constraints We perform an ablation study to compare the effects of monotonicity regularization and reversed monotonocity regularization. Monotonocity regulariza- tion introduces constraint loss to align the inserted or deleted sequence order to modify the prob- ability of correctness of the original sequence to follow insertion deletion. For example, when correct response is inserted to the sequence, the probability of correctness for the original sequence increases. Reversed monotonocity regularization modifies the probability of correctness in the op- posite manner, where inserting a correct response would decrease the probability of correctness in the original sequence.
For each aug E {cor_ins, incor incor_ins, cor,del, incor_del]. we can define reversed version of the monotonicity regularization loss Crev aug which impose the opposite constraint on the model's out- put, c.g. we define reg-oor.ing inn as
Crev reg-cor.ins
Eee (rrz)|maxx(0,.p0 PO. ins -P:)] L-og-inncor.ng
which forces model's output of correctness probability to decrease when correct responses are in- do not include KT loss for augmented sequences (set Xaug to
serted. In this experiments, observe the effects of consistency loss only. Also, the same hyperparameters (Oang and Ayeg-aug) are used for both the original and reversed constraints.
Table 3 shows the performances of DKT model with the original and reversed monotonicity regu- larizations. Second row represents the performance with no augmentations, the 3rd to the 6th rows represent the results from using original (aligned) insertion/deletion monotonicity regularization losses, and the last four rows represent the results when the reversed monotonicity regularization losses are used. The results demonstrate that using aligned monotonicity regularization loss outper- forms the model with reversed monotonicity regularization loss. Also, the performances of reversed monotonicity shows large decrease in performance with no augmentation. For example, in case the EdNet-KTI dataset, the model's performance with correct insertion along with original (aligned) regularization improves the AUC from 72.75% to 73.70%, while using the reversed regularization drops the performance to 69.67%.
several datasets even compared to the model
Ablation replacement. We compare our consistency regularization with the other two variations of replacements consistency regularization on replaced interactions and full interactions and qDKT (Sonkar et al., 2020). As mentioned in Section 3, there of the consistency loss for the replacement depends on whether we include replaced interaction's output in the loss not:
more possible variations
Lrog To = EIER((:: 1i:0--p--pp ) Creg-rep full = E - rep
where ro stands for replaced only. We compared such variations with the original consistency loss freg-rep that does not include predictions for the replaced interactions. For all variations, we used
8

---

Under review as a conference paper at ICLR 2021
the same replacement probability Grep and loss weight Areg-rep- and we do not include KT loss for replaced sequences (set Arep = 0) as before. Also, we compare replacement with qDKT that uses the following Laplacian loss which regularizes the variance of predicted correctness probabilities for questions that fall under the same skill:
Lipplacia = (o (, { - -p)
where Q is the set of all questions, pt: Py are the model's predicted correctness probabilities for the questions qt, 9j. and 1(i, is if g have common skills attached, otherwise O. It is similar to our variation of consistency regularization (Creg-rep.ro in (8)) that only compares replaced interactions' outputs, but it does not replace questions and it compares all questions (with same skills) at once. Since the hyperparameter that scales the Laplacian regularization term is not provided in the paper, we use the same set of hyperparameters we use for other losses, and report the best results among them. Table shows that including the replaced interactions' outputs hurt performances. For ex- ample, under the EdNet-KTI dataset, all the variations of consistency regularization and Laplacian regularization significantly dropped AUCs to under 70%, while the original consistency regulariza- tion boost up the performance from 72.75% to 73.87%.
To see the effect of using the skill information of questions for replacement, we compared skill-based replacement with three different random versions of replacement: question random replacement, in- teraction random replacement, and skill-set-based replacement. For question random replacement, replace questions with different ones randomly (without considering skill information), while interaction random replacement changes both question and responses (sample each response with 0.5 probability). Skill-set-based replacement is almost the same as the original skill-based replace- ment, but the candidates of the questions to be replaced are chosen as ones with exactly same set of skills are associated, not only have common skills (S STeP). The results Table 5 show that the performances of the question random replacements depends on the nature of dataset. It shows similar performance with skill-based replacement on ASSISTmentsChall and EdNet-KT1 datasets, but only give a minor gain even dropped the performance other datasets. However, applying interaction-randow replacement significantly hurt performances over all datasets, e.g. the AUC is decreased from 86.43% to 84.50% on STATICS2011 dataset. This demonstrates the importance of fixing responses of the interactions for consistency regularization. At last, skill-set-based re- placement works similar or worse than the original skill-based replacement. Note that each question of the STATICS201 dataset has single skill attached to, so the performance of skill-based and skill-set-based replacement coincide on the dataset.
4 CONCLUSION
We propose simple augmentation strategies with corresponding constraint regularization losses for KT and show their efficacy. We only considered the most basic features of interactions, question and response correctness, and other features like elapsed time or question texts enables us to exploit di- verse augmentation strategies if available. Furthermore, exploring applicability of our idea on other AiEd tasks (dropout prediction or at-risk student prediction) is another interesting future direction.
REFERENCES
Ghodai Abdelrahman and Qing Wang. Knowledge tracing with sequential key-value memory net- works. In Proceedings of the 42nd International ACM SIGIR Conference Research and De- velopment in Information Retrieval. pp. 175-184, 2019.
Yaser Abu-Mostafa. method for learning from hints. Advances in Neural Information Processing Systems, 5:73-80, 1992.
Yaser S Abu-Mostafa. Learning from hints in neural networks. Journal of complexity, 6(2):192-198, 1990.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5049-5059, 2019.
Penghe Chen, Yu Lu, Vincent W Zheng, and Yang Pian. Prerequisite-driven deep knowledge tracing. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 39 48. IEEE, 2018.

---

Under review as a conference paper at ICLR 2021
Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Back, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bac, and Jaewe Heo. Towards an appropriate query, key, and value compu- tation for knowledge tracing. arXiv preprint arXtv:2002.07033 2020a.
Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Scoyon Park, Seewoo Lee, Jineon Back, Chan Bac, Byungsoo Kim, and Jaewe Heo. Ednet: large-scale hierarchical dataset in education. In International Conference on Artificial Intelligence in Education, pp. 69-73.
Springer, 2020b.
Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, ((4):253-278, 1994.
Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction, 19(3): 243-266, 2009.
Aritra Ghosh, Neil Heffernan, and Andrew S Lan. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & 2020.
Data Mining, pp.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñssk, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471 17611147662200 2016.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-17800 1997.
Sheng-Wei Wei Huang, Che-Tsung Lin, Shu-Ping Chen, Yen-Y Yi Wu, Po-Hao Hsu, and Shang-Hon Lai. Auggan: Cross domain adaptation with gan-based data augmentation. In Proceedings the European Conference on Computer Vision (ECCV), pp. 718-731, 2018.
2018.
Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, Guoping Hu, al. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions Knowledge and Data Engineering, 2019.
Navdeep Jaitly and Geoffrey E. Hinton. Vocal tract length perturbation (vtlp) improves speech recog- nition. In Proc. ICML Workshop on Deep Learning for Audio, Speech and Language, volume 1 117 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pp- 452 457, 2018
Yunfei Liu, Yang Yang. Xianyu Chen, Jian Shen, Haifeng Zhang, and Yong Yu. Improving knowl- edge tracing via pre-training question embeddings.
10

---

Under review as a conference paper at ICLR 2021
Koki Nagatani, Qian Zhang, Masahiro Sato, Yan- Ying Chen, Francine Chen, and Tomoko Ohkuma. Augmenting knowledge tracing by considering forgetting behavior. In The World Wide Web Con- ference, pp. 3101-3107, 2019.
Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. Graph-based knowledge tracing: Mod- eling student proficiency using graph neural network. In 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 156-163. IEEE, 2019.
P Pandey and G Karypis. A self attentive model for knowledge tracin. In Proceedings of the 12th International Conference on Educational Data Mining, 2019.
Shalini Pandey and Jaidcep Srivastava. Rkt: Relation-aware self-attention for knowledge tracing.
arXiv preprint arXiv:2008. /2736, 2020.
Daniel S Park, William Chan, Yu Zhang, Chung- Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. (tt::::904.0008 2019.
arXiv preprint
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. In Advances in neural information process- ing systems, pp. 505-513, 2015.
Shi Pu, Michael Yudelson, Lu Ou, and Yuchi Huang. International Conference Artificial Intelligence in Education, pp- 252-25666666622222222 Springer, 2020.
Deep knowledge tracing with transformers. In
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-1171, 2016.
Rico Sennrich. Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 86-96, 2016
Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, and Shijin Wang. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1857-1860, 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
Kihyuk Sohn, David Berthelot, Chun-Lian Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.0740 107665, 2020.
Shashank Sonkar, Andrew E Waters, Andrew S Lan, Phillip Grimaldi, and Richard G Baraniuk. qdkt: Question-centric deep knowledge tracing, arXiv preprint arXiv:2005.1 12442, 2020.
Hanshuang Tong. Yun Zhou, and Zhen Wang. Hgkt: Introducing problem schema with hierarchical exercise graph for knowledge tracing. arXiv preprint arXiv:2006. 16915, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic embed- ding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In Proceedings of the 2015 conference empirical methods in natural language processing, pp. 2557-2563 2015.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP--JCCIPP) pp. 6383-6389, 2019.
11

---

Under review as a conference paper at ICLR 2021
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 12848, 2019.
Shanghui Yang, Mengxia Zhu, Jingyang Hou, and Xuesong Lu. Deep knowledge tracing with convolutions. arXiv preprint 2xxi:2008.001000 169, 2020a.
Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang, and Yong Yu. Gikt: A graph-based interaction model for knowledge tracing. arXiv preprint 2xXiv:009,05591, 2020b.
Adams Wei Yu, David Dohan, Minh- MaahThhng Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading compre- hension. In International Conference on Learning Representations, 2018.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.09412, 2017a
Jiani Zhang, Xingjaan Shi, Irwin King, and Dit- Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th international conference World Wide Web, pp. 765-774, 2017b.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASET STATISTICS AND PRE-PROCESSING
Detailted dataset statistics are given in the Table 6.
ASSISTments: For ASSISTments2015 dataset, filtered out the logs with CORRECTS not {0, 1}. Note that ASSISTmenns2015 dataset only provides question and no corre- sponding skills.
STATICS2011 concatenation of problem name and step name used as a ques- tion id, and the values in the column KC (F2011) regarded as skills attached each question.
EdNet-KTI Among 600K students, filtered out whose interaction length is in [100, 1000]. and randomly sampled 6000 users, where 5000 users for training and 1000 users for test.
logs
students
questions
skills
avg. length
avg. correctness
ASSIST2015
683801
19840
100
34.47
0.73
ASSISTChall
942816
1700
3162
102
551.68
0.37
STATICS2011
261937
333
1224
81
786.60
0.72
EdNet-KTI
2051701
6000
14419
188
341.95
0.63
Table Dataset statisties.
A.2 MONOTONICITY NATURE OF DATASETS
We perform data analysis to explore monotonicity nature of datasets, i.e. property that students are more likely answer correctly if they did the same more in the past. For each interaction of each student, we see the distribution of past interactions' correctness rate. Formally, for given interaction sequences (II,...,4y IT) with 4 (Qt, R&) and each 2 T, we compare the distributions of past interactions' correctness rate
correctness.ratee
1R,=1
where 1 RR, =1 is an indicator function which is (resp. 0) when R2 = (resp. R, = 0). We compare the distributions of correctness_ratee over interactions with R; = and Rg separately, and the results are shown in Figure 1. We can that the distributions of previous correctness rates of interactions with correct response lean more to the right than ones of interactions with incorrect response. This shows the positive correlation between previous correctness rate and the current response correctness, and it also explains why monotonicity regularization actually improve prediction performances of knowledge tracing models.
A.3 MODEL'S PREDICTIONS AND CONSISTENCY REGUL ARIUAAAIIATION LOSSES
Instead of analyzing consistency nature of datasets directly, we compare the test consistency loss for correctly and incorrectly predicted responses separately, with the DKT model on ASSIST- mentsChall, STATICS2011, and EdNet-KTT datasets. Table 7 shows the average consistency loss for correctly and incorrectly predicted responses, with the vanilla DKT model and the model trained with consistency regularization losses. When compute the test consistency loss, replaced each (previous) interaction's 's questions to another questions with overlapping skills with Orep = 0.3 probability. For all models, the average loss for the correctly predicted responses lower than the incorrectly predicted responses. This verifies that smaller consistency loss actually improves prediction accuracy.
13

---

Under review as a conference paper at ICLR 2021
dataset
target response
vanilla
regularized
ASSISTChall
correct incorrect
0.01028
0.00027
0.01713
0.00039
correct
0.00618
0.00049
STATICS2011
incorrect
0.01748
0.00093
0.00422
0.00091
EdNet-KTT
incorrect
0.00535
0.001 16
Table 7: Comparison of the average consistency loss for correctly and incorrectly predicted re- sponses of the DKT model.
A.4 HYPERPARAMETERS
Table 8 describes detailed hyperparameters for each augmentation and model that are used for the main results (Table 1). Each entry represents tuple of augmentation probability (Alaug) and weight for constraint loss (Areg-aug). which shows the best performances among Oaang E {0.1, 0.3, 0.5) and Areg-aug E {1, 10, 50, 100). Each entry represents (Qaug Areg-aug) for each augmentation. We use Aang 1 for all experiments with augmentations, except for the DKT model on STATICS2011 dataset with incorrect insertion augmentation (Ainenr_ins = 0).
To see the effect of augmentation probabilities and regularization loss weights, we perform grid search over Gang {0.1, 0.3, 0.5) and Aveg-aug E {1, 10, 50, 100} with DKT model, and the AUC results are shown as heatmaps in Figure 5.
insertion
delction
deletion replacement
cor_ins
incor_ins
cor.del
incor.del
cor.ins
incor_ins
cor.del
incor.del
DKT
(0.0)
(0.3, 100)
0)
(0.3, 100)
(0,
(0.1, 10)
ASSIST2015
DKVMN
(0.5. 100)
(0, 0)
(0, 0)
0)
(0.5, 100)
(0.0)
(0,
(0.3. 1)
SAINT
(0,0)
(0.5, 10)
(0,0)
(0, 0)
(0,0)
(0.5. 10)
(0, 0)
(0,0)
(0.3. 1)
(0,
(0.5.
(0.3,
ASSISTChall
DKVMN
(0.5. 1)
(0.0)
(0,
(0. 0)
(0.5. 1)
(0,
10,
(0,
(0.5, 100)
SAINT
0)
(0.0)
(0.3.
0)
(0.3,
(0.3,
(0.0)
100)
DKT
(0. 0)
(0.5, 10)
(0,
0)
(0, 0)
(0, 0)
(0, 0)
(0, 0)
(0.3, 100)
STATICS2011
DKVMN
(0,0)
(0,0)
(0.3, 10)
0)
(0,0) 0)
(0, ())
(0.3, I)
(0.0)
(0.3, 10)
0)
(0.5, I)
(0.
(0.5, I)
(0, 0)
(0.5,
(0.5,
(0.3, 100)
DKT
(0, 0)
(0,0)
(0.3, 50)
(0. 0)
(0,0)
(0.3. 1)
(0.3. l)
(0.0)
(0.1, 100)
EdNet-KTI
DKVMN
(0.0)
SAINT
(0.0)
(0.3, 50)
(0.0)
0)
(0.0)
(0.3. 50)
(0.0)
(0,
(0.5, 1)
Table 8: Hyperparameters for Table 1.
14

---

Under review as a conference paper at ICLR 2021
01112
612211
021096
3.194
0220
0/2717
01213
8.215
07777
02/19
07307
07226
07200
07199
07199
07225
01225
67223
6:225
07228
01203
8.227
01724
S.222
5.720
07236
07226
07222
27221
07209
07175
37150
(a) ASSISTments2015
8:1473
131472
0-144
0.101
0/339
07729
17224
0.44
-27
07705
07022
8/553
077000
01522
67558
01530
51343
011522
07345
57220
(b) ASSISTmentsChall
23AM
88737
8333
5537
03030
3.9000
04731
0C703
0.009
00003
0:002
0:0000
00000
00000
00723
62722
0.004
00017
0.7111
2.303
2.50
0.0033
03037
3.010
02002
12
30
150
10
30
330
30
100
300
123
(c) STATICS2011
07222
07375
07262
07222
01302
67144
07225
07237
0.7200
07200
0.226
07258
07801
61755
67744
07377
63757
07117
67785
07309
000
01135
63141
0T32
07750
07309
3740
0775
07727
(778)
37329
67482
315
03754
0730
63322
37701
63771
07306
0,000
01332
63339
77779
07166
07707
5332
27000
0777
0776
07383
37399
172
159
36
150
200
(d) EdNet-KTT KT
Figure Performances (AUCs) of the DKT model for each augmentation and corresponding regularization with different augmentation probabilities (Aaug) and regularization loss weights (Areg-aug). The hyperparameters are searched over asug E {0.1, 0.3, 0.5) and Areg-aug E (1, 10, 50, 100). For each dataset, each column represents results with replacement, correct inser- tion, incorrect insertion, correct deletion, and incorrect deletion, from left to right. We set Aaug = for all cases. We use question-random replacement for ASSISTments201 dataset.
15


Reviews:
[{"review_id": "f3Ql6C1gWAp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disclaimer: I am not familiar with the educational AI field. With strong argumentation, or if one of my co-reviewers is an expert in the field, I could be persuaded to change my score.\n\nThe paper investigates how well data augmentation can help improve the performance of contemporary deep learning models for Knowledge Tracing, which is a key task for educational AI. The authors propose three different types of data augmentation strategies: replacement, insertion, and deletion. The authors provide a detailed experimental section with ablation studies, highlighting the benefits of using their model in addition to recently proposed models. The authors provide confidence intervals for their results proving the significance of their solution.\n\nWhat this paper excels at is the breath and scale of their experimental section. I like that they have taken a large set of different datasets, ablated all improvements, and tried different training set size models.\n\nHowever, I am have the following concerns that leads me to reject this paper:\n\n- SAINT seems to be the most modern model, which is why this is particularly interesting as I assume any improvements would indicate a SOTA in the field. Though, in table 1, for the EdNet-KT1 dataset the authors report SAINT to have 74.78, while the original SAINT paper reports 78.11 AUC\n\n- I do not see any arguments for why these augmentation methods are of specific interest. What motivated you to try this? beyond just wanting to add noise in the training. I believe adding some noise could give a small improvement, but I do not believe that such finding on a niche NLP subfield is of general interest to the scientific community beyond a workshop.\n\n- In general, it is my understanding that this is 3 augmentation functions, something similar to dropout or synonym replacement. I think the explanation of these methods are overly complicated. I would like to see the authors making their method section easier to read and reduce the amount of unnecessary notation.\n\nAnd some more specific comments:\n\n- \"However, as the number of parameters of these models increases, e.g., the recent GPT-3 model (Brown et al., 2020) has 175 billion parameters, they may easily overfit on small datasets and hurt model’s generalizabiliy. Such an issue has been under-explored in the literature\" - The GPT-3 model doesnt overfit. Also I don't think massive language models are relevant to your problem. If it's the overfitting issue I would find something that reports on overfitting and the use of data augmentation to remedy it.\n\n- I don't get Figure 1.\n\n- What is consistency and contrastive learning? you reference 7 papers, but give no intuition about it's relevance to your work. Please elaborate.\n\n- I dont get figure 2 when reading the paper from end-to-end, I don't think it should be on the top of page 3 when it's referenced in the results section.\n\n- The math in 2.1 is unclear to me.\n\n- What is the metric in table 1? ACC or AUC? there's a huge difference. Also, is it on the validation or test set?\n\n- Are the ablation studies on the validation or test set?\n\nUPDATE:\n\nThank you for clarifying the ethics concern. However, this makes it much more difficult to assess whether I believe your method works as well as you state. After having read the rebuttal and the other reviews, I am more confident that the methodology proposed lacks connection to educational relevance and novelty for publication at this venue. My score stays the same.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W-W5LkyQZQN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show that various forms of augmentations can improve the performance on knowledge tracing. The experiments are conducted on ASSIST2015, ASSISTChall, STATICS2011 and EdNet-KT1. Data augmentation leads to a certain amount of improvements. However, consistency training provides more significant improvements. \n\nThe novelty part is limited since the proposed methods such as insertion, deletion and replacements are intuitive and also seen in prior works in NLP. The monotonicity constraint is specific to the knowledge tracing task though. \n\nThe improvements are consistent, and especially significant when the training data is limited. More ablation studies on the hyperparameters would be beneficial. \n\nMy major concern is that the novelty is limited. The paper tackles a less well-studied task so more experiments should be added. For example,\n1. Would consistency training leads to more improvements when the training data is limited?\n2. How would the hyperparameter in insertion, deletion and replacements impact the performance?\n3. Would more advanced augmentation lead to better performance?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OmKqaOQHhcz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Knowledge tracing is a longstanding task in educational data mining and has been tackled by various studies. This paper proposed that three data augmentation methods (along with different types of regularization losses) can be applied to boost the performance of existing deep neural network models for knowledge tracing. Overall, the methods developed by this paper seem technically sound. In particular, the experiments are rather extensive, i.e., four widely-used datasets were employed in the experiments and different variants of the methods were investigated and compared. However, my biggest concern for this paper is its connection with previous studies and the design principles behind the proposed methods. To be specific, there are a few places that need to be further justified or a more clear explanation.\n\n1. It would be good to provide a more detailed description of existing methods for knowledge tracing, e.g., what their limitations are and how the methods proposed can (theoretically) overcome their limitations?\n2. In the Introduction section, it would be good to further justify \"e.g., the recent GPT-3 model (Brown et al., 2020) has 175 billion parameters, they may easily overfit on small datasets and hurt model’s generalizability.\". Any other evidence to show that overfitting is a common problem in existing deep neural network models for knowledge tracing? For existing works or the current study?\n3. Also, it would be good to provide additional data analysis results to support the assumption behind the three data augmentation approaches? For example, in the existing datasets, to what extent can we observe that \"a student is more likely to answer correctly (or incorrectly) if the student did the same more in the past\"? In the experiments, to what extent such interaction sequences that were mistakenly modeled by previous studies can be modeled accurately by the newly-proposed methods?", "extra_scores": {}, "extra_text": {}}, {"review_id": "GWYZe3gM6V", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents some enhancements for Knowledge Tracing (KT), in which predictions are made about the odds of a student answering a question correctly given a sequence of correct/incorrect responses to previous questions.  The authors observe that the predictive model should obey certain 3 common sense constraints. If a question is replaced in the student's data by a very similar question, the prediction should not change much. If an additional correct question is added to the data, the odds of the student being correct on the next question should go up, and the odds should go down for questions being removed and/or added with incorrect responses.  The learning algorithm's objective function is augmented with additional terms which encourage the model to obey these constraints.\n\nPros:\n\nThe method for the most part makes sense. The experiments are reasonably thorough (4 benchmark datasets are tested) and non-trivial accuracy gains are demonstrated, although dramatic gains are only achieved on 1 of the 4 benchmarks.  Ablation experiments provide additional confidence that the interpretation of the impact of the method is correct.\n\nCons: \n\nThe paper should cite previous work from the 1990s from Yaser Abu-Mostafa, who pioneered the use of these kinds of 'regularization' enhancements under the name of 'hints'. See e.g. ' A Method for Learning from Hints', NeurIPs 1993, and several similar papers.  A Google Scholar search of other ML work on monotonicity may also be beneficial if the authors seek to continue this line of research. See e.g. the recent work of Maya Gupta et. al. \n\n I would have appreciated more information about the 'skill sets' associated with each question and how that impacts the replacement.  The authors say question is chosen as a replacement if it has some skill overlap with the original question (page 4).  However, if there are multiple skills associated with the question, wouldn't it make more sense to choose replacements based on percentage skill overlap than a simple binary detection of any overlap?\n\nFurther comments:\n\nOne comment I have (and I recognize that not everyone)\n\nSome typos:\n\nImpose certain consistency or monotonicity bias on model’s predictions -> biases on the model’s predictions\n\nFig 2 randomly insert intractions – interactions…\n\neven the student answered more questions correctly -> even if the student answered more questions correctly\n\nwhen other researchers will pursue to improve the generalization ability of KT models in the future-> for other researchers attempting to improve the generalization ability of KT models.\n\nSection 3.1 among all questios -> among all questions\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes new techniques for improving the generalization ability of deep learning models for Knowledge Tracing (KT). Instead of designing more sophisticated models, the paper investigates simple data augmentation techniques that can be applied to train existing models. In particular, three different augmentation strategies are proposed based on replacement, insertion, and deletion in the training data. These strategies are then applied with appropriate regularization loss ensuring consistency and monotonicity in the training process. Extensive experiments are performed using three popular neural models for KT and four publicly available datasets. Overall, the paper studies an interesting problem in an important application domain of online education. The results are promising and open up several exciting follow-up research directions to explore more complex data augmentation techniques for KT.

I want to thank the authors for actively engaging with the reviewers during the discussion phase and sharing their concerns about the quality of the reviews.  The reviewers generally appreciated the paper's ideas; however, there was quite a bit of spread in the reviewers' assessment of the paper (scores: 4, 5, 6, 7). In summary, this is a borderline paper, and unfortunately, the final decision is a rejection. The reviewers have provided detailed and constructive feedback for improving the paper. In particular, the authors should incorporate the reviewers' feedback to better position the work w.r.t. the existing literature on data augmentation and state of the art results, better motivate the data augmentation strategies in the context of educational applications possibly through additional data analysis, and add more ablation studies w.r.t. the hyperparameters associated with data augmentation. This is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing future revisions of the paper.

Decision: Reject