Year: 2020

Abstract:
Knowledge Tracing (KT), tracking a human’s knowledge acquisition, is a central component in online learning and AI in Education. In this paper, we present a simple, yet effective strategy to improve the generalization ability of KT models: we propose three types of novel data augmentation, coined replacement, insertion, and deletion, along with corresponding regularization losses that impose certain consistency or monotonicity biases on model’s predictions for the original and augmented sequence. Extensive experiments on various KT benchmarks show that our regularization scheme consistently improves the model performances, under 3 widely-used neural networks and 4 public benchmarks, e.g., it yields 6.3% improvement in AUC under the DKT model and the ASSISTmentsChall dataset.

heading: 1 INTRODUCTION
text: In recent years, Artificial Intelligence in Education (AIEd) has gained much attention as one of the currently emerging fields in educational technology. In particular, the recent COVID-19 pandemic has transformed the setting of education from classroom learning to online learning. As a result, AIEd has become more prominent because of its ability to diagnose students automatically and provide personalized learning paths. High-quality diagnosis and educational content recommendation require good understanding of students’ current knowledge status, and it is essential to model their learning behavior precisely. Due to this, Knowledge Tracing (KT), a task of modeling a student’s evolution of knowledge over time, has become one of the most central tasks in AIEd research.
Since the work of Piech et al. (2015), deep neural networks have been widely used for the KT modeling. Current research trends in the KT literature concentrate on building more sophisticated, complex and large-scale models, inspired by model architectures from Natural Language Processing (NLP), such as LSTM (Hochreiter & Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) architectures, along with additional components that extract question textual information or students’ forgetting behaviors (Huang et al., 2019; Pu et al., 2020; Ghosh et al., 2020). However, as the number of parameters of these models increases, they may easily overfit on small datasets and hurt model’s generalizabiliy. Such an issue has been under-explored in the literature.
To address the issue, we propose simple, yet effective data augmentation strategies for improving the generalization ability of KT models, along with novel regularization losses for each strategy. In particular, we suggest three types of data augmentation, coined (skill-based) replacement, insertion, and deletion. Specifically, we generate augmented (training) samples by randomly replacing questions that a student solved with similar questions or inserting/deleting interactions with fixed responses. Then, during training, we impose certain consistency (for replacement) and monotonicity (for insertion/deletion) bias on a model’s predictions by optimizing corresponding regularization losses that compares the original and the augmented interaction sequences. Here, our intuition behind the proposed consistency regularization is that the model’s output for two interaction sequences with same response logs for similar questions should be close. Next, the proposed monotonicity regularization is designed to enforce the model’s prediction to be monotone with respect to the number of questions that correctly (or incorrectly) answered, i.e., a student is more likely to answer correctly (or incorrectly) if the student did the same more in the past. By analyzing distribution of the previous correctness rates of interaction sequences, we can observe that the existing student interaction datasets indeed have monotonicity properties - see Figure 1 and Section A.2 for details. The overall augmentation and regularization strategies are sketched in Figure 2. Such regularization strategies
are motivated from our observation that existing knowledge tracing models’ prediction often fails to satisfy the consistency and monotonicity condition, e.g., see Figure 4 in Section 3.
We demonstrate the effectiveness of the proposed method with 3 widely used neural knowledge tracing models - DKT (Piech et al., 2015), DKVMN (Zhang et al., 2017b), and SAINT (Choi et al., 2020a) - on 4 public benchmark datasets - ASSISTments2015, ASSISTmentsChall, STATICS2011, and EdNet-KT1. Extensive experiments show that, regardless of dataset or model architecture, our scheme remarkably increases the prediction performance - 6.2% gain in Area Under Curve (AUC) for DKT on the ASSISTmentsChall dataset. In particular, ours is much more effective under smaller datasets: by using only 25% of the ASSISTmentsChall dataset, we improve AUC of the DKT model from 69.68% to 75.44%, which even surpasses the baseline performance 74.4% with the full training set. We further provide various ablation studies for the selected design choices, e.g., AUC of the DKT model on the ASSISTments2015 dataset is dropped from 72.44% to 66.48% when we impose ‘reversed’ (wrong) monotonicity regularization. We believe that our work can be a strong guideline for other researchers attempting to improve the generalization ability of KT models.

heading: 1.1 RELATED WORKS AND PRELIMINARIES
text: Data augmentation is arguably the most trustworthy technique to prevent overfitting or improve the generalizability of machine learning models. In particular, it has been developed as an effective way to impose a domain-specific, inductive bias to a model. For example, for computer vision models, simple image warpings such as flip, rotation, distortion, color shifting, blur, and random erasing are the most popular data augmentation methods (Shorten & Khoshgoftaar, 2019). More advanced techniques, e.g., augmenting images by interpolation (Zhang et al., 2017a; Yun et al., 2019) or by using generative adversarial networks (Huang et al., 2018), have been also investigated. For NLP models,
it is popular to augment texts by replacing words with synonyms (Zhang et al., 2015) or words with similar (contextualized) embeddings (Wang & Yang, 2015; Kobayashi, 2018). As an alternative method, back translation (Sennrich et al., 2016; Yu et al., 2018) generates an augmented sentence by translating a given sentence into a different language domain and translate it back to the original domain with machine translation models. Recently, Wei & Zou (2019) show that even simple methods like random insertion/swap/deletion could improve text classification performances. In the area of speech recognition, vocal tract length normalization (Jaitly & Hinton, 2013), synthesizing noisy audio (Hannun et al., 2014), perturbing speed (Ko et al., 2015), and augmenting spectrogram (Park et al., 2019) are popular as data augmentation methods.
The aformentioned data augmentation techniques have been used not only for standard supervised learning setups, but also for various unsupervised and semi-supervised learning frameworks, by imposing certain inductive biases to models. For example, consistency learning (Sajjadi et al., 2016; Xie et al., 2019; Berthelot et al., 2019; Sohn et al., 2020) impose a consistency bias to a model so that the model’s output is invariant under the augmentations, by means of training the model with consistency regularization loss (e.g. L2-loss between outputs). Abu-Mostafa (1992; 1990) suggested general theory for imposing such inductive biases (which are stated as hints) via additional regularization losses. Their successes highlight the importance of domain specific knowledge for designing appropriate data augmentation strategies, but such results are rare in the domain of AIEd, especially for Knowledge Tracing.
Knowledge tracing (KT) is the task of modeling student knowledge over time based on the student’s learning history. Formally, for a given student interaction sequence (I1, . . . , IT ), where each It = (Qt, Rt) is a pair of question id Qt and the student’s response correctness Rt ∈ {0, 1} (1 means correct), KT aims to estimate the following probability
P[Rt = 1|I1, I2, . . . , It−1, Qt], (1)
i.e., the probability that the student answers correctly to the question Qt at t-th step. Corbett & Anderson (1994) proposed Bayesian Knowledge Tracing (BKT) that models a student’s knowledge as a latent variable in a Hidden Markov Model. Also, various seq2seq architectures including LSTM (Hochreiter & Schmidhuber, 1997), MANN (Graves et al., 2016), and Transformer (Vaswani et al., 2017) are used in the context of KT and showed their efficacy. Deep Knowledge Tracing (DKT) is the first deep learning based model that models student’s knowledge states as LSTM’s hidden state vectors (Piech et al., 2015). Dynamic Key-Value Memory Network and its variation can exploit relationships between questions/skills with concept vectors and concept-state vectors with key and value matrices, which is more interpretable than DKT (Zhang et al., 2017b; Abdelrahman & Wang, 2019). Transformer based models (Pandey & Karypis, 2019; Choi et al., 2020a; Ghosh et al., 2020; Pu et al., 2020) are able to learn long-range dependencies with their self-attention mechanisms and be trained in parallel. Utilizing additional features of interactions, such as question texts (Huang et al., 2019; Pandey & Srivastava, 2020), prerequisite relations (Chen et al., 2018) and time information (Nagatani et al., 2019; Choi et al., 2020a; Pu et al., 2020) is another way to improve performances. Recent works try to use graph neural networks (Nakagawa et al., 2019; Liu et al.; Tong et al., 2020; Yang et al., 2020b) and convolutional networks (Yang et al., 2020a; Shen et al., 2020) to model relations between questions and skills or extract individualized prior knowledge.

heading: 2 CONSISTENCY AND MONOTONICITY REGULARIZATION FOR KT
text: For a given set of data augmentations A, we train KT models with the following loss: Ltot = Lori + ∑
aug∈A (λaugLaug + λreg-augLreg-aug), (2)
where Lori is the commonly used binary cross-entropy (BCE) loss for original training sequences and Laug are the same BCE losses for augmented sequences generated by applying augmentation strategies aug ∈ A.1 Lreg-aug are the regularization losses that impose consistency and monotonicity bias on model’s predictions for the original and augmented sequence, which are going to be defined in the following sections. Finally, λaug, λreg-aug > 0 are hyperparameters to control the trade-off among Lori, Laug, and Lreg-aug.
1For replacement and insertion, we do not include outputs for augmented interactions in Laug.
In the following sections, we introduce our three simple augmentation strategies, replacement, insertion and deletion with corresponding consistency and monotonicity regularization losses, Lreg-rep, Lreg-cor ins (or Lreg-incor ins) and Lreg-cor del (or Lreg-incor del), respectively.

heading: 2.1 REPLACEMENT
text: Replacement, which is motivated by the synonym replacement in NLP, is an augmentation strategy that replaces questions in the original interaction sequence with other similar questions without changing their responses, where similar questions are defined as the questions that have overlapping skills attached to. Our hypothesis is that the predicted correctness probabilities for questions in an augmented interaction sequence will not change a lot from those in the original interaction sequence. Formally, for each interaction in the original interaction sequence (I1, . . . , IT ), we randomly decide whether the interaction will be replaced or not, following the Bernoulli distribution with the probability αrep. If an interaction It = (Qt, Rt) with a set of skills St associated with the question Qt is set to be replaced, we determine Irept = (Q rep t , Rt) by selecting a question Q rep t with its associated set of skills Srept that satisfies St ∩ S rep t 6= ∅. The resulting augmented sequence (I rep 1 , . . . , I rep T ) is generated by replacing It with I rep t for t ∈ R ⊂ [T ] = {1, 2, . . . , T}, where R is a set of indices to replace. Then we consider the following consistency regularization loss:
Lreg-rep = Et 6∈R[(pt − prept )2] (3)
where pt and p rep t are model’s predicted correctness probabilities for t-th question of the original and augmented sequences, respectively. We do not include the output for the replaced interactions in the loss computation. For the replacement strategy itself we consider several variants. For instance, randomly selecting a question for Qrept from the question pool is an alternative strategy if a skill set for each question is not available. It is also possible to only consider outputs for interactions that are replaced or consider outputs for all interactions in the augmented sequence for the loss computation. We investigate the effectiveness of each strategy in Section 3.

heading: 2.2 INSERTION
text: When a student answers more questions correctly (resp. incorrectly), the predicted correctness probabilities of the KT models for the remaining questions should increase (resp. decrease). Based on this intuition, we introduce a monotonicity constraint by inserting new interactions into the original interaction sequence. Formally, we generate an augmented interaction sequence (I ins1 , . . . , I ins T ′ ) by inserting a correctly (resp. incorrectly) answered interaction I ins t = (Q ins t , 1) (resp. I inst = (Q ins t , 0)) into the original interaction sequence (I1, . . . , IT ) for t ∈ I ⊂ [T ′], where the question Qinst is randomly selected from the question pool and I with the size αins proportion of the original sequence is a set of indices of inserted interactions. Then our hypothesis is formulated as pt ≤ pinsσ(t) (resp. pt ≥ p ins σ(t)), where pt and p ins t are model’s predicted correctness probabilities for t-th question of the original and augmented sequences, respectively. Here, σ : [T ]→ [T ′]− I is the order-preserving bijection which satisfies It = I insσ(t) for 1 ≤ t ≤ T . (For instance, in Figure 2, σ sends {1, 2, 3, 4} to {2, 3, 4, 6}) We impose our hypothesis through the following losses:
Lreg-cor ins = Et∈[T ][max(0, pt − pinsσ(t))], Lreg-incor ins = Et∈[T ][max(0, p ins σ(t) − pt)] (4)
where Lreg-cor ins and Lreg-incor ins are losses for augmented interaction sequences of inserting correctly and incorrectly answered interactions, respectively.

heading: 2.3 DELETION
text: Similar to the insertion augmentation strategy, we propose another monotonicity constraint by removing some interactions in the original interaction sequence based on the following hypothesis: if a student’s response records contain less correct (resp. incorrect) answers, the correctness probabilities for the remaining questions would become decrease (resp. increase). Formally, from the original interaction sequence (I1, . . . , IT ), we randomly sample a set of indices D ⊂ [T ], where Rt = 1 (resp. Rt = 0) for t ∈ D, based on the Bernoulli distribution with the probability αdel. We remove the index t ∈ D and impose the hypothesis pt ≥ pdelσ(t) (resp. pt ≤ p del σ(t)), where pt and pdelt are model’s predicted correctness probabilities for t-th question of the original and augmented
sequences, respectively. Here, σ : [T ]−D→ [T ′] is the order preserving bijection with It = Idelσ(t) for t ∈ [T ]−D. We impose the hypothesis through the following losses:
Lreg-cor del = Et6∈D[max(0, pdelσ(t) − pt)], Lreg-incor del = Et6∈D[max(0, pt − p del σ(t))] (5)
where Lreg-cor del and Lreg-incor del are losses for augmented interaction sequences of deleting correctly and incorrectly answered interactions, respectively.

heading: 3 EXPERIMENTS
text: We demonstrated the effectiveness of the proposed method on 4 widely used benchmark datasets: ASSISTments2015, ASSISTmentsChall, STATICS2011, and EdNet-KT1. ASSISTments datasets are the most widely used benchmark for Knowledge Tracing, which is provided by ASSISTments online tutoring platform2 (Feng et al., 2009). There are several versions of dataset depend on the years they collected, and we used ASSISTments20153 and ASSISTmentsChall4. ASSISTmentsChall dataset is provided by the 2017 ASSISTments data mining competition. STATICS2011 consists of the interaction logs from an engineering statics course, which is available on the PSLC datashop5. EdNet-KT1 is the largest publicly available interaction dataset consists of TOEIC (Test of English for Interational Communication) problem solving logs collected by Santa6 (Choi et al.,
2https://new.assistments.org/ 3https://sites.google.com/site/assistmentsdata/home/2015-assistments-skill-builder-data 4https://sites.google.com/view/assistmentsdatamining 5https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=507 6https://aitutorsanta.com/
2020b). We reduce the size of the EdNet-KT1 dataset by sampling 6000 users among 600K users. Detailed statistics and pre-processing methods for these datasets are described in Appendix. With the exception of the EdNet-KT1 dataset, we used 80% of the students as a training set and the remaining 20% as a test set.
We test DKT (Piech et al., 2015), DKVMN (Zhang et al., 2017b), and SAINT (Choi et al., 2020a) models. For DKT, we set the embedding dimension and the hidden dimension as 256. For DKVMN, key, value, and summary dimension are all set to be 256, and we set the number of latent concepts as 64. SAINT has 2 layers with hidden dimension 256, 8 heads, and feed-forward dimension 1024. All the models do not use any additional features of interactions except question ids and responses as an input, and the model weights are initialized with Xavier distribution (Glorot & Bengio, 2010). They are trained from scratch with batch size 64, and we use the Adam optimizer (Kingma & Ba, 2015) with learning rate 0.001 which is scheduled by Noam scheme with warm-up step 4000 as Vaswani et al. (2017) suggest. We set each model’s maximum sequence size as 100 on ASSISTments2015 & EdNet-KT1 dataset and 200 on ASSISTmentsChall & STATICS2011 dataset. Hyperparameters for augmentations, αaug, λreg-aug, and λaug are searched over αaug ∈ {0.1, 0.3, 0.5}, λreg-aug ∈ {1, 10, 50, 100}, and λaug ∈ {0, 1}. For all dataset, we evaluate our results using 5-fold cross validation and use Area Under Curve (AUC) as an evaluation metric.

heading: 3.1 MAIN RESULTS
text: The results (AUCs) are shown in Table 1 that compares models without and with augmentations, and we report the best results for each strategy. (The detailed hyperparameters for these results are given in Appendix.) The 4th column represents results using both insertion and deletion, and the last column shows the results with all 3 augmentations. Since there’s no big difference on performance gain between insertion and deletion, we only report the performance that uses one or both of them together. We use skill-based replacement if skill information for each question in the dataset is available, and use question-random replacement that that selects new questions among all questios if not (e.g. ASSISTments2015). As one can see, the models trained with consistency and monotonicity regularizations outperforms the models without augmentations in a large margin, regardless of model’s architectures or datasets. Using all three augmentations gives the best performances for most of the cases. For instance, there exists 6.3% gain in AUC on ASSISTmentsChall dataset under the DKT model. Furthermore, not only enhancing the prediction performances, our training scheme also resolves the vanilla model’s issue where the monotonicity condition on the predictions of original and augmented sequences is violated. As in Figure 4, the predictions of the model trained with monotonicity regularization (correct insertion) are increased after insertion, which contrasts to the vanilla DKT model’s outputs.
Since overfitting is expected to be more severe when using a smaller dataset, we conduct experiments using various fractions of the existing training datasets (5%, 10%, 25%, 50%) and show that our augmentations yield more significant improvements for smaller training datasets. Figure 3 shows performances of DKT model on various datasets, with and without augmentations. For example, on ASSISTmentsChall dataset, using 100% of the training data gives AUC 74.4%, while the same model trained with augmentations achieved AUC 75.44% with only 25% of the training dataset.

heading: 3.2 ABLATION STUDY
text: Are constraint losses necessary? One might think that data augmentations are enough for boosting up the performance, and imposing consistency and monotonicity are not necessary. However, we found that including such regularization losses for training is essential for further performance gain. To see this, we compare the performances of the model trained only with KT losses for both original and augmented sequences
Ltot = Lori + ∑
aug∈A λaugLaug (6)
(where λaug = 1) and with consistency and monotonicity regularization losses (2) where A is a set that contains a single augmentation. Training a model with the loss (6) can be thought as using augmentations without imposing any consistency or monotonicity biases.
Table 2 shows results under the DKT model. Using only data augmentation (training the model with the loss (6)) gives a marginal gain in performances or even worse performances. However, training with both data augmentation and consistency or monotonicity regularization losses (2) give significantly higher performance gain. Under ASSISTmentsChall dataset, using replacement along with
consistency regularization improves AUC by 6%, which is much higher than the 1% improvement only using data augmentation.
Ablation on monotonicity constraints We perform an ablation study to compare the effects of monotonicity regularization and reversed monotonocity regularization. Monotonocity regularization introduces constraint loss to align the inserted or deleted sequence in order to modify the probability of correctness of the original sequence to follow insertion or deletion. For example, when a correct response is inserted to the sequence, the probability of correctness for the original sequence increases. Reversed monotonocity regularization modifies the probability of correctness in the opposite manner, where inserting a correct response would decrease the probability of correctness in the original sequence.
For each aug ∈ {cor ins, incor ins, cor del, incor del}, we can define reversed version of the monotonicity regularization loss Lrevreg-aug which impose the opposite constraint on the model’s output, e.g. we define Lrevreg-cor ins as
Lrevreg-cor ins = Et∈[T ][max(0, pinsσ(t) − pt)] = Lreg-incor ins (7)
which forces model’s output of correctness probability to decrease when correct responses are inserted. In this experiments, we do not include KT loss for augmented sequences (set λaug = 0) to observe the effects of consistency loss only. Also, the same hyperparameters (αaug and λreg-aug) are used for both the original and reversed constraints.
Table 3 shows the performances of DKT model with the original and reversed monotonicity regularizations. Second row represents the performance with no augmentations, the 3rd to the 6th rows represent the results from using original (aligned) insertion/deletion monotonicity regularization losses, and the last four rows represent the results when the reversed monotonicity regularization losses are used. The results demonstrate that using aligned monotonicity regularization loss outperforms the model with reversed monotonicity regularization loss. Also, the performances of reversed monotonicity shows large decrease in performance on several datasets even compared to the model with no augmentation. For example, in case of the EdNet-KT1 dataset, the model’s performance with correct insertion along with original (aligned) regularization improves the AUC from 72.75% to 73.70%, while using the reversed regularization drops the performance to 69.67%.
Ablation on replacement. We compare our consistency regularization with the other two variations of replacements - consistency regularization on replaced interactions and full interactions - and qDKT (Sonkar et al., 2020). As we mentioned in Section 3, there are two more possible variations of the consistency loss for the replacement depends on whether we include replaced interaction’s output in the loss or not:
Lreg-rep ro = Et∈R[(pt − prept )2], Lreg-rep full = Et∈[T ][(pt − p rep t ) 2], (8)
where ro stands for replaced only. We compared such variations with the original consistency loss Lreg-rep that does not include predictions for the replaced interactions. For all variations, we used
the same replacement probability αrep and loss weight λreg-rep, and we do not include KT loss for replaced sequences (set λrep = 0) as before. Also, we compare replacement with qDKT that uses the following Laplacian loss which regularizes the variance of predicted correctness probabilities for questions that fall under the same skill:
LLaplacian = E(qi,qj)∈Q×Q[1(i, j)(pi − pj) 2] (9)
where Q is the set of all questions, pi, pj are the model’s predicted correctness probabilities for the questions qi, qj , and 1(i, j) is 1 if qi, qj have common skills attached, otherwise 0. It is similar to our variation of consistency regularization (Lreg-rep ro in (8)) that only compares replaced interactions’ outputs, but it does not replace questions and it compares all questions (with same skills) at once. Since the hyperparameter λ that scales the Laplacian regularization term is not provided in the paper, we use the same set of hyperparameters we use for other losses, and report the best results among them. Table 4 shows that including the replaced interactions’ outputs hurt performances. For example, under the EdNet-KT1 dataset, all the variations of consistency regularization and Laplacian regularization significantly dropped AUCs to under 70%, while the original consistency regularization boost up the performance from 72.75% to 73.87%.
To see the effect of using the skill information of questions for replacement, we compared skill-based replacement with three different random versions of replacement: question random replacement, interaction random replacement, and skill-set-based replacement. For question random replacement, we replace questions with different ones randomly (without considering skill information), while interaction random replacement changes both question and responses (sample each response with 0.5 probability). Skill-set-based replacement is almost the same as the original skill-based replacement, but the candidates of the questions to be replaced are chosen as ones with exactly same set of skills are associated, not only have common skills (S = Srep). The results in Table 5 show that the performances of the question random replacements depends on the nature of dataset. It shows similar performance with skill-based replacement on ASSISTmentsChall and EdNet-KT1 datasets, but only give a minor gain or even dropped the performance on other datasets. However, applying interaction-random replacement significantly hurt performances over all datasets, e.g. the AUC is decreased from 86.43% to 84.50% on STATICS2011 dataset. This demonstrates the importance of fixing responses of the interactions for consistency regularization. At last, skill-set-based replacement works similar or even worse than the original skill-based replacement. Note that each question of the STATICS2011 dataset has single skill attached to, so the performance of skill-based and skill-set-based replacement coincide on the dataset.

heading: 4 CONCLUSION
text: We propose simple augmentation strategies with corresponding constraint regularization losses for KT and show their efficacy. We only considered the most basic features of interactions, question and response correctness, and other features like elapsed time or question texts enables us to exploit diverse augmentation strategies if available. Furthermore, exploring applicability of our idea on other AiEd tasks (dropout prediction or at-risk student prediction) is another interesting future direction.

heading: A APPENDIX
text: A.1 DATASET STATISTICS AND PRE-PROCESSING
Detailted dataset statistics are given in the Table 6.
A.2 MONOTONICITY NATURE OF DATASETS
We perform data analysis to explore monotonicity nature of datasets, i.e. a property that students are more likely to answer correctly if they did the same more in the past. For each interaction of each student, we see the distribution of past interactions’ correctness rate. Formally, for given interaction sequences (I1, . . . , IT ) with It = (Qt, Rt) and each 2 ≤ t ≤ T , we compare the distributions of past interactions’ correctness rate
correctness rate<t = 1
t− 1 t−1∑ τ=1 1Rτ=1
where 1Rτ=1 is an indicator function which is 1 (resp. 0) whenRτ = 1 (resp. Rτ = 0). We compare the distributions of correctness rate<t over all interactions with Rt = 1 and Rt = 0 separately, and the results are shown in Figure 1. We can see that the distributions of previous correctness rates of interactions with correct response lean more to the right than ones of interactions with incorrect response. This shows the positive correlation between previous correctness rate and the current response correctness, and it also explains why monotonicity regularization actually improve prediction performances of knowledge tracing models.
A.3 MODEL’S PREDICTIONS AND CONSISTENCY REGULARIZATION LOSSES
Instead of analyzing consistency nature of datasets directly, we compare the test consistency loss for correctly and incorrectly predicted responses separately, with the DKT model on ASSISTmentsChall, STATICS2011, and EdNet-KT1 datasets. Table 7 shows the average consistency loss for correctly and incorrectly predicted responses, with the vanilla DKT model and the model trained with consistency regularization losses. When we compute the test consistency loss, we replaced each (previous) interaction’s questions to another questions with overlapping skills with αrep = 0.3 probability. For all models, the average loss for the correctly predicted responses are lower than the incorrectly predicted responses. This verifies that smaller consistency loss actually improves prediction accuracy.
A.4 HYPERPARAMETERS
Table 8 describes detailed hyperparameters for each augmentation and model that are used for the main results (Table 1). Each entry represents a tuple of augmentation probability (αaug) and a weight for constraint loss (λreg-aug), which shows the best performances among αaug ∈ {0.1, 0.3, 0.5} and λreg-aug ∈ {1, 10, 50, 100}. Each entry represents (αaug, λreg-aug) for each augmentation. We use λaug = 1 for all experiments with augmentations, except for the DKT model on STATICS2011 dataset with incorrect insertion augmentation (λincor ins = 0).
To see the effect of augmentation probabilities and regularization loss weights, we perform grid search over αaug ∈ {0.1, 0.3, 0.5} and λreg-aug ∈ {1, 10, 50, 100} with DKT model, and the AUC results are shown as heatmaps in Figure 5.

Review summaries:
This paper presents some enhancements for Knowledge Tracing (KT), in which predictions are made about the odds of a student answering a question correctly given a sequence of correct/incorrect responses to previous questions.  The authors observe that the predictive model should obey certain 3 common sense constraints. If a question is replaced in the student's data by a very similar question, the prediction should not change much. If an additional correct question is added to the data, the odds of the student being correct on the next question should go up, and the odds should go down for questions being removed and/or added with incorrect responses.  The learning algorithm's objective function is augmented with additional terms which encourage the model to obey these constraints.

Knowledge tracing is a longstanding task in educational data mining and has been tackled by various studies. This paper proposed that three data augmentation methods (along with different types of regularization losses) can be applied to boost the performance of existing deep neural network models for knowledge tracing. Overall, the methods developed by this paper seem technically sound. In particular, the experiments are rather extensive, i.e., four widely-used datasets were employed in the experiments and different variants of the methods were investigated and compared. However, my biggest concern for this paper is its connection with previous studies and the design principles behind the proposed methods. To be specific, there are a few places that need to be further justified or a more clear explanation.