Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Knowledge tracing is a longstanding task in educational data mining and has been tackled by various studies. This paper proposed that three data augmentation methods (along with different types of regularization losses) can be applied to boost the performance of existing deep neural network models for knowledge tracing. Overall, the methods developed by this paper seem technically sound. In particular, the experiments are rather extensive, i.e., four widely-used datasets were employed in the experiments and different variants of the methods were investigated and compared. However, my biggest concern for this paper is its connection with previous studies and the design principles behind the proposed methods. To be specific, there are a few places that need to be further justified or a more clear explanation.

1. It would be good to provide a more detailed description of existing methods for knowledge tracing, e.g., what their limitations are and how the methods proposed can (theoretically) overcome their limitations?
2. In the Introduction section, it would be good to further justify "e.g., the recent GPT-3 model (Brown et al., 2020) has 175 billion parameters, they may easily overfit on small datasets and hurt model’s generalizability.". Any other evidence to show that overfitting is a common problem in existing deep neural network models for knowledge tracing? For existing works or the current study?
3. Also, it would be good to provide additional data analysis results to support the assumption behind the three data augmentation approaches? For example, in the existing datasets, to what extent can we observe that "a student is more likely to answer correctly (or incorrectly) if the student did the same more in the past"? In the experiments, to what extent such interaction sequences that were mistakenly modeled by previous studies can be modeled accurately by the newly-proposed methods?
review_title: This paper tackled the problem of knowledge tracing in education by proposing three data augmentation methods and demonstrated the effectiveness of the proposed methods on four widely-used datasets. However, the paper is limited in providing a clear connection between the proposed methods and previous studies, and did not provide an adequate description of relevant studies.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors show that various forms of augmentations can improve the performance on knowledge tracing. The experiments are conducted on ASSIST2015, ASSISTChall, STATICS2011 and EdNet-KT1. Data augmentation leads to a certain amount of improvements. However, consistency training provides more significant improvements. 

The novelty part is limited since the proposed methods such as insertion, deletion and replacements are intuitive and also seen in prior works in NLP. The monotonicity constraint is specific to the knowledge tracing task though. 

The improvements are consistent, and especially significant when the training data is limited. More ablation studies on the hyperparameters would be beneficial. 

My major concern is that the novelty is limited. The paper tackles a less well-studied task so more experiments should be added. For example,
1. Would consistency training leads to more improvements when the training data is limited?
2. How would the hyperparameter in insertion, deletion and replacements impact the performance?
3. Would more advanced augmentation lead to better performance?


review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper presents some enhancements for Knowledge Tracing (KT), in which predictions are made about the odds of a student answering a question correctly given a sequence of correct/incorrect responses to previous questions.  The authors observe that the predictive model should obey certain 3 common sense constraints. If a question is replaced in the student's data by a very similar question, the prediction should not change much. If an additional correct question is added to the data, the odds of the student being correct on the next question should go up, and the odds should go down for questions being removed and/or added with incorrect responses.  The learning algorithm's objective function is augmented with additional terms which encourage the model to obey these constraints.

Pros:

The method for the most part makes sense. The experiments are reasonably thorough (4 benchmark datasets are tested) and non-trivial accuracy gains are demonstrated, although dramatic gains are only achieved on 1 of the 4 benchmarks.  Ablation experiments provide additional confidence that the interpretation of the impact of the method is correct.

Cons: 

The paper should cite previous work from the 1990s from Yaser Abu-Mostafa, who pioneered the use of these kinds of 'regularization' enhancements under the name of 'hints'. See e.g. ' A Method for Learning from Hints', NeurIPs 1993, and several similar papers.  A Google Scholar search of other ML work on monotonicity may also be beneficial if the authors seek to continue this line of research. See e.g. the recent work of Maya Gupta et. al. 

 I would have appreciated more information about the 'skill sets' associated with each question and how that impacts the replacement.  The authors say question is chosen as a replacement if it has some skill overlap with the original question (page 4).  However, if there are multiple skills associated with the question, wouldn't it make more sense to choose replacements based on percentage skill overlap than a simple binary detection of any overlap?

Further comments:

One comment I have (and I recognize that not everyone)

Some typos:

Impose certain consistency or monotonicity bias on model’s predictions -> biases on the model’s predictions

Fig 2 randomly insert intractions – interactions…

even the student answered more questions correctly -> even if the student answered more questions correctly

when other researchers will pursue to improve the generalization ability of KT models in the future-> for other researchers attempting to improve the generalization ability of KT models.

Section 3.1 among all questios -> among all questions

review_title: Good experimental results, sensible technique, some additional citations suggested

final_score: rating: 4: Ok but not good enough - rejection
review_content: Disclaimer: I am not familiar with the educational AI field. With strong argumentation, or if one of my co-reviewers is an expert in the field, I could be persuaded to change my score.

The paper investigates how well data augmentation can help improve the performance of contemporary deep learning models for Knowledge Tracing, which is a key task for educational AI. The authors propose three different types of data augmentation strategies: replacement, insertion, and deletion. The authors provide a detailed experimental section with ablation studies, highlighting the benefits of using their model in addition to recently proposed models. The authors provide confidence intervals for their results proving the significance of their solution.

What this paper excels at is the breath and scale of their experimental section. I like that they have taken a large set of different datasets, ablated all improvements, and tried different training set size models.

However, I am have the following concerns that leads me to reject this paper:

- SAINT seems to be the most modern model, which is why this is particularly interesting as I assume any improvements would indicate a SOTA in the field. Though, in table 1, for the EdNet-KT1 dataset the authors report SAINT to have 74.78, while the original SAINT paper reports 78.11 AUC

- I do not see any arguments for why these augmentation methods are of specific interest. What motivated you to try this? beyond just wanting to add noise in the training. I believe adding some noise could give a small improvement, but I do not believe that such finding on a niche NLP subfield is of general interest to the scientific community beyond a workshop.

- In general, it is my understanding that this is 3 augmentation functions, something similar to dropout or synonym replacement. I think the explanation of these methods are overly complicated. I would like to see the authors making their method section easier to read and reduce the amount of unnecessary notation.

And some more specific comments:

- "However, as the number of parameters of these models increases, e.g., the recent GPT-3 model (Brown et al., 2020) has 175 billion parameters, they may easily overfit on small datasets and hurt model’s generalizabiliy. Such an issue has been under-explored in the literature" - The GPT-3 model doesnt overfit. Also I don't think massive language models are relevant to your problem. If it's the overfitting issue I would find something that reports on overfitting and the use of data augmentation to remedy it.

- I don't get Figure 1.

- What is consistency and contrastive learning? you reference 7 papers, but give no intuition about it's relevance to your work. Please elaborate.

- I dont get figure 2 when reading the paper from end-to-end, I don't think it should be on the top of page 3 when it's referenced in the results section.

- The math in 2.1 is unclear to me.

- What is the metric in table 1? ACC or AUC? there's a huge difference. Also, is it on the validation or test set?

- Are the ablation studies on the validation or test set?

UPDATE:

Thank you for clarifying the ethics concern. However, this makes it much more difficult to assess whether I believe your method works as well as you state. After having read the rebuttal and the other reviews, I am more confident that the methodology proposed lacks connection to educational relevance and novelty for publication at this venue. My score stays the same.
review_title: Not significant improvement over baseline

Meta-review: The paper proposes new techniques for improving the generalization ability of deep learning models for Knowledge Tracing (KT). Instead of designing more sophisticated models, the paper investigates simple data augmentation techniques that can be applied to train existing models. In particular, three different augmentation strategies are proposed based on replacement, insertion, and deletion in the training data. These strategies are then applied with appropriate regularization loss ensuring consistency and monotonicity in the training process. Extensive experiments are performed using three popular neural models for KT and four publicly available datasets. Overall, the paper studies an interesting problem in an important application domain of online education. The results are promising and open up several exciting follow-up research directions to explore more complex data augmentation techniques for KT.

I want to thank the authors for actively engaging with the reviewers during the discussion phase and sharing their concerns about the quality of the reviews.  The reviewers generally appreciated the paper's ideas; however, there was quite a bit of spread in the reviewers' assessment of the paper (scores: 4, 5, 6, 7). In summary, this is a borderline paper, and unfortunately, the final decision is a rejection. The reviewers have provided detailed and constructive feedback for improving the paper. In particular, the authors should incorporate the reviewers' feedback to better position the work w.r.t. the existing literature on data augmentation and state of the art results, better motivate the data augmentation strategies in the context of educational applications possibly through additional data analysis, and add more ablation studies w.r.t. the hyperparameters associated with data augmentation. This is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing future revisions of the paper.

Decision: Reject