Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing

Authors: Yilmazcan Ozyurt, Stefan Feuerriegel, Mrinmaya Sachan
Primary area: learning on time series and dynamical systems
Keywords: knowledge tracing, time series prediction, knowledge concept, KC annotation, representation learning, contrastive learning, large language models

Abstract:
Knowledge tracing (KT) is a popular approach for modeling students' learning progress over time, which can enable more personalized and adaptive learning. However, existing KT approaches face two major limitations: (1) they rely heavily on expert-defined knowledge concepts (KCs) in questions, which is time-consuming and prone to errors; and (2) KT methods tend to overlook the semantics of both questions and the given KCs. In this work, we address these challenges and present KCQRL, a framework for automated knowledge concept annotation and question representation learning that  can improve the effectiveness of any existing KT model. First, we propose an automated KC annotation process using large language models (LLMs), which generates question solutions and then annotates KCs in each solution step of the questions. Second, we introduce a contrastive learning approach to generate semantically rich embeddings for questions and solution steps, aligning them with their associated KCs via a tailored false negative elimination approach. These embeddings can be readily integrated into existing KT models, replacing their randomly initialized embeddings. We demonstrate the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets, where we achieve consistent performance improvements.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730604529865

thread_type:
official_review

<thread_object>
creator: reviewer_nywh

<content>
summary:
This paper proposes a framework named KCQRL, designed to address key limitations in traditional knowledge tracing (KT) models. KT models typically require extensive manual annotation of knowledge concepts (KCs) and often overlook the semantic content of questions. KCQRL introduces automated KC annotation using large language models (LLMs) and employs contrastive learning to create semantically rich question representations. The framework is intended to integrate seamlessly with existing KT models, enhancing performance on prediction tasks across datasets.

strengths:
- The proposed framework’s use of LLMs for KC annotation reduces dependency on manual input, potentially streamlining the annotation process. KCQRL is presented as a flexible solution that can enhance multiple KT models. 

- The experimental results across various KT models demonstrate consistent performance gains, supporting the efficacy of integrating semantic embeddings.

- The framework shows potential for low-sample settings, which could benefit platforms with limited data.

weaknesses:
- The quality of KC annotations and question representations is heavily influenced by the LLM used. Reporting the effects of using different LLMs would provide insights into the adaptability of the proposed method. For example, instead of using GPT-4o, employing domain-specific LLMs might enhance the performance? 

- The framework's heavy reliance on contrastive learning to handle false negatives might reduce model interpretability, particularly when working with similar KCs. 

- Experiments focus solely on two math-focused datasets, making it unclear how well KCQRL would generalize to other subject areas or to question types that are more complex, such as open-ended responses.

questions:
- How does KCQRL handle ambiguous or multi-topic questions that might not map clearly to a single KC?
- What steps, if any, are planned to improve interpretability of the model’s embeddings, especially with similar or overlapping KCs? 
- How does KCQRL adapt when LLMs used for KC annotations become outdated or unavailable?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719248348

thread_type:
official_review

<thread_object>
creator: reviewer_h76s

<content>
summary:
In this paper, authors bridge the gap by emphasizing the importance of learning question embeddings and propose to automatically annotate the knowledge concepts by using BERT. The idea is somehow recognized, however, the design of both components is not satisfactory. I have commented below with some suggestions for further improvements.

strengths:
1. Traditional methods are more like a sequential prediction task which ignores the semantic information in both Qs and KCs. This paper leverages the power of LLMs to automatically annotate the KCs for each question, and learn a better question embedding guided by the contrastive learning.
2. Figures are well-drawn for readers to follow.
3. Extensive experiments are conducted to support the claim and the importance of question embedding.

weaknesses:
* The authors claim to have a careful negative sampling in this paper to avoid false negatives for contrastive learning. However, the auto-annotation is generated by LLMs which is difficult to control for a consistent and high-quality output all the time. There will definitely be noise introduced or similar concepts annotated in different ways during the annotation which may greatly decrease the performance during semantic learning.
> Authors cluster the similar KCs in the same cluster to reduce the impacts, however, this will still definitely affect the learning performance since they are indeed different KCs in the data level. 

> Reversely, I suggest to carefully clean the annotation by asking LLMs to bootstrap a pool or seed bank for standardized KC candidates. Each time, a new KC will be verified with the seeds we already have to avoid chaos, and a checked new one will be added into the bank.

* The design of question embedding is partially recognized for its advantages. However, there is totally nothing to do with the Student history! Seems like authors are just considering them as labels. This is a big problem where in KT tasks, student interactions are very important to guide the personalized prediction based on his/her historical performance. But in this paper, authors completely abandon the semantic or sequence guidance from student histories, which is one of the biggest problems here.

* The notations are confusing, both student and s solution steps are represented as 's'.

* BERT is not an LLM at all. The title is mis-claimed.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1731001555166

thread_type:
official_review

<thread_object>
creator: reviewer_mphq

<content>
summary:
The authors address two primary shortcomings of existing work on knowledge tracing (KT): (1) mapping questions to knowledge concepts (KCs) being evaluated (I believe from a KC taxonomy defined by subject-matter experts) and (2) a lack of pre-trained embeddings for questions aligned with KCs when applying to KT systems. Methodologically, (1) is addressed by using a SotA LLM to produce a chain-of-thought (CoT)-based answer to a given question and annotate each step with automatically generated KCs and (2) is addressed through a contrastive learning approach to producing questions embeddings that align questions and reasoning steps with KCs produced in (1). Since the learned embeddings are application and not system specific (i.e., it is designed for this task, but not a specific KT model), they can be used as pre-trained embeddings for KT tasks. Accordingly, the authors experiment with adding these to 15 recent KT models with two widely used benchmarks for predicting student response from a sequence of retrospective student interactions, showing consistent improvements over the baseline implementations. Secondary results are provided for the amount of training data (the proposed system needs to see fewer student training examples), quality of the automated KC annotations (generated KCs are better than default annotations as evaluated by a LLM), and the value of the proposed contrastive learning procedure (it is better than embedding generation alternatives). Overall, the authors make a convincing case that the proposed improvements utilizing SotA LLMs advance the SotA for KT problems.

strengths:
This is primarily an applied research submission as it relies on both SotA LLM capabilities (i.e., CoT, knowledge concept generation) within the target domain and existing methods for representation learning with contrastive learning (albeit with domain specific considerations). As previously stated, I think the authors make a convincing case in establishing SotA for the KT setting.

From an originality perspective: (1) other than the task specification and benchmark datasets, there is ostensibly significant methodological innovation within this domain and (2) from a ML/NLP perspective, there is very limited originality; the LLM prompts are relatively straightforward and the contrastive learning procedure follows from a relatively standard 'cookbook'. However, the existing methods are applied correctly, perform well, and introduce new methods to this domain.

From a quality perspective, overall the formulation is well-motivated, conceptually sensible, and performs well. Again, as an applied piece of research, I am convinced that this is advancing the SotA for knowledge tracing. Additionally, the developed methods produce advancements that apply to many existing KT systems (and likely future KT systems) -- and show consistent improvements when applied to existing systems. The 'secondary' experiments are solid from the perspective of supporting the chosen models (e.g., contrastive learning) and answering practical questions with respect to deployment (e.g., number of students in training).

Regarding clarity, the paper is well-motivated, rigorously presented, and easy to understand. The figures are clear overall and any ambiguities can be clarified by the prompts in the appendices. The discussion of experimental results highlights the key findings and makes specific references to table to help the reader focus on the most important quantifications.

Regarding significance, I am not an expert in education, but am convinced that this is a successful application of LLMs to this domain and a notable improvement in performance for the KT task.

weaknesses:
Conversely, below are some of my concerns with this paper:
- From an applied perspective, the experiments are conducted on mathematical reasoning datasets. Since this work is heavily dependent on CoT and KC annotation, it isn't clear that it would work for other domains (e.g., physics, history, etc. -- lines 168-169).
- From a methods perspective, there are limited experiments (e.g., simpler embeddings as a baseline, variety in LLM quality). These aren't mandatory from an applied perspective, but would possible broaden the interest of the paper (which is another potential weakness).
- I am not an education expert, but I am guessing that evaluating KCs with another LLM isn't the standard evaluation procedure. This isn't a big deal since the primary goal of the paper is improving KT results. However, some human-in-the-loop evaluation would help strengthen this specific evaluation.

questions:
The one clarification question I had was regarding if KCs are generally taken from a SME-generated taxonomy and this is relying on (instruction-constrained, but still open-ended) LLM generation. If there is a fixed taxonomy, it would be interesting to see references and/or summary statistics. For the LLM generated cases, it would be interesting to see the statistics of this generated set (e.g., unique labels, perplexity). Clarifying my understanding would be helpful.

The second question is any discussion regarding additional education domains, simpler embeddings, and lower-power LLMs. I don't expect full experiments, but any additional information would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732131662092

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank all the reviewers for their time and their helpful comments! We took all the suggestions at heart, and made a substantial revision to our paper. We highlighted the changes in **blue color** (see the rebuttal PDF for download). Our **main improvements** are the following:

1. **We compared the quality of KC annotations with varying sizes of LLMs.** To better observe the impact of the LLM size, we picked the KC annotations from the KT dataset and the annotations from three Llama models with 3B, 8B, and 70B parameters. We run the detailed evaluation for the performance of these models (**new Appendix J**).

2. **We conducted human evaluation to assess the quality of KC annotations.** Specifically, we picked 100 random questions, and asked 7 evaluators to compare the KC annotations of the original dataset, Llama-3.1-70B and GPT-4o in a completely blind manner, with no knowledge of which annotation came from which source. Evaluators chose the LLM annotations ~95% of the time (**new Appendix K**). 

3. **We performed experiments with simpler embeddings for the question embeddings**. We further included the performance Word2Vec, OpenAI’s  text-embedding-3-small, and BERT embeddings against our complete KCQRL framework. We found that our representation learning module (Sec. 3.2) clearly improves the performance with respect to these models (**new Appendix L**).

4. **We added a comprehensive discussion section regarding the real-world deployment of our KCQRL framework.** We specifically addressed the questions from our reviewers such as how our framework can be used if the LLM for KC annotation becomes obsolete or how our framework can be extended to domains other than Math. We additionally answered other questions our readers might have (**new Appendix M**)

5. **We further included the summary statistics and key observations about our KC annotations.** Of note, we have already **provided the full set of our generated solution steps and KC annotations in our repository**. In the paper, we aim to provide the key insights to the readers (**new Appendix I**).

With the improvements above, we are confident that we remedied all weaknesses as a result, and are convinced that our paper is a valuable contribution to the literature and a good fit for ICLR 2025. We hope you agree.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732730308336

thread_type:
official_comment

<thread_object>
creator: area_chair_xean

<content>
comment:
Dear reviewers,

Thank you for your efforts reviewing this paper. If you haven't, can you please check the authors' responses and see if your concerns have been addressed? Please acknowledge you have read their responses. Thank you!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732872582424

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder to Reviewers

comment:
Dear Reviewers,

We greatly value the time and effort you’ve dedicated to reviewing our work and engaging in the discussion process. As the discussion period approaches its conclusion, we kindly encourage you to take a moment to review our rebuttal and the improvements made to the manuscript, if you haven’t already.

**We have carefully addressed all the points raised in your initial reviews and made corresponding updates to the paper to reflect these improvements.** Your confirmation on whether our revisions adequately address your concerns would be highly appreciated, and we would welcome any further feedback or suggestions you might have.

Thank you once again for your thoughtful insights and for helping us improve the quality of our work!


Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:

This paper presents KCQRL, a framework for automated knowledge concept (KC) annotation and question representation learning to improve the effectiveness of any existing knowledge tracing (KT) model. It proposes an automated KC annotation process using large language models (LLMs), which generates question solutions and then annotates KCs in each solution step of the questions. Then it introduces a contrastive learning approach to generate semantically rich embeddings for questions and solution steps, aligning them with their associated KCs via a tailored false negative elimination approach. These embeddings can be readily integrated into existing KT models, replacing their randomly initialized embeddings. The paper demonstrates the effectiveness of KCQRL across 15 KT algorithms on two large real-world Math learning datasets. 

Strengths:

Reviewers generally agree that the paper is well motivated, well presented, and easy to understand. The authors conducted extensive experiments to support the claims on the focused settings. 

Weaknesses:

After rebuttal, the following weaknesses still exist:

It is a complete piece of work, but methodological advancements are incremental (Reviewer Mphq, h76s). Focus and experimental scope are a bit narrow, and it is less clear how the proposed framework would work for other domains (Reviewer Mphq, NywH).