Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracing
Authors: Yilmazcan Ozyurt
Tunaberk Almaci
Stefan Feuerriegel
Mrinmaya Sachan
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Tracing
Exercise Recommendation
Reinforcement Learning
Educational AI
Semantic Representation Learning
Primary area: applications
TL;DR: We present a general framework for personalized math exercise recommendation that integrates semantic knowledge tracing and model-based reinforcement learning.

Abstract: We introduce ExRec, a general framework for personalized exercise recommendation with semantically-grounded knowledge tracing. Our method builds on the observation that existing exercise recommendation approaches simulate student performance via knowledge tracing (KT) but they often overlook two key aspects: (a) the semantic content of questions and (b) the sequential, structured progression of student learning. To address this, our ExRec presents an end-to-end pipeline, from annotating the KCs of questions and learning their semantic representations to training KT models and optimizing several reinforcement learning (RL) methods. Moreover, we improve standard Q-learning-based continuous RL methods via a tailored model-based value estimation (MVE) approach that directly leverages the components of KT model in estimating cumulative knowledge improvement. We validate the effectiveness of our ExRec using various RL methods across four real-world tasks with different educational goals in online math learning. We further show that ExRec generalizes robustly to new, unseen questions and that it produces interpretable student learning trajectories. Together, our findings highlight the promise of KT-guided RL for effective personalization in education.

Personalized Exercise Recommendation with Semanticaly-Grounded Knowledge Tracing
Yilmazcan Ozyurt* ETH Zürich
Tunaberk Almaci"
ETH Zürich
Stefan Feuerriegel
Mrinmaya Sachan ETH Zürich
Munich Center for Machine Learning & LMU Munich
Abstract
We introduce ExRec, a general framework for personalized exercise recommen- dation with ceaaanicalll-gronndee knowledge tracing. Our method builds on the observation that existing exercise recommendation approaches simulate student performance via knowledge tracing (KT) but they often overlook two key aspects: (a) the semantic content of questions and (b) the sequential, structured progression of student learning. address this, our ExRee presents an end-to-end pipeline, from annotating the KCs of questions and learning their semantic representations to training KT models and optimizing several reinforcement learning (RL) methods. Moreover, we improve standard Q-learning-based continuous RL methods via a tailored model-based value estimation (MVE) approach that directly leverages the components of KT model in estimating cumulative knowledge improvement, We validate the effectiveness of our ExRec using various RL methods across four real- world tasks with different educational goals in online math learning. We further show that ExRec generalizes robustly to new, unseen questions and that it produces interpretable student learning trajectories. Together, our findings highlight the promise of KTTguidde RI. for effective personalization in education.
Introduction
The rapid rise of online learning platforms has revolutionized education by offering students access to interactive learning materials and exercises 3 29
53). 3). A key factor in enhancing student learning is personalization, where exercises are recommended based on students' evolving knowledge states
22 77).
central method for assessing learning progress knowledge tracing (KT), which models the temporal dynamics of student learning by predicting responses to future exercises. This enables real-time monitoring of knowledge states, which are structured around fundamental skills known as knowledge concepts (KCs), Over the years, numerous KT approaches have been developed [e. g.. Yet, only a small number works have actually leveraged KT for personalized exercise recommendations.
I 12
63 70
Recent methods for personalized exercise recommendation have integrated KT into reinforcement learning (RL) frameworks to simulate student behavior and learn optimal recommendation policies
14 42 75 82 However, these methods suffer from several key limitations: 1 they often rely on ID-baaedd embeddings, which neglects the semantics of questions; @ they define states as the full exercise history, making them computationally impractical for long exercise sequences; 3 the
'Equal contribution.
Correspondence Yilmazcan Ozyurt <yozyurt@ethz. ch> For a more comprehensive overview, we refer to 2 and 68
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Knowledge
nowrd
Limitations:
Ovorockinng semant questions.
Exercise Histery
d-3
excrdis??
which computaiionlly impractical
Defiting
fisalry,
Reward computation requiing inforence over quaetions, which limite realdime applistbillty
Palicy
6-1
1+1
4 Incompotiiie with divense Rl, algorthmm.
current state
Figure I: Overview of standard exercise recommendation and its limitations.
reward computation requires inference over all questions, which limits real-time applicability; and ④ they typically support only single RL algorithm.
To address these challenges, introduce ExRec, novel framework for personalized exercise recommendation with aaraanticallyggoonnde knowledge tracing. ExRec operates with minimal requirements, relying only on question content and exercise histories. It automates end pipeline: (i) it annotates each question with solution steps and KCs, (ii) learns semantically meaningful embeddings of questions and KCs, (iii) trains KT models to simulate student behavior and calibrates them to enable direct prediction of KC-level knowledge states, and (iv) supports efficient RL by designing compact student state representations and KC-aware reward signals. In addition, we propose model-based value estimation (MVE) approach that leverages the structure of the KT environment guide and stabilize the Q-learning of continuous RL algorithms.
end-to-
Overall, our contributions are as follows
We integrate automated KC annotation and contrastive learning modules to learn rich semantic representations of questions for downstream exercise recommendation.
We design a compact state representation for students, which eliminates the need to process full exercise histories, and efficient knowledge state compulation, eliminating the need to run inference over a large set of questions.
We introduce model-based value estimation technique that directly leverages the KT model for computing value functions to improve Q-learning-bant baaee continuous RI methods for our task.
We validate ExRec through extensive experiments using various RL algorithms to demonstrate its effectiveness in improving knowledge state estimation and exercise recommendation quality.
2 Related Work
Knowledge tracing (KT) models the temporal dynamics of students' learning process and predicts
the response of the student to the next exercise (20] Over the years, numerous KT methods have
been proposed [3] 15 21 28 30 35
38
40
43
44
49 50 52 56
63
67 69 70 79, 78
80 8 3
Yet, only few studies have explored KT for optimizing exereise recommendations.
primarily as a means to simulate student behavior and inform reinforcement learning (RL)-based policies for personalized sequencing. However, existing methods suffer from key limitations. Early approaches [ 36 42 rely on handcrafted rewards and treat questions predefined mappings KCs, preventing them from leveraging (iiilariyy between exercises. Similarly, methods such as TGKT-RL [14] require a predefined question-KCC graph, which is often unavailable, while others operate solely the KC level without considering individual exercises [10] 82). RCLAER (75) does not consider KCs of questions
Exercise recommendation has been widely studied in conjunction with KT
and learns embeddings for each question ID.
The existing works 440 36 42
75 82 face further common challenges. They do not effectively leverage question semanties, often relying ID-based embeddings simple heuristics. These methods define states as the entire exercise history, making them computationally impractical for long exercise sequences. Moreover, reward calculation in these methods requires inference over the full question set, making real-time decision-making inefficient. Finally, they support only a single RL method for exercise recommendation.
We address these limitations by ① learning rich semantic representations of questions, additionally allowing generalization to unseen exercises, 2 modeling compact student states, which eliminates
2 Code and trained models are provided htpp://gttuh.com/eerrryttytt
We discuss the exercise recommendation methods without KT in Appendix A

---

CL CARVEN
11 Sakiia
compppodiin
aronddnng 350
PN
Al irfonence:
Peond cusaton
Exneciss Hastory
slould roo surrzac??
Target KG
pil proekka red
Butoe
Policy
quedon unbedding Fle
Callosatio Makka
-niaalis with moddlt neleer
Phetdk
$11
prodicioon
Crodda
Carfert
rovodd
severd anvitonnee
76
poodicionn
predecs
Figure 2: Overview of ExRec framework. Numbered gray arrows show how modules interact.
the need for full exercise histories, 3 computing knowledge states directly, avoiding exhaustive inference over question sets, and supporting broad range of RL algorithms, including both discrete and continuous action spaces. Unlike prior works, we make our entire pipeline open-source to enable researchers to instantly build and test new exercise recommenders within our framework.
3 Preliminaries
Knowledge tracing (KT). KT aims to predict student's performance on the next exercise based on their learning history [63 70 83 student's exercise history over time steps is repre- sented as {e:M=1>> where each exercise consists of question q: E Q. of associated knowl- Na, edge concepts (KCs) )= with a.j C. and the student's binary response y; E {0, 1}. The KT model Gø predicts the probability of correct response for the next question, i.e..
Ør+1 = Go (g (c4ng))=1 Na411 {ex)i=)).
Exercise recommendation via RL. KT models can serve as RL environments to simulate student
learning behavior, formulated as Markov decision process (MDP) (S, A, R, 7) 4 I0 75). The state 8, E S represents the exercise history 8: = {e.)_..- and the action E A corresponds to selecting the next question q++1 E Q. The transition function P determines the next state based on the student's response: P( St, (t+1): S A x S > [0. 1]. The reward R measures the improvement the student's knowledge state after solving qt+ Finally, 7 the discount factor. In reward calculation, existing methods (..4 6.4400 [0] compute knowledge states by running Go over all
questions from a given KC at each step, leading to high computational overhead.
4 ExRec Framework
Our complete ExRec framework has four modules (see Fig. 2. (1) The first module takes the question content and annotates its solution steps and associated knowledge concepts (KCs) in an automated manner. (2) The second module learns the semantics of questions using the solution steps and KCs via a tailored contrastive learning objective. (3) The third module trains a KT model using question semanties and student histories. To better simulate student performance in an RL setup, we (i) train the KT model for next-exercise performance prediction, and (ii) calibrate its knowledge state predictions over each KC. (4) The final module involves an RL framework for exercise recommendation, using the calibrated KT model as the environment. For Q-learning-based continuous RL methods, we further improve training with model-based value estimation (MVE) approach, which incorporates the KT model itself to simulate future interactions and is beneficial for estimating the policy value.
3

---

4.1 KC Annotation via LLMs (Module 1)
Existing works on modeling learning progress [e. g., 16 51 73 typically require datasets containing thousands of questions with manually-annotated KCs. However, this annotation process is expensive and error-prone, it requires domain experts to assign KCs from hundreds of categories while maintaining consistency [8] In contrast, our framework circumvents the need for manual labeling, as described below
Informed by the recent literature (58), our ExRec framework automates KC annotation using large language model (LLM) through a three-step process. In doing so, we further instruct the LLM to align the KC annotation with the Common Core State Standards for Mathematic, to improve the consistency in the KC annotation and to support uptake of our framework in education @
(i) Solution step generation. Given question q Q. the LLM generates step-by-step solution Σ (01, 22, Jn using chain-of-thought prompting. Each step σы is sampled sequentially via
Pa(aT 1,01,..... ok-1)
(1)
where Ps is the LLM's probability distribution conditioned on the question and the previous steps.
(ii) KC annotation. The LLM then assigns a set of knowledge concepts {(i, C2. ...,, .., Cm based on the question and its solution steps, sampled iteratively via
Cj Po(ej q, 2 ,,,..., C--)).
(2)
(iii) Solution SC mapping. Not all KCs are applied in every solution step. To establish structured mapping, the L.M generates pairs iteratively via
(ok, ~ Po(a,; Cj g. Σ, M'),,
(3)
where M' represents previously assigned mappings. The final mapping
M = ((Tk, oj) ok E A Ck),
where C the set of KCs the solution step 0g mapped to. The mapping then serves as the foundation for representation learning in the next module.
4.2 Representation Learning via Contrastive Learning (Module 2)
Our ExRec framework employs contrastive learning (CL) to generate semantically meaningful embeddings for questions, solution steps, and knowledge concepts (KCs). Instead of using general- purpose embeddings, CL explicitly aligns questions and solution steps with their associated KCs while mitigating false negatives.
Embedding generation. Recall that, formally, after the annotation steps from the previous module, a question q has N solution steps {ax \ and M knowledge concepts {er JM We first encode the question content, its solution steps, and KCs using learnable (LLM) encoder (.):
20 Eg),, z = (ok), 2 = Ec
(5)
These embeddings are later optimized via contrastive learning.
False negative climination via clustering. Some KCs describe the same underlying skill using slightly different wording (e.g., interpreting a bar chart vs. reuding information from bar graph). Although semantically equivalent, these variants may yield different embeddings and be mistakenly treated negatives. To prevent this, pre-cluster KCs using semantic clustering function A(-), thereby ensuring that KCs within the same cluster are not considered as negatives:
A(c²) = A(er) = Cy' & &N(c;;;
(6)
where (hereNN(c)) M(cx) denotes the set of valid negative KCs for contrastive learning.
Contrastive learning for questions. Since each question q is associated with multiple KCs C {cI, (2....,,m.. J. its embedding should be closer to its relevant KCs while being pushed apart from
*In our experiments, we observed that existing KC annotations are often noisy and that LLM based annotation yields higher-quality and more coherent results. This observation echoes the findings of recent works [55] (58). Nevertheless, we provide an in-depth analysis in Appendix C Therein, we illustrate a representative example where our framework provides consistently higher-quality KC annotations than the original dataset.
5 http:://www.thcoorrstttnd.. We also experimented with Bloom's Taxonomy
org/Mathh
5 but found that was not equally helpful for setting.

---

irrelevant ones. For this, we use the loss
exp (sim(22, 22)/T)
1, 25) = 1--log log
(7)
exp sim (29, 25)/T)
Σ exp (immm22, (g))
cgEN(())
where sim(---) is the cosine similarity and is a temperature parameter. Contrastive learning for solution steps. Similarly, each solution step oę should be aligned with the KCs it practices. We thus use the loss
exp (sim( Ak 2f)/T)
(= 25) log
(8)
exp (sim(zg. <5)/T) +
Σ exp Gim 2 23))/T)
CxrEN(e,)
Final objective. Our framework jointly optimizes the embeddings of questions and solution steps:
M
L
M Lz 25)
Σ ==1
Σ L. 27).
eyEC
The learned embeddings enrich the input space KT models with semantically meaningful repre- sentations, enabling us to accurately capture the temporal dynamics of students' learning process.
4.3 KT Training with KC Calibration (Module 3)
Finally, we replace the standard, randomly initialized question embeddings of a KT model with our learned representations. Specifically, for each question 9; in the exercise history, define
2 with 27
N
Sk
(10)
where we average the question embedding =! with the (mean-pooled) solution-step embeddings aç. We then feed these vectors directly into an existing KT model.
Input preparation to KT. For a student's history of exercises {e: each exercise is now
modeled as a tuple, i. c., e (24 2. ). where 2 is the embedding of student's binary response (incorrect/correct) to the question qi and it is learned during the KT training. Note that our exercise modeling for e; different from a typical KT formulation (see Sec. 3). As the recent research [58] suggests, the integration of question embeddings enhances predictive accuracy, the KT model benefits from explicitly encoded question semantics rather than relying on question/KC IDs alone.
State representation. The state s; of a KT model equivalently represents the state of the RL environment in our ExRec framework (details Sec. (.4). Following prior literature (50) 59 63 81 the history of exercises (e:)_=1 can be encoded into a latent state via state encoder Eg as
(a) 8 = (o = ==)) or (b) 8, = Eø(5r-15 1; =y, =²"),
(11)
where (a) represents a flexible way encoding the entire sequence into a latent representation and (b) represents the recurrent way of encoding the sequence.
our framework, we use (b) to compute a compact state representation 8t+1 from the current state 8t. The benefit is that it avoids the need to keep the entire exercise history in the replay buffer.
KT training. The KT model predicts the performance of the student the next question g++1 via classifier Fg based on the current state st via
Di+1 (o 2241).
(12)
Overall, during the KT training, Fyg and F, are jointly trained on the entire history of exercises via binary cross entropy loss
Lired
Σ (o) log of (1 (1-v8h (1- (())
(13)
where 3f E {0, 1) is the ground-truth response of the student to the given question.
Calibration of the KT model for knowledge state prediction on any KC. Aligned with the education literature [19 20 (2), we define the knowledge state of a student for particular KC the expected performance of the student on all questions from the same KC. Specifically, for a particular KC E C time f, formalize the knowledge state of student
yf E ( [
Σ 07.
(14)
Q(c)
gEG(c)
where ((c) is the set of all questions from the KC
5

---

Of note. the above approach to computing knowledge state yf requires multiple inferences over set of questions, which becomes computationally challenging in real-time, especially for large corpus. We are first to address this challenge by allowing the KT model to directly predict the knowledge state at the inference time.
To speed up knowledge state prediction, we proceed as follows. Recall that, with the introduction of our representation learning module (Sec. (122). both question embeddings (i. e., 2²) and KC embeddings (i. e., =²) are already in the same embedding space. Therefore, we can formulate the knowledge state prediction as a prediction task over the KC embeddings. However, applying this prediction directly over the KC embeddings yields suboptimal results in practice (see Ap To achieve the desired performance, we further calibrate the KT model and bring its prediction over a KC embedding closer to the knowledge state estimated in Eq. [4 Specifically. for each student and each time step, we sample a KC e E C at uniformly random, and predict the knowledge state via
Vit1 = Fe (s,, =(+1);
(15)
I1 Then, we define our knowledge state prediction loss as where 8 is calculated as earlier in Eq.
Lxc = Σ
(uf log vf + (1 - -f) log(1 vf))
(16)
and define our calibration loss as
Lealib Lirred + LKc,
(17)
where we keep original prediction loss Lpred to retain the prediction performance of the KT model.
4.4 RL Framework for Exercise Recommendation (Module 4)
The final module of our ExRec formulates exercise recommendation as a reinforcement learning (RL) task, where the calibrated KT model serves the RL environment simulate student learning behavior. Here, we define a Markov decision process (MDP) based on the learned representations and knowledge state predictions from our KT model, which enables seamless integration with any standard RL algorithm.
MDP formulation. We define the RL problem as MDP M = (S,A, 7), where:
State space (S): The student's state at time I, denoted as 8,, is represented by the KT model's compact student state, i.c., S (s 2², = ), where Ef is the question embedding and 2!" represents the embedding of student's s past response.
Action space (A): The RI. agent selects the next exercise (t+1, represented by its embedding 2/+1 E Rd. The action space is originally continuous, and exercises are retrieved via semantic similarity in the learned representation space at the test time. Our ExRec framework additionally allows RL agents with discrete action, whose output represents the question ID g++1 Q. This question ID is then mapped to its original embedding æ[+1 as an action for our RL environment ?
Transition dynamics (P): The enviromment transition is governed by the KT model, which updates the student's ' state St-1 based on their response to the selected question. Specifically, the transition probability is defined as P(s++1 $t, 24 S x A x S + [0. 1], where the next state St+1 1 is determined by the student's correctness on the recommended question q++1. Given the KT model's 'sdel's predicted probability of correct response d P( = 8t, 1), the next state follows probabilistic update [8]
Fo 2141 =
with probability Di
St41
Eg 21+1 2(+1), with probability Di+1
2=0))
(18)
All other next states have zero probability. Therefore, the transition mechanism is aligned with the temporal dynamics of student's learning process, as modeled by the calibrated KT model.
Reward function (R): The reward reflects the improvement in the knowledge state. Given a KC c, we define the reward as the change in predicted knowledge state, i.c.
rt Di+1 Vf.
(19)
where of is the student's predicted knowledge state for e at time t, computed directly via the calibrated KT model.
Discountt factor (7): Controls the trade-off between short-term and long-term term learning gains.
'Regardless of action space being continuous discrete, the RI. environment, i.e the calihrated KT model, simulates the student hehavior the same way as it only processes the question emheddings. For clarity, we distinguish incomret/correet binary response embeddings 2/=0 and ==1 respectively.

---

Integrating KT into RL. Our approach natively integrates KT within the RL framework. This is enabled by the calibrated KT model, which provides: Compact student state representation without requiring full student history encoding, and (ii) direct knowledge state estimation instead of relying on indirect performance proxies. Thanks to this, various RI. algorithms (e.g., PPO [66]. TD3 [27].
SAC (31) can be used in seamless manner for optimizing exercise selection policies.
4.4.1 Model-Based Value Estimation
optimize exercise selection by estimating the expected future knowledge improvement using a critic network. Instead of training randomly initialized Q-network from scratch, we leverage the "full access to environment", i.e., the KT model, to design a model-based value estimation method [26].
For continuous-actinn RL algorithms with Q-learing [e. 8. 46
Bellman optimality in our RL environment. In standard Q-learning, the optimal action-value
function satisfies the Bellman equation 77:
Q" (St. 1) E [r. + + ($(+1,, (5(+1, s,,
|t,, 24,1
(20)
f112
Here, (*t 1)+1) represents the expected cumulative knowledge improvement if the RI agent selects question @++1 in state st.
Model-based critic design. Rather than learning Q(st, 1 1) from trial-and-errov interactions, we exploit the KT model's structure to directly estimate student progression. We construct novel critic network initialized with components from the pre-trained KT model:
The state transition function En is initialized as Eg, the recurrent knowledge tracing module from our KT model.
The value prediction function F., is initialized Fø, the calibrated KT prediction module.
Estimating the value function. Given the current student state 8, and selected question 2/ compute the expected future knowledge improvement based on two possible responses:
S241 (st 24 2/11) st+1
(0) (s بي_ =
(21)
(1) where 8441 and 8,41 are the next states if the student answers correctly incorrectly, respectively.
(0)
The predicted accumulated knowledge state for the targeted KC each scenario is obtained via
Vi+1 (1) (s (1) 20),) Vi+1 (0) (s (0) (0))
(22)
Final value computation. The final Q-value is computed as the expected knowledge improvement over the current knowledge state, weighted by the KT model's response prediction 21+11:
Q(t., D841 vi+1 (1) + (1 01+1) - DE.
(23)
where Df is the knowledge state of the student before answering the question, acquired by the KT
model. This formulation ensures that the value estimate aligns with the student's expected learning progression by directly leveraging the KT model rather than relying solely on learned Q-values.
By integrating this model-based critic, our RL agent benefits from (i) accurate, structured knowledge estimation, which reduces the need for excessive environment interactions, (ii) efficient policy learning, as the Q-function is informed by the calibrated KT model, and (iii) seamless adaptation to new students and exercises, as the critic inherently captures the student's 's evolving knowledge state.
Experimental Setup
Dataset. We use the XES3G5M dataset [5
large-scale KT benchmark with high-quality math questions. It contains 7,652 unique questions and 5.5M interactions from 18,066 students. As the original questions are in Chinese, we have translated them into English. See Appendix B for details.
RL environment. To ensure realistic student behavior, we initialize the RL environment by sampling student and encoding their first 100 exercises. The 100th latent state serves the initial state of the RL agent, ensuring a sufficient number of knowledge states across different environments to avoid cold-start problems. Following initialization, horizon 10 steps, where each step corresponds to an exercise recommendation. In evaluation, we compare RL algorithms across 2048 students, i. e., environments, from the test set of the dataset.
RL agents interact with the environment for fixed
Non-RI Baselines. To assess the extent to which RI. agents learn meaningful exercise recommen- dation policies, we first implement two RLL baselines: (i) random policy, which recommends
7

---

exercises uniformly at random from the existing question corpus. and (ii) historical data, where the knowledge state evolves based on the actual student responses
RL algorithms. To evaluate the effectiveness of our ExRec framework, we integrate a broad range of RI. algorithms for personalized exercise recommendation T0] We start with (i) continuous state-action methods, which include both value-based approaches (DDPG [46], SAC [31], and TD3 (27]) and policy-based algorithms (TRPO [65] and PPO
[66]). We then consider (ii) discrete action methods such as Discrete SAC 17. C5I 6 Rainbow 34 and DQN [34]. For DDPG, SAC, and TD3, we
also provide variants with model- value estimation (denoted "w/ MVE"), which utilize our calibrated KT model to improve value estimation and long-term reward propagation.
Implementation. Appendix piroovdds the details of our ExRec, including model configurations, training procedures, runtime, and hyperparameters, to ensure reproducibility and fair comparison.
Ablation studies. We also evaluate the performance of the earlier modules (e. g.. KC annotation, representation learning, and KT training and calibration) in Appendix D
5.1 Evaluation Tasks
We evaluate our ExRec framework across four real-world tasks, each designed to reflect a different educational objective. The first task aligns with prior literature 4 TO 14], while the remaining three are novel and address practical use cases that have been previously underexplored. A brief summary of each task is provided below, with full formulations and scoring details in Appendix E
Task 1: Global knowledge improvement. We first follow the earlier literature and perform the standard task focusing on holistic student learning by optimizing improvements across all KCs. Instead of targeting a single concept. we define the reward as the aggregate change in the student's knowledge state, averaged over all KCs. Compared to the existing works, we achieve this efficiently without running inference over the full question set, thanks to our calibrated KT model.
Task 2:
Knowledge improvement in practiced KC. We simulate a scenario where the student is focused on mastering specific KC. For each student, we identify the most frequent KC in their last 10 exercises and set it as the target. This allows the RL agent to support ongoing learning by recommending conceptually aligned questions.
Task 3: Knowledge improvement in upcoming KC. To emulate curriculum progression, we compute a transition matrix between recently practiced and upcoming KCs across students. The next target KC is sampled from this distribution, enabling the RL agent to recommend questions that align with natural learning paths observed in the data.
Task 4: Knowledge improvement in weakest KC. At each step, we adaptively target the KC with the lowest estimated knowledge among all KCs. This way, the RL agent focuses the student's weakest areas and personalizes learning toward balanced conceptual understanding. For evaluation, we compute the average knowledge improvement in these targeted KCs.
Results
Figure shows the performance of each method as a percentage of the maximum achievable improve- ment in the knowledge state. We find the following: (1) Across all tasks, non-RL baselines yield marginal even negative gains, which highlights the need for tailored recommendation policies. (2) Discrete-action methods generally perform well in simpler tasks (1-3), which is due to their ability optimize toward static target. (3) Among continuous aatiin RL methods, value-based approaches (DDPG, TD3, SAC) consistently outperform policy-based ones (TRPO, PPO). Our model-based value estimation (w/ MVE) proves especially effective, consistently boosting the perfor- mance of continuous balue-based RL methods across all tasks. (5) Particularly at task 4, which is substantially more challenging due that the target KC may change at each step per student, our MVE approach boosts the performance of continuous value-based RL methods beyond the level of discrete methods. Takeaway: Our ExRec shows the importance of tailored exereise recommendation policies. It also demonstrates how fully leveraging the KT environment, via our model-based value estimation, can lead to large improvements in RL performance for educational settings.
shows
9 Unlike all other methods. historical data does not rely on the model sample student responses, as ground-truth interactions are available. We implement RL algorithms using the Tianshou library 74
8

---

Heterca Pite
103
19P
monee Mec
Kyeeem Falp
Ropo
101
1990
AR
monte MeC
pipe
(a) Task I: Global Knowledge Improvement.
(b) Task 2: Knowledge Improvement in Practiced KC.
ALPAL
103
lanp
Dlort MC
Aendem Macy
Dar
36
MCC
(c) Task 3: Knowledge Improvement in Upcoming KC (d) Task Knowledge Improvement in Weakest KC.
Figure 3:
Knowledge improvements across four tasks, averaged over 2048 students in the set. Our framework supports range of RL algorithms and enables extensive comparison among methods.
6.1 Use Case: Extending the Question Corpus
To test whether our framework generalizes to new, unseen questions, we extend the original question corpus by generating three times more questions using GPT-4o (see Appendix H for prompt details and examples). Each generation introduces additional KCs while preserving the conceptual grounding of the original question. As shown in Figure 4 default models gener- ally experience drop in performance under the extended corpus. In contrast, models aug- mented with model-based value estimation (w/ MVE) are robust and, in many cases, even improve student knowledge more effectively by leveraging the broader question set.
Interd
Original Curua
Exiended Corpe
Figure 4: Results extending question corpus.
6.2 Use Case: Visualizing Conceptual Growth
We visualize how our framework supports learning trajectories. We use a random student from Task 4 for illustration. Figure 5 shows how the student's knowledge states evolve across steps |T| Here, we compare a value-based continuous- action model (DDPG) and its improved versions w/ MVE and w/ MVE + extended corpus against a random policy. (1) The random policy fails to produce meaningful gains, as it recommends exercises without addressing the weaknesses of the student. (2) DDPG achieves moderate improvements, underscoring the benefit of learning tailored policies through our framework. (3) Our MVE approach leads to significantly larger gains. (4) Further
improvements observed when training the extended corpus, which broadens the action space while remaining compatible with our framework. (5) Notably, while vanilla DDPG may take several steps to address the same KC (c.g., KC 1326), our enhanced variants treat KCs more efficiently by adapting quickly, which enables focusing on other weak concepts.
Discussion
ExRec addresses limitations of earlier methods for personalized exercise recommendations by in- troducing semantically grounded question representations, compact student state modeling, and efficient knowledge state estimation for reward computation. As result, we demonstrate that the personalized exercise recommendation benefits greatly from modeling both the semantics of exercises and the structure of student learning. Further, our model-based value estimation (MVE) consistently improves the performance of Q-learning- continuous RI. methods, particularly in settings like
The knowledge state evolution for wider set of RL policies be found in Appendix

---

Random Policy
DOPG
DDPG MV!
DDPG ml NVE (Eatended Corpus)
1240
326
0ito 30000
1300
965
1220
1304
1040
965
1220
1451
11:
Time 2013
Time
Tims
Stop
Figure 5: Knowledge state evolution for a single student across different policies (Task 4), At each step, the weakest KC is targeted, and the corresponding knowledge trajectory is shown. White dashed boxes mark changes after each recommendation. A KC may appear more than once if it remains the weakest. KC IDs are for visualization only: our framework uses semantic KC embeddings, not IDs.
targeting the weakest KC, where the target KC shifts dynamically over time. Finally, our framework generalizes to unseen exercises and supports fine-grained analyses of evolving student know knowledge. As ExRec relies on KT models as environment and semantic embeddings as question representations, future work could explore stronger KT models or difficulty-aware question representations for even richer personalization. Together, these findings suggest that leveraging KT as an environment opens new avenues for scalable personalization in education.
10

---

References
[1] Ghodai Abdelrahman and Qing Wang. Knowledge tracing with sequential key-value memory networks. In SIGIR, 2019.
Ghodai Abdelrahman, Qing Wang, and Bernardo Nunes. Knowledge tracing: A survey. ACM Computing Surveys, 2023.
[3] Olasile Babatunde Adedoyin and Emrah Soykan. Covid-19 19 pandemic and online learning: The challenges and opportunitics. Interactive learning environments, 2023.
[4] Fangzhe Ai, Yishuai Chen, Yuchun Guo, Yongxiang Zhao, Zhenzhu Wang. Guowei Fu, and Guangyan Wang. Concept-aware deep knowledge tracing and exercise recommendation in an online learning system. EDM. 2019.
[5] Lorin W Anderson and David R Krathwohl. A taxonomy for learning, teaching, and assessing: A revision of Bloom's 's taxonomy of educational objectives: complete edition. 2001.
[6] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforvement learning. In ICML, 2017.
[7] Richard Bellman. Dynamic programming. Science, 1966.
[8] Norman Bier, Stephen Moore, and Martin Van Velsen. Instrumenting courseware and leveraging data with the Open Learning Initiative (OLI). In Learning Analyties & Knowledge, 2019.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1rXiv: 2016.
[10] Dejun Cai, Yuan Zhang, and Bintao Dai. Learning path recommendation based on knowledge tracing model and reinforcement learning. In ICCC, 2019.
[11] Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. Advances Knowledge Discovery and Data Mining, 2013.
[12] Hao Con, Kenneth Koedinger, and Brian Junker. Learning factors analysis a general method for cognitive model evaluation and improvement. In International conference on intelligent tutoring systems, 2006.
[13] Jiahao Chen, Zitao Liu, Shuyan Huang, Qiongqiong Liu, and Weiqi Luo. Improving interpretability of deep sequential knowledge tracing models with question- centric cognitive representations. In AAAI, 2023.
[14] Zhanxuan Chen, Zhengyang Wu, Yong Tang, and Jinwei Zhou. TGKT-based personalized learning path recommendation with reinforcement learning. In KSEMr, 2023.
[15] Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Back, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. Towards an appropriate query, key, and value computation for knowledge tracing. In Learning @ Scale, 2020.
[16] Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. EdNet: A large-scale hierarchical dataset in education. In AIED, 2020.
[17] Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint 110.00911. 2019.
[18] Richard Clark. Cognitive task analysis for expert-based instruction in healthcare. In Handbook of Research on Educational Communications and Technology. Springer, New York, NY, 2014.
[19] Albert Corbett, Megan McLaughlin, and K Christine Scarpinatto. Modeling student knowledge: Cognitive tutors in high school and college. User modeling and user-adapted interaction, 2000.
[20] Albert Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 1994.
[21] Chaoran Cui, Yumo Yao, Chunyun Zhang, Hebo Ma, Yuling Ma, Zhaochun Ren, Chen Zhang, and James Ko. DGEKT: A dual graph ensemble learning method for knowledge tracing. Transactions on Information Systems, 2024.
[22] Peng Cui and Mrinmaya Sachan. Adaptive and personalized exercise generation for online language learning. In ACL, 2023.

---

[23] Jacob Devlin. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[24] Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap. Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeey Arora. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving. ICML, 2024.
[25] Ecdi. Ecdi dataset. https://wvvt www.:eeeeeicoomoo 2024.
[26] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. In ICML, 2018.
[27] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICMI,, 2018.
[28] Aritra Chosh, Neil Heffernan, and Andrew S Lan. Context-aware attentive knowledge tracing. In KDD, 2020.
[29] Begoña Gros and Francisco affordances of e-learning. In Learning, design, and technology: An international compendium of theory, research, practice, and policy. 2023.
J García-Peñillvo Future trends in the design strategies and technological
[30] Xiaopeng Guo, Zhijie Huang. Jie Gao, Mingyu Shang, Maojing Shu, and Jun Sun. Enhancing knowledge tracing via adversarial training. In ACM Multimedia, 2021.
[31] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. In ICML, 2018.
[32] Ibrahim Abou Halloun and David Hestenes. The initial knowledge state of college physics students. American journal of Physicx, 1985.
[33] Yu He, Hailin Wang, Yigong Pan, Yinghua Zhou, and Guangzhong Sun. Exercise recommendation method based on knowledge tracing and concept prerequisite relations. CCF Transactions on Pervasive Computing and Interaction, 2022.
[34] Matteo Hessel. Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement lcarning. In AAAI, 2018.
[35] Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, and Jian Weng. Towards robust knowledge tracing models via k-sparse attention. In SIGIR, 2023.
[36] Zhenya Huang, Qi Liu, Chengxiang Zhai, Yu Yin, Enhong Chen, Weibo Gao, and Guoping Hu. Exploring multi-objective exercise recommendations in online education systems. In CIKM, 2019.
[37] Yujia Huo, Derek F Wong, Lionel M Ni, Lidia Chao, and Jing Zhang. Knowledge modeling via contextualized representations for Istm-based personalized exereise recommendation. Information Sciences, 2020.
[38] Yoonjin Im, Eunseong Choi, Heejin Kook, and Jongwuk Lee. Forgetting aware linear bias for attentive knowledge tracing. In CIKM, 2023.
[39] Tanja Küser, Severin Klingler, Alexander G Schwing, and Markus Gross. Dynamic bayesian networks for student modeling. Transactions on Learning Technologies, 2017.
[40] Fucai Ke, Weiqing Wang, Weicong Tan, Lan Du, Yuan Jin, Yujin Huang, and Hongzhi Yin. HITSKT: A hierarchical transformer model for cession-aware knowledge tracing. Bhooleddge-Bssedd Systems, 2024.
[41] Deepak Kem. Personalised and adaptive learning: Emerging learning platforms in the of digital and smart learning. International Journal of Social Science and Human Research, 2022.
[42] Scounghun Kim, Woojin Kim, and Hyconcheol Kim. Learning path construction using reinforcement learning and bloom's 's taxonomy. In Intelligent Tutoring Systems, 2021.
[43] Jinseok Lee and Dit- Yan Yeung. Knowledge query network for knowledge tracing: How knowledge interacts with skills. In Learning Analytics & Knowledge, 2019.
12

---

[44] Unggi Lee, Jiyeong Bae. Dohee Kim, Sookbun Lee, Jaekwon Park, Taekyung Ahn, Gunho Lee. Damji Strattonn, and Hyeoncheol Kim. Language model can do knowledge tracing: Simple but effective method to integrate language model and knowledge tracing task. arXiv preprint urXiv:2406.02888 2024.
[45] Shufei Li. Xingwu Liu, Xiaolan Tang. Xi Chen, and Juhua Pu. Mikt4rec: Enhancing exercise recommen- dation through multitask learning with knowledge tracing. IEEE Transactions on Computational Social Systems, 2024.
[46] Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[47] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong. Yu Su, and Guoping Hu. EKT: Exercise-aware knowledge tracing for student performance prediction. Transactions on Knowledge and Data Engineering. 2019.
[48] Zitao Liu. Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang. and Weiqi Luo. pyKT: A Python library to benchmark deep learning based knowledge tracing models. In NewrlPS, 2022.
[49] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang. Boyu Gao, Weiqi Luo, and Jian Weng. Enhancing deep knowledge tracing with auxiliary tasks. In The Web Conference, 2023.
[50] Zitao Liu. Qiongqiong Liu, Jiahao Chen, Shuyan Huang, and Weiqi Luo. simpleKT: A simple but tough-to-beat baseline for knowledge tracing. In ICLR, 2023.
[51] Zitao Liu, Qiongqiong Liu, Teng Guo, Jiahao Chen, Shuyan Huang, Xiangyu Zhao, Jiliang Tang. Weiqi Luo, and Jian Weng. XES3G5M: A knowledge tracing benchmark dataset with auxiliary information. In NewrIPS, 2023.
[52] Ting Long. Yunfei Liu, Jian Shen. Weinan Zhang, and Yong Yu. Tracing knowledge state with individual cognition and acquisition estimation. In SIGIR, 2021.
[53] Fitriatul Masitoh, Ima Fitriyah, and Irma Khairany Mumtaza. Ef teachers' perceived efficacy of digital platforms in varied online english classes in higher education. Journal of Languages and Language Teaching, 2024.
[54] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[55] Steven Moore, Robin Schmucker. Tom Mitchell, and John Stamper, Automated generation and tagging of knowledge components from multiple-choice questions. In Learning@ Scale, 2024.
[56] Koki Nagatani, Qian Zhang, Masahiro Sato, Yan-Ying Chen, Francine Chen, and Tomoko Ohkuma. Augmenting knowledge tracing by considering forgetting behavior. In WWW, 2019.
[S7] Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In JEEE/WIC/ACM International Conference on Web Intelligence, 2019.
[58] Yilmazcan Ozyurt, Stefan Feuerriegel, and Mrinmaya Sachan. Automated knowledge concept annotation and question representation learning for knowledge tracing. arXiv preprint arXiv:2410.01721 2024.
[59] Shalini Pandey and George Karypis. A self-attentive model for knowledge tracing. In EDM, 2019.
[60] Shalini Pandey and Jaideep Srivastava, RKT: Relation-aware self-attention for knowledge tracing. In CIKM, 2020.
[61] Philip Pavlik, Hao Cen, and Kenneth R Koedinger. Performance factors analysis- a new alternative to knowledge tracing. In Artificial intelligence in education, pages 531-538. los Press, 2009.
[62] Pei Pei, Rodolfo C Raga Jr, and Mideth Abisado. Enhanced personalized learning exercise question recommendation model based on knowledge tracing. International Journal of Advances in Intelligent Informatics, 2024.
[63] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Dicks-iicsttin.. Deep knowledge tracing. In NeurlPS, 2015.
[64] Nils Reimers and Iryna Gurevych. Seatence-BERTT Sentence embeddings using siamese BERT- networks. In EMNLP, 2019.
13

---

[65] John Schulman, Sergey Levine. Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
[66] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[67] Shuanghong Shen, Zhenya Huang, Qi Liu, Yu Su, Shijin Wang, and Enhong Chen. Assessing student's dynamic knowledge state by exploring the question difficulty effect. In SIGIR, 2022.
[68] Shuanghong Shen, Qi Liu, Zhenya Huang, Yonghe Zheng, Minghao Yin, Minjuan Wang, and Enhong Chen. survey of knowledge tracing: Models, variants, and applications. Transactions on Learning Technologies. 2024.
[69] Xiangyu Song, Jianxin Li, Qi Lei, Wei Zhao, Yunliang Chen, and Ajmal Mian. Bi-CLKT: Bi-graph contrastive learning based knowledge tracing. Knowledge-Bad Based Systems, 2022.
[70] Shashank Sonkar, Andrew E Waters, Andrew S Lan, Phillip Grimaldi, and Richard G Baraniuk, gDKT: Question-centric deep knowledge tracing. EDM, 2020.
[71] Jill-Jënn Vie and Hisashi Kashima. Knowledge tracing machines: Factorization machines for knowledge tracing. In AAAI, 2019.
[72] Han Wan, Zihao Zhong, Lina Tang, and Xiaopeng Gao. Pedagogical interventions in SPOCs: Learning behavior dashboards and knowledge tracing support exercise recommendation. IEEE Transactions on Learning Technologies, 2023.
[73] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Herúández- Lobato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Instructions and guide for diagnostic questions: The NeurlPS 2020 education challenge. arXiv preprint NkK::2007..106.1 2020.
[74] Jiayi Weng, Huayu Chen. Dong Yan, Kaichao You. Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: highly modularized deep reinforcement learning library. JMLR, 2022.
[75] Siyu Wu, Jun Wang, and Wei Zhang. Contrastive personalized exercise recommendation with reinforcement learning. IEEE Transactions on Learning Technologies, 2023.
[76] Zhengyang Wu, Ming Li, Yong Tang, and Qingyu Liang, Exercise recommendation based on knowledge concept prediction. Knowledge- -Bssed Systems, 2020.
[77] Austin Xu, Will Monroe, and Klinton Bicknell. Large language model augmented exercise retrieval for personalized language learning. In Learning Analytics and Knowledge Conference, 2024.
[78] Chun-Kit Yeung. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. EDM, 2019.
[79] Chun-Kit Yeung and Dit-Yan Yeung. Addressing two problems in deep knowledge tracing via prediction- consistent regularization. In Learning Scale, 2018
[80] Yu Yin, Le Dai, Zhenya Huang, Shuanghong Shen, Fei Wang, Qi Liu, Enhong Chen, and Xin Li. Tracing knowledge instead of patterns: Stable knowledge tracing with diagnostic transformer. In The Web Conference, 2023.
[81] Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In WWW, 2017.
[82] Xiankun Zhang, Yuhu Shang, Yimeng Ren, and Kun Liang. Dynamic multi-ohjective sequence- wise recommendation framework via deep reinforcement learning. Complex & Intelligent Systems, 2023.
[83] Hanqi Zhou, Robert Bamler, Charley M Wu, and Álvaro Tejero-Cantero. Predictive, scalable and inter- pretable knowledge tracing on structured domains. In /CLR, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We elaborated on the technical novelties of our framework in Section 4aadd showed the effectiveness of our method in the detailed experiments Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explained in our discussion section that the performance of exercise recommendation policies relies the performance of KT models for accurate student simulation. Hence, the future work can focus on more accurate modeling of student trajectories. Further, the current work does focus on the difficulty of the questions, which can be incorporated into the representation learning module of our framework.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provided the full derivation of our complete framework (including all of its individual components) in Section and carefully provided the assumptions when applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We described our complete architecture in Section We further made our code publicly available and added clear instructions in our repository. We provided implementation details and listed computing resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide full access to our code as provided in our repository with clear instructions to framework. Further, provided implementation details and listed computing resources in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/nips.cc/public/guue gwtcc//gueee/eeeeeeee CodeSubmi www.sbmmmsssinno. ssiobooiiisiiocccccicooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provided all the experimental details comprchensively in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provided the error bars in our results when the standard deviations overlap.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each module of our framework, we listed the compute requirements in Appendix C
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We read NeurIPS Code of Ethics and confirm that our paper conforms, in every respect, with the NeurIPS Code of Ethics. Specifically, our work is using the publicly available datasets which do not demonstrate any privacy concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [Yes]
Justification:
We discuss the potential positive and negative societal impacts of our work in
Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: In our work, we are not releasing new data and/or models, therefore, our work does not pose a primary risk. Although our work does not directly release new data or models, we recognize that the exercise recommender RI. agents must be trained for the right set of rewards, and must intend no harm.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We carefully cited the creators or original owners of all the models and the datasets presented in paper.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Both in the paper and the repository, we provided the detailed documentations regarding the assets provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Module I and Module (in Section 4) of our framework involve the usage of LLMs in terms of both prompting and finetuning. We provided the details of their usage in both Appendix and our repository.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
21

---

A Alternative Approaches to Exercise Recommendation
Various works exereise recommendation have explored range of strategies, often without simulat- ing student learning trajectories. Several approaches rely on static, rule-based logic, recommending knowledge concepts (KCs) based on manually defined prerequisite graphs [33], proxy objectives like novelty [76), or instructor-specte
difficulty levels [62], rather than learning an adaptive policy, Other methods frame recommendation as supervised learning problem, optimizing for immediate performance or success 37 45 instead of prioritizing long-term knowledge improvement. A com- mon thread these works the reliance on hand-crafted features or ID-based representations, which overlooks the semantic content of questions and KCs [33 37 45), with some systems providing decision support dashboards for instructors rather than a fully automated recommendation policy [72]. These limitations motivate more dynamic approaches that can model the evolution of student knowledge over time.
22

---

B Dataset
The XES3G5M dataset [5]] is large-scale benchmark for knowledge tracing, collected from real-world online math learning platform. It comprises 18,066 student histories, totaling over 5.5 million interaction records across 7,652 unique math questions. These questions are annotated with 865 leaf-level knowledge concepts (KCs). Compared to existing KT datasets, XES3G5M provides rich auxiliary information, including: (1) full question texts and types (multiple-choice and fill-n-tthe-blanke blank), (2) and the final answers of the provided questions.
For compatibility with framework, translated all question texts from Chinese English using high-quality commercial translation tool, and ensured formatting consistency through post-processing. We provide the translated dataset along with our annotations in the repository.
We note that, while the dataset provides manually annotated KCs, we found that these annotations often low-quality (see Appendix G Therefore, systematically re-annotate questions using our Module (KC Annotation via LLMs), resulting in improved consistency and interpretability. In the original dataset, each question is associated with 1.16 KCs average. In comparison, ExRec framework identifies KCs for the majority of questions. This can be attributed to our framework's 's ability to provide more comprehensive and modular KC annotations compared the original distributions. Fig. shows the distribution of number of KCs per question.
Number
1110
1034
205
23
11
5
Number Knowledge Concopts per Question
Figure 6: Distribution of number of KCs per question annotated by our framework.
Our framework produces 5,139 unique KC annotations, which are grouped into 1,377 clusters based semantic similarity. Compared to our earlier work [58), which produced 8,378 unique KCs and 2,024 clusters, this represents substantial improvement in the consistency of the KC annotations. We attribute this improvement to two key design choices: (i) instructing the LLM align with the Common Core State Standards for Mathematics to ensure canonical phrasing of KCs, and (ii) prompting the model to explicitly reason about the relevance of each KC in context (see Appendix (F). Fig. | iillustrates the most frequent clusters across the dataset. We release all annotated KCs and their clustering results support reproducibility and further research.
23

---

Praperties of paraliclograms
Combining like terms
Inclusion Exclusion on Principle
Salving systems linear equations
Relative speed
Setting up and solving linear equations
Counting Cardinality
Properties squares
Set operations and Venn diagrams
Place Value
Logical reasoning and deduction
Writing algebraic expressions
Solving Equations
Understanding averages
Divisibility rules
Proportional reasoning
Area of rectangles
Perimeter of a square
Variables and Expressions
Creating equations problems
Pomutations
Inaqualitiss
Solving real-world problems with simple linear equations
Combinatorics
Modular arithmetic
Permutations and combinations
Perimeter rectangle
Distributive Property
using variables in mathematical expressians
Algebraic expressions and equations
Order operations
value understanding
Substitution
Algebraic ma onniulaation
Logical reasoning
Linear Equations
Artthmetic Sequences
Nieebraic expressions
Solving linear equations
Arithmetic Operations
200
400
600
800
1000
1200
1400
Number of Questions
Figure 7: Most representative KCs across all questions in the dataset. The result is shown after clustering semantically similar KCs.
24

---

C Implementation Details
KC Annotation (Module 1). We leverage the reasoning capabilities of OpenAl's GPT-4g to automate all three stages: solution step generation, KC annotation, and solution step KC mapping. We carefully steer model towards more consistent KC annotations across different Math questions, which is ignored in earlier works [55 58.. For this, specifically instruct the LLM to (1) use Common Core State Standards for Mathematics as a reference and (2) reason about why and how each KC is partieularly relevant for the current question. We set the temperature to 0 for deterministic outputs. For each question in the dataset, the model is queried three times, once per stage. Across the 7,652 questions XES3G5M, the total cost of prompting remains approximately 50 USD.
Question Representation Learning (Module 2). To learn semantically rich question embeddings, adopt multi-stage contrastive training approach similar [58]. First, we cluster the Sentence- BERT 64 embeddings of KCs using HDBSCAN [T] to identify semantically close groups and mitigate false negatives during contrastive training. We use cosine similarity the distance metric, and we set the minimum cluster size and minimum samples to
We then fine-tune a BERT encoder (23 using contrastive loss over question-KC and solution step-KC pairs. To differentiate the input types, we introduce three special tokens-[Q], [S], and pre]-preeended to the question content, solution step, and KC, respectively. Their [CLS] token embeddings are used to represent the full input sequence.
We train the model for 50 epochs using a batch size of 32, a learning rate of 5e-5, dropout of 1. and a temperature of 0.1 in the similarity function. The training is performed on an NVIDIA A100 GPU (40GB) and completed in under 6 hours.
After training, question embeddings are computed by encoding both the question content ((Q)...) and its solution steps ([S]... and aggregating the resulting embeddings as described in Sec. 4.3 Importantly, this inference does not require KC annotations newly added questions without running the L.M again.
possible to directly embed
KT training with KC calibration (Module 3). For efficient state representation, we leverage a KT model that encodes the exercise history into a latent state in a recurrent manner. Specifically, choose an LSTM architecture for the state encoder Es, which has been used in earlier KT works The classifier F9 is a multi-layer perceptron. During our entire experiments, fixed the dimensionality of state 8 300. To ensure equal representation with the question embeddings, we further increase the dimensionality of s, to 768 (same as the dimensionality of question embeddings) via a linear layer before feeding it into the classifier Fo.
44 56
63
70 79
During KC calibration of the KT model, we approximate a student's knowledge state on given concept E C by sampling 20 relevant questions associated with from the question corpus and averaging the predicted correctness scores. To ensure stability and prevent the moving target problem, these predictions are computed using a fixed checkpoint of the KT model obtained after standard training is completed.
Both the initial KT training and the subsequent KC calibration are performed using batch size of 512 and a learning rate of 2e-5. We used NVIDIA GeForce RTX 3090 with 24GB GPU for the training of KT models. The training takes approximately 2.5 hours on a single GPU and the inference (per question) takes 8.7 milliseconds. Of note, fast inference the key element to ensure efficient. student simulation in RI. environment.
For implementation, we customize the рyКT library 48 to support our custom model architecture and KC-level supervision. We provide our custom KT architecture and its training details in our repository.
RL framework for exercise recommendation (Module 4). We integrate our trained KT model as an RL environment within the Tianshou library (74). following the OpenAl Gym API specification [9] to ensure seamless compatibility. This design allows multiple RL agents to interact with the KT-based environment for comprehensive and flexible benchmarking of exercise recommendation policies.
12 httpss//plaatorr wwtpaaattooooooooo......oa
15 We note that we do not sacrifice the accuracy by choosing an LSTM encoder instead of attention-base encoder. As found in earlier research (58). the after incorporating the rich question representations.
KT models achieve state-of of ohe-the-rt performances
25

---

Unlike the default Tianshou strategy, which creates N environment instances and manages them asynchronously via tuntiithreeding our framework supports parallel student simulations by pro- cessing all N environments significantly reduces memory overhead. We release all necessary environment wrappers and code for easy integration and reproducibility.
single batch on the GPU. This enables more efficient training and
Similar to the training of KT models, used NVIDIA GeForee RTX 3090 with 24GB GPU for the training of RL algorithms and each training is completed under an hour. RL inference per exercise recommendation takes 78.5 milliseconds on single GPU, which shows the effectiveness when recommending the exercise in real time.
The hyperparameter details of RL algorithms can be found in Table 1
26

---

Table 1: Hyperparameter tuning of RL algorithms.
Method
Hyperparameter
Tuning Range
methods
Question and KC Emhedding Size
768
Critic Network Up Projection Size
1200
Critic Network Hidden Size
300
Max Steps
10
Reward Scale
1000
Gamma
0.99
Learning Rate
[5 10 1.-
10"11
DDPG
Tau
[0.001, 0.01, 0.05, 0.1]
SAC
Tau
[0.01, 0.015, 0.02, 0.05]
Alpha
[0.1, 0.2, 0.3,0.4, 0.5]
Deterministic Evaluation
[True, False]
TD3
Tau
[0.05, 0.005,0.01)
Actor Update Frequency
4]
Policy Noise
[0.1. 0.2]
Noise Clip
[0.5]
TRPO
Actor Step Size
[0.5, 0.025]
Advantage Normalization
[False]
Discount Factor
[0.99, 0.995]
Gae Lambda
[0.95, 0.97]
Max KL
[0.01, 0.02]
Optim Critic Iters
[5]
Reward Normalization
[False]
Repeat Per Update
[1]
PPO
Advantage Normalization
[False]
Deterministic Evaluation
[False]
Discount Factor
[0.99, 0.95]
Entropy Coefficient
[0.01, 0.05]
L-Clip
10.1, 0.2]
Gae Lambda
[0.9, 0.95]
Value Loss Coefficient
[0.5, 1.0]
Discrete SAC
Alpha Estimation Step
[0.1, 0.2]
[1. 10]
[0.005, 0.01, 0.05]
CS1/Rainbow
Clip Loss Gradient
[True, False] [0.99, 0.995]
Discount Factor
Double Network
[True, False]
Num Atoms
[17]
Target Update Frequency
[0, 10]
Vnax
[1000]
Vann
1000]
DQN
Clip Loss Gradient
(ruee False]
Discount Factor
[0.99, 0.995]
Estimation Step
[1]
Double DQN
(True False| [0.1.10]
Target Update Frequency
specified parameters, use values Tianshou library.
27

---

D Evaluating Individual Modules of ExRec
While the main paper focuses on evaluating the effectiveness of our framework for exereise rec- ommendation, this section provides a closer examination of the upstream modules: KC annotation, representation learning, and knowledge tracing with KC calibration. Understanding the quality and behavior of these modules is essential, as they form the foundation upon which our recommendation policy is built.
KC Annotation (Module 1). Our framework builds on recent advances demonstrating the viability of using large language models (LLMM)) for automated knowledge concept (KC) annotation of math problems [55] (24]. Prior work shows that LLM LLM-generated annotations not only improve downstream reasoning performance [24], but are also preferred by human annotators over expert-created labels [55]. Our approach follows the annotation pipeline of Ozyurt et al. [58], which leverages step-by-step solutions to improve both the accuracy and conceptual granularity of KC labels. Their study found that LLMs using solution steps were preferred by annotators 86.9 % of the time compared to those without steps, and were favored 95 % of the time over the original KC annotations in the XES3G5M dataset. We extend this approach by explicitly instructing the LLM to align its output with the Common Core State Standards and to reason explicitly about the relevance of each KC. This leads to substantially more consistent annotations: our framework produces 5,139 unique KC labels grouped into 1,377 semantically coherent clusters, compared to 8,378 KCs and 2,024 clusters in Ozyurt et al. [58]. indicating improved coherence and reduced redundancy in the KC space.
Question Representation Learning (Module 2). To assess the effectiveness of our representation learning module, we evaluate how well the learned embeddings reflect the KC structure. Specifically, we measure the retrieval quality of semantically related questions via aieeraa F1 score at the cluster level. For each KC cluster, we randomly select a representative KC and use the LLM to compute its embedding. We then retrieve the top-N nearest questions in the embedding space, where N is the number of questions associated with that cluster. Retrieval performance is calculated by comparing the retrieved questions against the true cluster members, and we aggregate scores across clusters to compute the final micro-F1.
Without representation learning, the default LLM embeddings yield micro-F1 score of 0.2305, reflecting limited alignment between questions and their associated KCs. After applying our con- trastive training objective, this score rises substantially to 0.8865, indicating a dramatic improvement in semantic coherence.
Figure 8 visualizes this improvement qualitatively. On the left, questions related to the same KC are scattered across the embedding space, resulting in poor semantic grouping. On the right, representa- tion learning produces tightly clustered embeddings for related questions, clearly highlighting the benefit of our training procedure. This clustering not only boosts retrieval accuracy but also enhances downstream modules that rely on precise semantic alignment
KT training with KC calibration (Module 3). In contrast to prior KT frameworks that train question embeddings from scratch or continue updating them during KT training [88] 51 58] our approach preserves the semantically enriched embeddings obtained in Module 2 by freezing them throughout KT training. This design ensures that both questions and KCs remain in shared, interpretable embedding space, allowing for direct querying of KC mastery. While freezing these embeddings leads to a small drop in predictive performance (AUC decreases from 82. to 81.26), it still substantially outperforms standard ID-based initialization (78.33 AUC). More importantly, this setup enables interpretable mastery estimation the KC level by querying the KT model with KC embedding directly. To evaluate this capability, we sampled 20 questions associated with each KC and compared their average predicted mastery against the KC-query prediction. The resulting mean absolute error was 0.08. We further enhanced this alignment through our KC calibration step (see Sec 4.3), which boosts the AUC to 81.65 and reduces the KC-level MAE 0.028. These results validate that our framework not only maintains high predictive accuracy but also introduces a scalable and interpretable mechanism to estimate student mastery over individual KCs.
28

---

No Representation Learning
With Representation Learning
Other Questions
KC 1
KC
KC
KC
Original Question
Figure 8: Effect of representation learning on question embeddings. Each color indicates different
KC cluster. and the black star marks the original query question. Our representation learning module yields tightly grouped clusters. In comparison, the clusters on the left (w/o rep. learning) are scattered.
29

---

E Detailed Description of Evaluation Tasks
In this section, we provide detailed formulations and motivations for the four evaluation tasks introduced in Sec. Each task defines a different set of target knowledge concepts (KCs), reflecting distinct pedagogical goals. In all tasks, the RL agent interacts with calibrated KT environment for 10 steps. While this interaction begins at the 100th exercise in the original student history, we denote this point as step (0) from the RL perspective.
Let Foo denote the predicted knowledge state of a student for concept c at the beginning the RL episode, and V(1)) the state after 10 recommended exercises. The evaluation score is computed as the net improvement:
1
Score
(i0)
(24)
[7|
CET
where T is the set of target KCs defined for each task.
Task 1: Global knowledge improvement. Here, the agent is rewarded for general learning progress, with the entire concept set considered:
==
where C is the set of all annotated knowledge concepts. This task evaluates broad improvement across the curriculum and is aligned with standard KT metrics. Unlike prior work, our framework allows direct inference the concept level, avoiding costly evaluations over all questions.
Task 2: Knowledge improvement in practiced KC. This task models student actively focusing on specific concept. For each student, we extract the KCs from their last 10 exercises:
100
Crecent
U C., 6=91
where C, is the set of KCs associated with exercise Crecent selected as the target: T' {c*}.
The most frequently occurring concept
Task 3: Knowledge improvement in upeoming KC. This task simulates progression in curriculum learning. We first build KC-to-KC transition matrix M/c + e'I by co-occurrence statisties across student histories:
100
110
Chefore
UC, Ci i=91
Caner = U C,
x=101
and for each pair (c, c') E Chetore x Catters we increment the matrix:
Mic + d'l +=1
After normalization, obtain transition probabilities:
M[c
P(d ()) = Mic Mic
To sample the target KC for each student, we marginalize over the set of previously encountered KCs:
P(c) = P(d P(c),
I(d
CECretic
where P(c) is set to uniform over Chefore The sampled target et P(c') defines the evaluation set {c*).
Task 4: Knowledge improvement weakest KC. This task encourages the agent to target the student's weakest areas. At each step I, the current weakest concept is:
= arg min Dig- aEC
Let T Go} be the union of selected weakest concepts over the episode. The final score is computed as:
30

---

10 Σ
1 10
Score
Pio;
(25)
1=1
This task models adaptive remediation by dynamically selecting and addressing knowledge gaps.
Scoring details. To evaluate performance, we measure the net change in the student's knowledge state after 10 exercise recommendations, averaged over 2048 students from the test set. Given that the dataset has an average correctness rate of 78 %, our calibrated KT model yields a mean knowledge state of 0.78 across students and KCs, Consequently, the maximum attainable improvement (on average) is +0.22, while the worst possible decline is -0.78.
Rather than reporting raw gains, adopt a normalized metric that expresses the observed improve- ment as percentage of the maximum achievable gain. For instance, if the upper bound is 0.22 and model improves a student's knowledge by 0.15, we report 68. 2 % of the maximum possible improvement. This normalization improves comparability across models and tasks. Note that while Tasks 1-3 share a common upper bound of approximately 0.22, Task (due dynamically targeting the weakest KC) permits larger gains, with upper bounds reaching up to 0.64 depending on the model.
31

---

Prompts for KC Annotation via LLMs
We show below the prompt templates used in Module for generating solution steps, KC annotations and solution step-KC mappings. Our approach follows system-user prompt format, which reflects how large language models are typically queried in practice.
F1
Solution Step Generation
System Prompt
Your task is to generate the clear and concise step by step solutions of the
provided Math problem. Please consider the below instructions in your
generation:
You will be provided with the final answer. When generating the step by step solution, you leverage this information piece.
It important that your generated step by step solution should be understandable
as stand-alone, meaning that the student should not need to additionally check final answer or explanation provided.
Your solution steps will be later used to identify the knowledge concepts associated at each step. Therefore, please don't write final conclusion
sentence as the last step, because won't contribute to any knowledge concept.
Don't generate text other than the step by step solution described earlier. Don't divide any equation multiple lines, i.e. an equation should start and finish at the same line.
Make your step-by-step solution concise (e.g. not much verbose, and not a longer list than necessary) as described earlier.
You must provide your step by step solution in structured and concise manner in list of steps, i.e. [<step1>, .... <stepli>] Don't
Solution_Stedt field
enumerate the steps.
You have limited tokens, try to make each <atep> as concise as possible. IMPORTANT: If your final answer does not match the provided final answer, provide one last solution atep with <error>. This will help identifying potential in your solution generation.
IMPORTANT: Don't use any invalid character, i.e., it should be safe to 'ast. literal_eval' on your response message.
Please follow the example output in json format below as template when structuring your output:
("Solution_Stepa", [<stepi>, <step2>,
<stepN>>}>
User Prompt Template
Question: <QUESTION TEXT>
Final Ansver: <FINAL ANSWER>
F.2 KC Annotation
System Prompt
You will be provided with Math question and its step step solution. Your task to provide concise and comprehensive list of knowledge concepts (KCs) Math curriculum required to correctly answer the questions.
Your task has mainly 2 phases, whose details will be provided below. Each phase has its field in your json output format.
Reasoning:
1. Identify all the relevant KCs required to solve this problem.
2. Justify why each KC is relevant, considering the question and solution steps.
32

---

3. You have limited space, please 100 words maximum.
List KCs: Provide a list unique KCs with the help of your reasoning above,
.e. [<KC 1>, <KC M>]. Don't enumerate the KCs.
1. Provide multiple knowledge concepts only when it is actually needed.
2. Some questions require figure, which you won't be provided. As the step-by- step solution is already provided, knowledge concept(s) might be needed.
your judgement to infer which
3. For small set of solutions, their last st) might be missing due to limited token size. Use your judgement based your input and your ability to infer how the solution would conclude.
4. Remember that knowledge concepts should be appropriate for Math curriculum.
If annotated step-by-step solution involves advanced techniques, use your judgment for more simplified alternatives.
IMPORTANT NOTE: For your task, try Mathematics for the Knowledge Concept (XC) annotations. The reason is, we aim to get consistent texts for the same KCs across different questions.
the Common Core State Standards for
Please follow the example output in json format below template when structuring your output, IMPORTANT: Don't use any invalid character, 1.c., should be safe to call 'ast. lis..literal_eval. on your response message.
("Reasoning":
<Your reasoning to identify relevant KCs.>, "List_KCs"; [<KC 1>, <KC 2>, .... <KC M>]}
User Prompt Template
Question: <QUESTION TEXT>
Solution steps: <SOLUTION STEPS>
F.3
Solution SC Mapping
System Prompt
You will be provided with Math question, its step by step solution and its associated knowledge concepts (KCs), Your task is to map each solution step with its associated KC(s).
Mapping between solution steps and KCs: All solution steps and all knowledge concepts
mapped, while many-to-many mapping indeed possible.
IMPORTANT:
Each solution step is already numbered from 1 to and each knowledge concept is numbered from to X, where M is the number of KCs you found earlier. For consistency, output should enumerate solution step- ktop-knowledge concept pairs as numbers.
the ordering your output of list of KCs. Your
1. Each solution step has to be paired.
2. Each knowledge concept has to be paired.
Map solution step with knosledge concept only if they relevant..
4. Your pairs cannot contain artificial solution steps. For instance, if there are solution steps, the pair "5-2" is illegal.
5. Your pairs cannot contain artificial knowledge concepts. For instance, if there are knowledge concepts, the pair "3-5" is illegal.
IMPORTANT: For this field, you will output solution step- knowlegee concept pairs in comma-separater manner and in single line. For example, if there are 4 solution steps and KCs, potential output could be: "1-1, 1-3, 1-5, 2-4, 3-2, 3-5, 4-2, 4-3, 4-5"
The provided example is illustrative only. Your output should reflect the actual mapping derived from the given question and solution.
33

---

Please follow the example output in json format below as template when structuring your output. IMPORTANT: Don't any invalid character, i.e., it should be safe to call last.literal_eval. on your response message.
("mapping_step :c": <solution step knouledge concept pairs in comma separated manner and in single line.>)
User Prompt Template
Question: <QUESTION TEXT>
Solution steps: <SOLUTION STEPS>
KCs: <LIST OF KNOWLEDGE CONCEPTS>
34

---

G Example KC Annotation
We provide an example question from the XES3G5M dataset to compare the quality of our automated KC annotation (Module 1) against the KC annotation from the original dataset.
There is bridge. To cross the bridge, you need go uphill first, then walk level road, and finally go downhill. The total distance is 18 kilometers, and the uphill, flat, and downhill distances are equal. When someone walks across a bridge, the speed of going uphill, walking level road, and downhill is 2km/h, 3km/h, h, and 6km/h /h respectively. What is his average speed across the bridge in kilometers per hour?
Question:
Solution steps: Below is the solution steps annotated by our Module
The total distance of the bridge is 18 kilometers, and the distances for uphill, flat, and downhill are equal. Therefore, each section 18 + 3 = 6 kilometers long.
The speed going uphill is km/h, so the time taken to cover the uphill section 6 + hours.
The speed on the flat section 3 km/h, so the time taken to cover the flat section is 6 hours.
The speed going downhill is km/h, so the time taken to cover the downhill section is + 6 1 hour.
The total time taken to cross the bridge is the sum of the times for each section: 3 + 2 + 6 hours.
The average speed across the bridge is the total distance divided by the total time: 1 km/h.
KC annotation of the original dataset: Single person speed change problem.
KC annotation of our Module 1: a) Understanding of speed, distance, and time relationship, b) Average speed calculation, Division, d) Addition.
Takeaway: Our ExRec provides more accurate, comprehensive, and modular KC annotations for Math questions compared to original annotations. These annotations facilitate semantically more meaningful embeddings for questions, which are leveraged for both downstream KT models and RL policies.
35

---

H
Artificial Question Generation for Extending the Corpus
We show below the system and user prompts used generate diverse artificial Math questions to extend the exercise corpus. The LLM is conditioned on original question, its step-by-step solution, and list of annotated knowledge concepts (KCs). It is instructed to generate variations that maintain conceptual grounding while introducing structural and reasoning diversity.
System Prompt
You will be provided with Nath question, its step-by-step solution, and its annotated knowledge concepts (KCs). Your task to generate conceptually diverse Math questions that still use the core KCs but introduce variations in mathematical reasoning, question framing, or constraints. You will do this 3 times.
You can use your creativity this task, because the variations are highly valued. Your generations just need to be Mathematically sound.
Please follow the detailed instructions below:
For the first generation, you can keep the same set of KCs as in your original input. For the other generations, you will add one more relevant KC that you find appropriate. Keep in mind that, those added KCs won't accumulate over your add
last generation (i.e. you will explained the example output format, you will first decide those KCs at the beginning of each generation, and then you will condition your question on it.
KC only over the original KCs). As
You will generate the queation based the gemerated KCs you decided. Note that each question should modify the problem structure beyond simple rewording (e.g., introducing constraints, varying input conditions, changing the logical setup).
Then you will generate the solution steps based on the KCs question you generated. You should generate a solution that genuinely reflects the conceptual shift rather than being a template copy. Just like the question generation, your solution steps should be modified beyond simple rewording and they ahould include meaningful variations.
You will repeat this procedure times.
IMPORTANT:
Ensure that your response is always complete and well-formed JSON object. you are unable to generate proper response, provide meaningful default example rather than an empty dictionary.
IMPORTANT: Diversity between your generations is highly appreciated.
Output format: will provide a dictionary json format. you can see from the format, you will first generate the KCs, then the question content, and proper
finally the solution steps. KCs and solution steps must be provided lists in their respective fields. In the below example output format, M and N refer number of KCs solution step, which course may differ across different generations. Note that generation ids are 0- O-indexed, 1.e., from 0 to 2. Don't enumerate the KCs or steps within the list.
Example format below (o: ("list_XCs": [<KC 1>, list_sol_steps" [<step 1>, <step NO>]),
<KC MO>], "question": qqustiio_text>,
1: ("list_KCs"; [<KC 1>, ... <XC Mi>], "question": <question_text>, list_sol_stepe": [<step 1>, .... <step Ni>]}, ""list_KCa" [<xc 1>, sist_sol_steps": [<step 1>, <step N2>]}}
<XC M2>], "question": equestion_text>,
36

---

User Prompt Template
Question: <QUESTION TEXT>
Solution steps: <SOLUTION STEPS>
Knowledge concepta: <LIST OF KCs>
H.1 Example Artificial Questions
We present example original question from the dataset along with its solution steps and knowledge concepts (KCs). Then, we show two artificial questions generated based on it using the prompt deseribed in this section. New conceptual elements solution changes introduced by the model are highlighted in purple.
Original Question
Eddie went to the stationery store to buy stationery for the teacher. The unit prices of the stationery are follows: a mechanical pencil yuan, pencil box 26 yuan, a pen yuan, and a school bag is 128 yuan. Eddie wants to buy 32 pencils, 24 pencil cases, 65 pens, and school bags. How much does Eddie want to spend in total?
Question:
Solution Steps:
1. Calculate the cost of 32 mechanical pencils: 4 yuan each 32 128 yuan.
Calculate the cost 24 pencil boxes: 26 yuan each x 624 yuan.
Calculate the cost 65 pens: 45 yuan each x 65 2925 yuan.
Calculate the cost school bags: 128 yuan each x 9 1152 yuan.
Add all the costs together: 128 + 624 + 2925 + 1152 = 4829 yuan.
KCs: a) Understanding multiplication, b) Solving multiplication and division problems in real-world contexts, c) Adding and subtracting multi-digi whole numbers
Artificial Generation
Question: Eddie is buying supplies for school event. The prices are as follows: a box of markers costs 4 yuan, a pack of paper costs 12 yuan, set of paints costs 30 yuan, and roll of tape costs 8 yuan. Eddie wants to buy 50 boxes of markers, 40 packs of paper, 20 sets of paints, and 15 rolls of tape. How much will Eddie spend in dollars, if dollar equals 7 yuan?
Solution Steps:
1. Calculate the cost of 50 boxes of markers: 4 yuan each 50 = 200 yuan.
Calculate the cost of 40 packs of paper: 12 yuan each 40 480 yuan.
Calculate the cost of 20 sets of paints: 30 yuan each x 20 600 yuan.
Calculate cost 15 rolls tape: yuan each x 15 120 yuan.
Add all the costs together: 200 + 480 + 600 + 120 = 1400 yuan.
6. Convert the total cost to dollars: 1400 yuan 7 = 200 dollars.
KCs: a) Understanding multiplication, b) Solving multiplication and division problems in real- world contexts, e) Adding and subtracting multi-digit whole numbers, Understanding unit conversion
Artificial Generation
Question:
Alice is purchasing items for a charity event. The prices are as follows: notebook costs 6 yuan, folder costs 10 yuan, calculator costs 60 yuan, and backpack costs 150 yuan. Eddie wants to buy 20 notebooks, 30 folders, 10 calculators, and backpacks. Alice receives a 10% discount on the total purchase, how much will he spend?
Solution Steps:
1. Calculate the cost of 20 notebooks: yuan each x 20 = 120 yuan.
2. Calculate the cost of 30 folders: 10 yuan each x 30 300 yuan.
3. Calculate the cost of 10 calculators: 60 yuan each 10 600 yuan.
37

---

4. Calculate the cost of backpacks: 150 yuan each 5 = 750 yuan.
5. Add all the costs together: 120 300 600 + 750 1770 yuan.
Calculate the discount: 10% of 1770 yuan 177 yuan.
Subtract the discount from the total cost: 1770 177 1593 yuan.
KCs: Understanding multiplication, b) Solving multiplication and division problems in real-world contexts, c) Adding and subtracting multi-digit whole numbers, d) Understanding percentages
38

---

I Visualization of Conceptual Growth (Extended)
Figure 2 demonstrates how the knowledge state of student evolves for wide range of policies and also nL. baselines. Overall, the results are consistent with our earlier findings.
The non-RL baselines not targeting the weaknesses of the students, and in fact, they decline in their knowledge state. On the other hand, most RL policies provide meaningful knowlede wedge gains across multiple concepts. Further, we observe better improvements for the RL policies improved by our model based value estimation (w/ MVE). When tested on an extended corpus, the default version of the continuous action models perform poorly. Yet, MVE approach provides large gains to these models on the extended corpus. This demonstrates that these value-based continuous action models can adapt to new set of questions with the ability of planning ahead via our MVE.
Randive Boodd
2010
312
001
102
200
120
Janh
100
120
140-
LP%
LANS
LAG
1A0
Teve Whpp
Tino Wrr
Starnta
1941
1156
441
1176
152
101
134
1154
11.0
1104
1334
1154
1230
1334
444
L4t
2011
140
Unh
U00
135
2099
Shop
Time
S4C w/nE
TED on MVC
C0CC WVE Sohiniddd Copur
SAC (Emarcdddd Corpal
1314
300
101
101
1912
11.0
101
133
101
1334
XX3
Sxp
Tie Sxp
334
Figure Knowledge state evolution for single student across various policies and non-RL baselines
(Task 4). At each step, the weakest KC is targeted, and its knowledge trajectory is shown. White dashed boxes mark changes after each recommendation. A KC may appear more than once if remains the weakest. KC IDs are for visualization only; our framework operates on semantic KC embeddings, not IDs
39

---

J Generalization to an Additional Dataset
To evaluate the generalizability of our framework, we conduct further experiments on the Eedi dataset [25]. This large-scale dataset contains 2,324,16 learning interactions from 47,560 students. The corpus consists of 4,019 unique questions which are annotated with 1,215 distinct knowledge concepts (KCs).
Table 2: Global Knowledge Improvement the Eedi Dataset (Reported: % Max. Improvement)
Random
Historical Data
DDPG
SAC
TD3
PPO
TRPO
Discrete SAC
C51
Rainbow
DQN
Policy
w/ MVE
MVE
MIVE
-2.31
-0.21
16.65
18.87
8.97
20.08
9.62
16.85
1.78
18.81
20.83
37,00
30.51
Specifically, evaluate our framework the Bedi dataset using the Global Knowledge Improvement task (Task 1) from the main paper, where the objective is to maximize the average knowledge improvement across all KCs. The results, presented in Table 2 confirm the key findings from our primary experiments on the XES3G5M dataset. Notably, our model-based value estimation (MVE) again provides a significant performance boost to continuous value-based RL methods, and discrete-action algorithms also perform strongly, reinforcing the robustness of our approach.
It important to note that the reported percentage improvements the Eedi dataset appear lower than those on XES3G5M. This is because the maximum possible absolute improvement is substantially larger for Eedi (0.56 vs. 0.21). As result, students must achieve larger absolute knowledge gains to reach full mastery. In absolute terms, the improvements on Eedi are comparable to those on XES3G5M, even though the percentages appear smaller.
40

---

K Future Direction: Incorporating Misconception Analysis
An important future direction for enhancing personalized learning the analysis of students' incorrect responses to a given question (e.g., choosing distractor B vs. C). Different incorrect choices often represent distinct cognitive models and specific knowledge confusions. This type of error-specific analysis could reveal not just what students got wrong, but why they made specific errors. This section outlines current data limitations that hinder this approach and provides a detailed recipe for how the ExRec framework could be extended to incorporate such a misconception analysis.
Data Limitation. A practical challenge for this approach is that most existing knowledge tracing datasets, including those used in our work, only provide binary correctness label for each student interaction. The specific incorrect option chosen by the student is not recorded, making it impossible to distinguish between different error patterns or map them to specific misconceptions.
Should future datasets become available that include students' specific answer choices, the ExRec framework could be naturally extended to model misconceptions. We outline detailed recipe below:
Module 1 (LLM Annotation) could be augmented with the following steps to produce fine- grained labels that inform the representation learning in Module 2. This would enable the model to capture not only what misconception occurred, but also where it occurred in the reasoning process:
(a) Misconception annotation: Given the question, its solution steps, KCs, and specific incorrect option, the I.MM would be prompted to annotate the underlying misconception reflected by that choice.
(b) Incorrect solution generation: Using the correct solution steps, the identified miscon- ception, and the incorrect option, the LLM would then generate an incorrect solution that deviates from the correct one at the point where the misconception is applied.
Incorrect step-misconception mapping: Finally, the LLM would pinpoint exactly which
step(s) in the generated incorrect solution correspond to the annotated misconception.
2. Module (Contrastive Learning) could then incorporate these new annotations to learn richer representations:
(a) New token: A new special token, [MC], would be introduced to mark misconception text, alongside the existing tokens ([Q), [S], [KC]), allowing the encoder to learn distinct representations.
(b) Extended loss function: In addition to the original contrastive loss T., for correct solution steps, new loss term Line would be introduced to align the embeddings of incorrect solution steps with their corresponding misconception embeddings.
3. Module (Knowledge Tracing) would be adapted process these new, richer inputs:
(a) Input embedding: Instead of combining the question embedding with simple binary correctness indicator, the input would formed using either the correct solution step embeddings (if the student answered correctly) or the incorrect solution step embeddings (if the student's answer is linked to known misconception).
(b) Dual-task prediction: The model would have two outputs. The primary output for per- formance prediction would remain. A second output would predict the misconception embedding, with an auxiliary loss based on the cosine similarity to the ground-truth mis- conception (this loss would be masked for correct answers). This avoids a rigid categorical formulation and allows for the seamless integration of new misconceptions.
(c) KC calibration: The calibration process for knowledge states would remain unchanged.
4. Module 4 (Reinforcement Learning) would require no structural changes. The KT model, now enhanced with misconception modeling, would serve more sophisticated RL environment, inherently providing the necessary state transitions and rewards to the RL agent.
We believe this extension represents a powerful direction for future research. The proposed recipe demonstrates both the feasibility and the natural fit of misconception analysis within the ExRec framework. While current dataset limitations preclude its immediate implementation, we hope this detailed outline inspires future work in data collection and modeling to better understand the nuances of student learning.
41


Reviews:
[{"review_id": "71dtmqnZS3", "reviewer": "Reviewer_XF4M", "summary": "This paper introduces ExRec, a framework for personalized exercise recommendation combining semantically grounded KT and RL. Key innovations include automated KC annotation via llm, semantic question embeddings, a calibrated KT model, and a model-based value estimation method. The framework is evaluated across four math learning tasks and shows generalization to unseen questions using GPT-4o-generated content.", "questions": "1. Is the claim that existing methods “typically support only a single RL algorithm” fully accurate, and could the authors clarify why those methods cannot be easily extended to support multiple RL algorithms?\n2. How robust and generalizable are the LLM-generated KC annotations and solution steps across different mathematical domains or languages, especially in the absence of additional fine-tuning?\n3. To what extent is the RL policy itself interpretable, and can the authors explain how educators or students might understand why specific exercises are recommended?\n4. The paper uses a fixed horizon of 10 steps for RL agent interaction. How does the performance of the RL policies, especially those with MVE, scale with a significantly longer recommendation horizon, and what challenges might arise in such scenarios?\n5. Beyond the XES3G5M dataset, which focuses on math, how generalizable is the ExRec framework to other subjects or domains where knowledge concepts might be less structured or more ambiguous, and where LLM annotation might face greater challenges?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The pipeline is a well-integrated and novel approach. Using LLMs for KC mapping addresses the costly manual annotation problem.\n2. ExRec generalizes effectively to unseen questions, aided by strong semantic modeling and LLM-augmented corpora.\n3. Experiments demonstrate the method's effectiveness.\n\nWeakness:\n1. The motivation could be refined. The manuscript claimed that current related methods \"typically support only a single RL algorithm\". IMHO, these methods could conveniently extend their methods with other RL algorithms.\n2. The paper claims that llm improve KC annotation quality. However, it relies heavily on llm for key tasks like KC annotation and solution step generation. This raises concerns about whether the results are robust and generalizable across different math domains or languages, especially without additional fine-tuning. The original annotations are described as \"noisy\" in this manuscript, but LLM outputs can also be inconsistent, particularly with complex or ambiguous problems.\n3. For educational applications, understanding why a particular exercise is recommended can be crucial for educators and students. Although the paper mentions interpretable student learning trajectories, it does not delve into the interpretability of the RL policy's recommendations themselves.", "was_revised": true, "final_justification": "The authors clarify:\n1. Clarified motivation regarding support for multiple RL algorithms and how ExRec achieves this in a scalable, modular way.\n2. Addressed concerns about LLM reliance by explaining the design for robustness, presenting EEDI results, and showing adaptability to new domains.\n3. Detailed how framework enables interpretability both at the RL and KT levels.\n4. Reported new experiments with longer RL horizons, highlighting trends and observed challenges.\n5. Discussed about the generalizability beyond Math, including how ExRec handles less-structured KCs.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback! We truly appreciate your recognition of the novelty and integration quality of our framework, and we’re grateful that you took the time to engage deeply with our work. \n\nYour comments helped us better articulate the strengths and scope of our contributions. As your review rightly notes, **educational personalization is a complex challenge**, and we believe that presenting ExRec at a venue like NeurIPS is vital to **spark meaningful discussion and catalyze progress at the intersection of large language models, reinforcement learning, and education**. \n\nWe hope that our responses have clarified the key distinctions and practical innovations introduced in our framework, and we would be truly grateful for your support in helping bring this work to the broader research community.\n\n**W1: The motivation could be refined. The manuscript claimed that current related methods \"typically support only a single RL algorithm\". IMHO, these methods could conveniently extend their methods with other RL algorithms.**\n\nThank you for raising this point. Our claim is not that existing methods *theoretically* cannot adopt multiple RL algorithms, but rather that they are **not implemented in a modular or scalable way** to make this practical.\n\nMany prior works rely on custom, tightly coupled environments and pipelines, making integration of alternative RL methods non-trivial and error-prone. Most did not release code, and their scalability remains unclear, especially on large-scale datasets.\n\nIn contrast, **ExRec is fully built on the standardized Tianshou library**, supporting plug-and-play use of various RL algorithms. We also addressed critical bottlenecks such as memory-efficient state representations and concept-based reward computation.\n\nWe will revise the manuscript to clarify that **our contribution lies in enabling scalable, efficient, and modular integration of diverse RL policies**, which has not been realized in prior work.\n\n**W2: Heavy reliance on LLMs for KC annotation and solution step generation raises concerns about robustness and generalizability across domains/languages without fine-tuning.**\n\nThank you for this point. Our motivation for using LLMs goes beyond quality—it is also about *scalability and efficiency*. Manual annotation is costly, while our approach scales to large systems and new domains.\n\nOn quality, prior work [1,2] shows *LLM-generated annotations outperform expert tags* in coherence and generalizability. Original dataset labels were often noisy and hard to reuse (e.g., *“Application question module–Chicken rabbit cage problem–Hypothesis method–Prototype question”*), unlike our concise labels (e.g., *“Linear Equations”*, *“Area of Triangle”*).\n\nWe acknowledge LLM outputs can be inconsistent. Our *contrastive learning module* (Module 2) mitigates this by aligning question/step embeddings with concepts, reinforcing consistency.\n\nTo test generalizability, we applied the pipeline—unchanged—to the *EEDI dataset* (2.3M interactions, verified question content) and observed that *it transfers well across math domains*.\n\nThese results show our framework is robust and adaptable, while still allowing for domain-specific validation where needed.\n\n**W3:For educational applications, understanding why a particular exercise is recommended can be crucial for educators and students. Although the paper mentions interpretable student learning trajectories, it does not delve into the interpretability of the RL policy's recommendations themselves.**\n\nThank you for raising this point. While interpretability was not the paper’s primary focus, we fully agree it is important for real-world use, and we designed ExRec with this in mind.\n\nOur RL policy operates in the **same embedding space as the knowledge concepts (KCs)**, so one can analyze the action embedding’s proximity to KC embeddings. For example, if a student struggles with “multiplication” and “factoring out a common factor,” the RL policy will produce an action embedding close to these KCs, therefore making it possible to **infer which concepts are being targeted** and select a question accordingly.\n\nInterpretability also extends **beyond RL**: our calibrated KT module supports **real-time, concept-level inspection**, which allows educators to query student knowledge on any KC at any time step. We will make these aspects clearer in the final version.\n\n**Action:** While we are not allowed to add figures here, we will add an example with visualizations to the revised paper. \n\n\n\n**Q1: Is the claim that existing methods “typically support only a single RL algorithm” fully accurate, and could the authors clarify why those methods cannot be easily extended to support multiple RL algorithms?**\n\nThank you very much for your question! We would like to kindly refer to **our response to your W1** for our detailed answer! \n\n**Q2: How robust and generalizable are the LLM-generated KC annotations and solution steps across different mathematical domains or languages, especially in the absence of additional fine-tuning?**\n\nThank you for the question. We clarify that our framework does **not require any LLM fine-tuning** to enable easy deployment across new datasets. This is supported by prior research: [1] demonstrated the effectiveness of LLM-based KC annotation in domains like Chemistry and Physics. Further our own experiments on the **EEDI math dataset** confirm generalizability within Math.\n\nWe still acknowledge that  _LLM outputs should be evaluated before large-scale annotation_, especially in new subjects or languages. \n\nOur modular pipeline allows flexible integration of custom prompts, LLMs, or expert-curated annotations. To apply ExRec in other domains, we offer the following guidance:\n\n- If questions lack multi-step solutions, skip step generation and directly use KC annotation.\n\n- Use the resulting annotations for representation learning with question-level loss.\n\n- The rest of the pipeline (KT + RL) remains unchanged.\n\nFinally, **public KT datasets with full question content are rare**, mostly limited to Math. We hope our work encourages broader dataset releases to support cross-domain research.\n\n**Q3: To what extent is the RL policy itself interpretable, and can the authors explain how educators or students might understand why specific exercises are recommended?**\n\nThank you very much for raising this important concern. We would like to kindly refer to **our response to your W3** above. \n\n**Q4: The paper uses a fixed horizon of 10 steps for RL agent interaction. How does the performance of the RL policies, especially those with MVE, scale with a significantly longer recommendation horizon, and what challenges might arise in such scenarios?**\n\nThank you very much for the thoughtful question! We now performed **additional experiments with varying horizons**, particularly for 20, 50 and 80 steps. \n\n**Task 4: Knowledge Improvement in Weakest KC.** (Reported: % Max. Improvement as in paper.)\n\n**Original Corpus**\n\n\n|  |Random Policy|Historical Data|DDPG|DDPG w/ MVE|\n|--|--|--|--|--|\n|20-step| -0.20 | 1.27 | 35.19 | **50.87**|\n|50-step| -2.21 | 1.56 | 31.52 | **55.99** |\n|80-step| -2.66 | 1.58 | 30.19 | **58.31** |\n\n\n**Extended Corpus**\n\n\n|  |Random Policy|DDPG|DDPG w/ MVE|\n|--|--|--|--|\n|20-step| -1.29 | 36.09 | **50.76**|\n|50-step| -2.67 | 32.91 | **54.13**|\n|80-step| -3.70 | 32.45 | **55.21** |\n\nWe observe that our framework continues to provide notable gains at longer horizons. In fact, our MVE framework benefits from longer horizons, albeit improvements being marginal.\n\nOne challenge we noticed is that, even with MVE, the model may need to revisit the same weakest KC approximately 30–40 steps after it was first addressed. We hypothesize that the KT model tends to lower the knowledge state of certain KCs over time, particularly when no relevant exercises are encountered for an extended period. We believe that future KT models could explicitly account for this forgetting behavior, better distinguishing when a knowledge state should (or should not) decrease if a KC is left unpracticed for a long time.\n\nWe will incorporate these new findings in the paper. Thank you very much for helping us improve our submission! \n\n**Q5: How generalizable is ExRec to other domains where KCs are less structured and LLM annotation may be harder?**\n\nThank you for this question. Applying ExRec to non-math domains can pose challenges, especially with less-structured KCs, but we believe our framework is well-suited.\n\nFirst, **datasets with full question text are very limited**, restricting cross-domain experimentation. We hope our work encourages broader dataset releases.\n\nImportantly, ExRec **does not require hierarchical KC structures or expert ontologies**. It infers semantically meaningful KCs directly from question text, lowering barriers in less-structured domains.\n\nWhile math is considered one of the most demanding domains for LLMs, we believe areas like **reading comprehension or social sciences** may be easier to model semantically. In those cases, LLM-based annotation may face fewer obstacles and still produce usable KC representations for downstream learning and recommendation.\n\nWe will reflect this discussion in the revised limitations section.\n\n**References**\n\n[1] Moore, Steven et al. “Automated generation and tagging of knowledge components from multiple-choice questions”” Learning@ Scale, 2024.\n\n[2] Ozyurt, Yilmazcan et al. “Automated knowledge concept annotation and question representation learning for knowledge tracing”. arXiv, 2024."}, {"review_id": "yojiL7WtOw", "reviewer": "Reviewer_WyTm", "summary": "This paper proposes the ExRec framework for personalized exercise recommendation based on semantic knowledge tracing. The framework consists of four modules: (1) LLM automated KC annotation: using GPT-4o to automatically generate problem-solving steps and annotate knowledge concepts. (2) Contrastive learning representation: learning semantic embedding representations of questions and KC. (3) KC-calibrated knowledge tracking: training the KT model and calibrating the knowledge state prediction at the KC level. (4) Reinforcement learning recommendation: using multiple RL algorithms to optimize the exercise recommendation strategy and proposing the model-based value estimation (MVE) method.", "questions": "Questions:\n- Line 27, add the literature on personalized practice recommendation through KT, such as [1] or more.\n- Footnote [1], there are some new KT reviews that have conducted in-depth analysis from different perspectives, and it is recommended to add references.\n-  Why do you think the KT model can simulate the RL environment? What is its computational speed and resource requirements?\n- Line 132, which LLM Encoder is used?\n-  Contrastive learning optimizes (1) questions and concepts (2) steps and concepts. What is the consideration for doing so? Why not questions and steps?\n-  Does the representation of the problem (Formula (10)) come from the average representation of the problem and steps?\n-  As far as I know, there is a certain percentage of picture questions in XES3G5M dataset. How do you handle this type of input?\n\nOverall, this is a novel perspective paper. However, while the paper evaluates the downstream effects of recommendation, it does not directly assess the quality of the recommended exercises. This makes it difficult to judge whether the recommendations are truly beneficial or just favorable under the simulated KT environment. I would consider raising my score if this concern is addressed.", "limitations": "Yes. The paper includes a brief but clear discussion of its limitations. Specifically, the authors acknowledge that the effectiveness of the proposed recommendation policies depends heavily on the accuracy of the underlying KT model used to simulate student behavior. They suggest that future work could explore more accurate student modeling to improve the realism of the RL environment. In addition, they point out that their current framework does not explicitly model question difficulty, which could be incorporated into the representation learning module in future work. A more discussion of how KT model inaccuracies or annotation noise may affect policy robustness would further strengthen the limitations section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper treats the KT model as an RL environment and proposes a novel model-based value estimation (MVE) mechanism. This perspective is relatively new in the field and opens up new possibilities for leveraging student modeling in sequential decision-making.\n\n- The design of the ExRec model is reasonable. The framework is composed of clearly defined and interpretable modules: LLM-based KC annotation, contrastive representation learning, KC-calibrated KT, and an RL-based recommendation policy. \n\nWeaknesses:\n- The title of the paper is \"Personalized Exercise Recommendation\", but the experimental part is more like evaluating the effect of a knowledge tracing + reinforcement learning framework in improving students' knowledge status, without evaluating the relevance and accuracy of exercise recommendation.\n- The existing KT dataset contains annotated knowledge components (Q-matrix), and the motivation for introducing more KCs through steps and LLMs is unclear.\n- Since the method (Module 1) requires LLMs pre-processing, it is difficult to judge whether the model is an end-to-end structure.\n- The design choices in the contrastive learning module. It is unclear why the model does not consider question–step alignment, which seems natural and potentially beneficial given the step-by-step structure generated by the LLM.\n- The paper lacks external validation of its exercise recommendation outcomes. All evaluations are based on simulations from the KT model, with no comparison to real student behavior, expert recommendation, or human judgment.", "was_revised": true, "final_justification": "The authors provide a comprehensive technical roadmap for extending ExRec to handle step-level misinterpretation modeling in their rebuttal, effectively addressing my concerns about the framework's limitations. I value its strong foundational contributions and potential for scalability over its current data limitations, which represent a natural progression rather than a flaw. I hope the authors can take your feedback into consideration to further improve the camera-ready version.", "rebuttal": "Thank you for your thoughtful and constructive review! We appreciate your recognition that our work brings a new perspective by combining student modeling with sequential decision-making, a direction we believe has strong potential for personalized education.\n\n**Our goal with ExRec is to lay a foundation for this emerging area**, and we see NeurIPS as the ideal venue to foster further exploration and community engagement. We are grateful for the opportunity to clarify and strengthen our work based on your feedback.\n\n**W1: The title suggests personalized exercise recommendation, but the experiments mainly evaluate a KT + RL framework’s impact on knowledge improvement — without assessing the accuracy or relevance of the recommended exercises.**\n\nThank you for the comment. By personalized exercise recommendation, we mean **selecting exercises tailored to each student’s knowledge needs to improve their knowledge state**. Since experimenting on real students is infeasible (ethical and scalability issues), we simulate learning through KT. While this is bounded by KT model’s quality, our modular design allows future integration of more advanced KT models.\n\nRegarding “accuracy,” there is no single “correct” action in RL; we instead measure effectiveness via long-term knowledge improvement, as captured in our reward design.\n\nYet, to address this, we add a *clairvoyant-style* metric: % of times the chosen exercise falls in the top-10, top-20, or top-50 actions based on immediate (1-step) reward. For each of 1024 students and each step, we compute rewards for all 7652 possible questions. \n\n|Method|Top-10|Top-20|Top-50|\n|--|--|--|--|\n|Random Policy|0.18|0.43|0.92|\n|SAC|7.11|12.82|22.64|\n|SAC w/ MVE|24.94|34.41|46.33|\n\nWe believe this metric complements our main results by showing that **our trained policies (especially SAC w/ MVE) consistently select high-reward exercises** significantly more often than a random policy.\n\n**W2: The dataset has KC annotations, so why add more via steps and LLMs?**\n\nThank you for the observation. While KC annotations exist, many are noisy or overly specific—e.g., *“Single person speed change problem”* or *“Application question module–Chicken rabbit cage problem–Hypothesis method–Prototype question”*. Such labels are hard to generalize or align with learning standards.\n\nInstead of using these as-is, we generated concise, semantically grounded KCs with LLMs, aligned with Common Core standards. Recent studies [1,2] show this approach produces more interpretable and preferred annotations, yielding better semantic representations and a stronger foundation for recommendation.\n\n**W3: Since Module 1 requires LLMs pre-processing, it is difficult to judge whether the model is an end-to-end structure.**\n\nThank you for the comment. While Module 1 uses LLMs for KC annotation, it is fully automated and requires no manual input. This aligns with common definitions of “end-to-end,” as the system operates directly from raw question texts and student logs to exercise recommendations.\n\nImportantly, *Module 1 is optional*: if reliable KC labels are available, it can be skipped. We include it to enhance flexibility and reduce reliance on manual or noisy annotations, which are often missing in real-world datasets.\n\n**W4: The contrastive learning module design is unclear. Why is question–step alignment not used, given the stepwise structure from the LLM?**\n\nThank you. ExRec is the first KT-based framework to learn embeddings for both questions and KCs simulatenously, aiming for a shared semantic space. Instead of collapsing the full question (all steps) into a single embedding aligned with all KCs, we use step–KC alignment for **two reasons**:\n\n1. *Avoiding representation collapse:* Full question–KC alignment forces the encoder to compress diverse concepts into one vector, which prior work [2] and our experiments show is ineffective.\n\n2. *Finer-grained supervision:* Aligning each step only with its relevant KCs provides a clearer learning signal, improving representation quality and downstream performance.\nThis design produces more generalizable embeddings, critical for handling unseen questions at test time.\n\n**W5: The paper lacks external validation. Results are based solely on KT model simulations, with no comparison to real student behavior, expert suggestions, or human judgments.**\n\nThank you for raising this important point. We fully agree that real-world validation is the next, but non-trivial next step. Note that this is not unique to our paper, but such external validation is infeasible and therefore generally missing in related work. However, we took your suggestion to heart and compared 100 student traces against expert suggestions. We found that **ExRec’s recommendations were preferred 92% of the time** with respect to targeting the knowledge concepts students had yet to master. \n\nFor example, when a student lacked understanding of both *“Addition of decimal numbers”* and *“Alignment of decimal numbers”*, ExRec first recommended a question that practiced both:  \n`What is the result of 42.62 + 4.1 = ?`  \nAfter the student answered incorrectly, ExRec adapted and recommended a simpler question that still targeted *“Addition of decimal numbers”* but removed the alignment requirement:  \n`What is the result of 1.5 + 0.3 = ?`  \n\nAs one of the less-preferred cases, when a student struggled with *“Multiplication of decimal numbers”*, ExRec recommended a relevant question but one that also involved a unit conversion. Our experts suggested that a simpler question should have been given first. \n\nWe will include this external validation as additional evidence supporting the practical relevance of ExRec.\n\n**Q1: Line 27, add the literature on personalized practice recommendation through KT, such as [1] or more.**\n\nWe couldn’t find [1] in your review, but we are happy to incorporate it in our paper once we have it!. \n\n**Q2: Footnote [1], there are some new KT reviews that have conducted in-depth analysis from different perspectives, and it is recommended to add references.**\n\nThank you for your contribution! Upon your suggestion, we further incorporated [3] and [4], and we are happy to incorporate if you have other recommendations. \n\n**Q3: Why do you think the KT model can simulate the RL environment?** \n\nThank you for the question. We address it in two parts: (1) **why KT is well-suited for simulating an RL environment**, and (2) **why prior attempts failed to scale**.\n\n1.  Many KT works (e.g., those cited in line 26) are motivated by the idea that online learning benefits from personalization. This requires modeling how a student’s knowledge evolves in response to different questions, a natural fit for simulation and policy learning. However, most of these works stop at next-question prediction and do **not** explore how KT can actually be used for personalized recommendation.\n\n2. Only a few works (e.g., cited in line 30) attempted to use KT for student simulation and RL-based recommendation. Yet they failed to scale to realistic datasets: their state representations often exceeded memory limits, and reward computation required expensive inference over all questions at each time step. To address this, we propose an entirely new framework that is scalable by design.\n\nOur framework overcomes these limitations by: (1) enabling recommendation of unseen questions, (2) supporting various RL algorithms, and (3) integrating into a scalable, public library for broader use -> a practical contribution we believe is often underappreciated.\n\n**Q4: Line 132, which LLM Encoder is used?**\n\nAs described in Appendix B line 559, we used and finetuned BERT encoder through our contrastive learning module. \n\n**Q5: Contrastive learning optimizes (1) questions and concepts (2) steps and concepts. What is the consideration for doing so? Why not questions and steps?**\n\nThank you for your questions! Please see our response to your W4. \n\n**Q6: Does the representation of the problem (Formula (10)) come from the average representation of the problem and steps?**\n\nYou are correct. The representation used in downstream KT and RL modules is obtained by averaging the embeddings of the question text and its solution steps, as described.\n\nWe avoided alternatives like concatenation or more complex fusion methods for the following reason:\n\n- To address Limitation 3 (inefficient reward computation), we enable the KT model to perform inference not only on questions (as in prior work) but also directly on knowledge concepts (KCs), a novel feature in our framework.\n\n- After Module 2 (contrastive learning), question texts, steps, and KCs share a unified embedding space. For KT to process both exercises and KCs seamlessly, they must also have same dimensionality. Thus, in Eq. (10), we average question and step embeddings to match the size of a KC embedding.\n\nWe hope this clarifies our motivation for the design in Eq. (10).\n\n**Q7: As far as I know, there is a certain percentage of picture questions in XES3G5M dataset. How do you handle this type of input?**\n\nThank you for your question. For module 1, we leveraged gpt-4o’s visual understanding to annotate solution steps and KCs. For the LLM encoder to work on question content at module 2 (contrastive learning), we first prompted gpt-4o to write the description of the visual in the question, and appended this description to the question content as input. \n\n**References**\n\n[1] Moore, Steven et al. “Automated generation and tagging of knowledge components from multiple-choice questions”” Learning@ Scale, 2024.\n\n[2] Ozyurt, Yilmazcan et al. “Automated knowledge concept annotation and question representation learning for knowledge tracing”. arXiv, 2024.\n\n[3] Liu, Zitao et. al. “Deep Learning Based Knowledge Tracing: A Review, A Tool and Empirical Studies”, Knowledge and Data Engineering, 2025. \n\n[4] Zhou, Yiyun et al. \"Revisiting applicable and comprehensive knowledge tracing in large-scale data.\" arXiv, 2025."}, {"review_id": "oW9H20IEuP", "reviewer": "Reviewer_ABnB", "summary": "This paper presents a knowledge tracing (KT) based framework for personalized exercise recommenddation. The proposal annotates the knowledge concepts of questions and learning their semantic representations to training KT models, while optimizing various reinforcement learning methods. Additionally, a model-based value estimation approach is designed to improve the standard Q-learning-based continuous reinforcement learning methods. Experiments on one publicly available knowledge tracing datasets are conducted to validate the effectiveness of the proposal.", "questions": "1. Missing several related works:\nThe literature review lacks several important recent works on knowledge tracing based exercise recommendation methods, such as \n“Exercise recommendation method based on knowledge tracing and concept prerequisite relations”, CCF Transactions on Pervasive Computing and Interaction, 2022;\n“Exercise recommendation based on knowledge concept prediction”, Knowledge-Based Systems, 2020;\n“Pedagogical interventions in SPOCs: Learning behavior dashboards and knowledge tracing support exercise recommendation”, IEEE Transactions on Learning Technologies, 2023;\n“Enhanced personalized learning exercise question recommendation model based on knowledge tracing”, International Journal of Advances in Intelligent Informatics, 2024;\n“MLKT4Rec: Enhancing Exercise Recommendation Through Multitask Learning With Knowledge Tracing”, IEEE Transactions on Computational Social Systems, 2024;\n“Knowledge modeling via contextualized representations for LSTM-based personalized exercise recommendation”, Information Sciences, 2020;\n\nThe authors should provide a comprehensive discussion of these works and explicitly state how their proposal advances beyond existing approaches. Furthermore, empirical comparisons with these methods would better demonstrate the effectiveness of the proposal.\n\n2. Limited dataset evaluation:\nThe proposal is validated on only one knowledge tracing dataset. Several well-established benchmark datasets are available, such as NIPS34 and EdNet. The authors are encouraged to add these representative datasets to provide more comprehensive validation of their proposal.\n\n3. Lack of mutual impact analysis:\nWhile the proposal is designed for exercise recommendation, it would be valuable to investigate whether the proposal could, in turn, improve knowledge tracing performance.\n\n4. Insufficient efficiency analysis: \nThe authors are encouraged to report the inference time performance, as it directly impacts the feasibility of real-time implementation in educational systems.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. This paper is well-structured, and a decent amount of technical details are given.\n2. The idea of modelling personalized exercise recommendation with the semantically-grounded knowledge tracing is interesting. \n\nWeaknesses\n1. Missing several related works\n2. Limited dataset evaluation \n3. Lack of mutual impact analysis\n4. Insufficient efficiency analysis", "was_revised": true, "final_justification": "Through discussions with the authors, all of my concerns—including (i) Missing several related works, (ii) Limited dataset evaluation, (iii)  Lack of mutual impact analysis and (iv) Insufficient efficiency analysis—have been satisfactorily addressed.\n\nI believe this paper makes a promising contribution to the field of educational data mining, and I am therefore increasing my overall score.", "rebuttal": "Thank you for your thoughtful and constructive review! We appreciate your recognition of the novelty and potential of our work.\n\nOur goal with ExRec is to provide a **foundational, extensible framework** that bridges knowledge tracing and reinforcement learning for personalized education. Built for **flexibility, modularity, and scalability**, ExRec allows researchers to plug in their own datasets, use advanced KT models, and test alternative RL strategies in a unified, semantically grounded environment.\n\nWe believe NeurIPS is the ideal venue to **catalyze the next wave of research in personalized education**, and we have addressed all your suggestions to further strengthen and clarify our contributions.\n\n**Q1: Missing several related works**\n\nThank you for your valuable suggestions. Our original submission focused on works aiming to improve knowledge state using RL-based frameworks built on simulated student behavior, which aligns with our central goal: learning a recommendation policy that optimizes long-term knowledge improvement through semantically grounded KT.\n\nFollowing your suggestion, **we now broaden our discussion to include a wider range of exercise recommendation approaches.** While relevant, these works do **not directly optimize knowledge state**, nor do they develop a trainable RL-based policy. Most rely on rule-based heuristics, optimize simpler proxy metrics (e.g., difficulty, novelty), and generally overlook semantic representations of questions and KCs.\n\nWe elaborate on each paper below in the order provided.\n\n`Exercise recommendation method based on knowledge tracing and concept prerequisite relations:` This method recommends KCs rather than actual exercises, treating all exercises within a KC as interchangeable. It relies on manually defined prerequisite graphs and uses only KC IDs, ignoring the semantic content of both KCs and questions. The recommendation is rule-based and static. It *neither* learns from data *nor* simulates how student knowledge evolves over time.\n\n`Exercise recommendation based on knowledge concept prediction:` This work models KC mastery and recommends KCs based on predicted gains. The recommendation logic is rule-based and *not* trainable. It does *not* model question semantics, nor does it use RL or simulate student behavior. It focuses on proxy objectives like difficulty or novelty, whereas ExRec directly optimizes for actual knowledge state improvement.\n\n`Pedagogical interventions in SPOCs: Learning behavior dashboards and KT support:` This work applies KT to estimate students’ knowledge but does not perform exercise recommendation. Instead, it provides a dashboard interface for instructors to decide what to recommend. The KT model relies on *hand-crafted* features, unlike ExRec, which operates directly on question semantics and automates the full pipeline.\n\n`Enhanced personalized learning exercise question recommendation based on KT:` This approach predicts question difficulty and filters exercises accordingly, based on instructor-specified difficulty ranges. It applies KT but does *not* simulate student learning or use RL. It assumes external input of difficulty to guide recommendation and does not attempt to optimize knowledge improvement directly.\n\n`MLKT4Rec: Enhancing Exercise Recommendation Through Multitask Learning with KT:` MLKT4Rec casts recommendation as a multitask supervised problem, jointly predicting correctness and recommending exercises to maximize immediate success. Therefore, it does *not* prioritize long-term knowledge gain. It does *not* incorporate RL, simulate knowledge gain, or model semantic content or KCs. In contrast, ExRec optimizes for long-term knowledge improvement, incorporates LLM-based KC annotations, and enables generalization to unseen questions.\n\n`Knowledge modeling via contextualized representations for LSTM-based personalized exercise recommendation:` This method uses LSTM and attention to recommend exercises based on historical interactions. It focuses on predicting immediate performance without RL, semantic modeling, or KC representations. ExRec advances this line by using traced knowledge states, semantically enriched representations, and RL to guide students toward lasting knowledge gains.\n\n**->** We will incorporate these works into our extended Related Work section to clearly position ExRec in relation to these approaches and to highlight our unique contributions.\n\n**Q2: Limited dataset evaluation: The proposal is validated on only one knowledge tracing dataset. Several well-established benchmark datasets are available, such as NIPS34 and EdNet. The authors are encouraged to add these representative datasets to provide more comprehensive validation of their proposal.**\n\nThank you for the suggestion. While we agree that broader dataset evaluation is important, our framework specifically leverages the semantic content of questions, which is central to our contribution. Unfortunately, the suggested datasets are incompatible with this requirement:\n\n- **EdNet** does not provide question text — only question IDs — making it unusable for semantically grounded recommendation.\n\n- **NIPS34** includes only *image*-based questions, many with geometric content. Converting them to meaningful text requires reliable multi-modal understanding, which is beyond this paper’s scope. However, we appreciate this suggestion and will include it as a limitation and a future direction.\n\nTo address your feedback, **we extended our evaluation to the EEDI dataset**, developed by the same creators as NIPS34. EEDI provides OCR-transcribed and manually verified textual question content. Therefore, it is well-suited for our framework. It also reflects real-world conditions, with 47,560 students, 4,019 unique questions, and over 2.3M interactions.\n\nThanks to ExRec’s modular design, we seamlessly applied our full pipeline to this dataset. Please find the results below:\n\n**Task 1: Global Knowledge Improvement.** (Reported: % Max. Improvement as in paper.)\n\n|Random Policy|Historical Data|DDPG|DDPG w/ MVE|SAC|SAC w/ MVE|TD3|TD3 w/ MVE|PPO|TRPO|Discrete SAC|C51|Rainbow|DQN|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|-2.31|-0.21|16.65|18.87|8.97|20.08|9.62|16.85|1.78|2.83|18.81|20.83|37.00|30.51|\n\n*Note:* The reported improvements on the EEDI dataset appear lower than those on XES3G5M because the maximum possible absolute improvement is much larger (0.56 vs. 0.21). As a result, students must achieve larger absolute knowledge gains to reach full mastery. In absolute terms, the improvements on EEDI are comparable to those on XES3G5M, even though the percentages appear smaller.\n\nWe are grateful for the opportunity to demonstrate that **our framework generalizes easily to other large-scale math datasets**. This has been a great opportunity to demonstrate its practical applicability and extensibility.\n\n\n**Q3: Lack of mutual impact analysis: While the proposal is designed for exercise recommendation, it would be valuable to investigate whether the proposal could, in turn, improve knowledge tracing performance.**\n\nThank you very much for your valuable (and quite interesting) suggestion! As a disclaimer, please correct us if we misinterpreted your suggestion — if that’s the case, we are more than happy to investigate it more deeply! \n\nFollowing your suggestion, we treated the generated trajectories through our RL environment as the synthetic dataset (i.e., the new question-answer pairs). We then appended this synthetic dataset to the original KT dataset and performed another round of KT training. \n\nAfter the above procedure, we found that the AUROC score of KT model dropped from 81.65 to 81.32. We believe the main reason is that the original dataset was large enough to train our KT model and that the synthetic dataset propagated the errors of a KT model that caused a decrease in the performance. \n\nTo confirm the above hypothesis, we trained another KT model (from scratch) with only 10% of the existing students in the dataset and repeated the above procedure of producing synthetic data through our RL environment. This time, we observed an increase in the prediction performance from 79.41 to 79.98 AUROC. \n\nThese findings suggest that our framework can serve not only as a recommendation policy but also as a data augmentation tool to improve KT performance. This is particularly relevant in **low-data regimes.** \n\n\n**->** We will incorporate this analysis into the paper as an additional use case, especially highlighting its value when the number of available student interactions is limited.\n\n**Q4: Insufficient efficiency analysis: The authors are encouraged to report the inference time performance, as it directly impacts the feasibility of real-time implementation in educational systems.**\n\nThank you very much for this insightful suggestion! While we had already reported the training time and hardware requirements of our computationally intensive modules in Appendix B, we agree that **inference time** is an important factor for real-time deployment in educational systems.\n\nIn response, we conducted additional experiments to measure inference efficiency. Below, we report:\n\n- **KT training** takes around 2.5 hours on a single NVIDIA Quadro RTX 6000 GPU with 32 GB of memory.\n\n- **KT inference per question** takes  8.7 milliseconds on a single NVIDIA Quadro RTX 6000 GPU with 24 GM of memory.\n\n- **RL inference per exercise recommendation** takes 78.5 milliseconds on a single NVIDIA Quadro RTX 6000 GPU with 24 GM of memory.\n\nWe will add these details to our Appendix B."}, {"review_id": "xnbWcIOupD", "reviewer": "Reviewer_SGqL", "summary": "This paper presents a knowledge tracing (KT) algorithm based on reinforcement learning (RL) for personalized education.", "questions": "Do the authors have any plan how to ensure QA of the generated solutions by LLMs?", "limitations": "The authors have used only one dataset. \nThey can also explore relevant K-12 math dataset on their proposed model, such as the ASSISTments datasets.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1.\tProduces interpretable learning trajectories via reinforcement learning (RL)\n\n2.\tKT generalizes to new, unseen question (Section 6.1)\n\n3.\tExperimented on four evaluation tasks: global knowledge improvement, knowledge improvement in practiced knowledge component (KC), weakest KC, upcoming KC.\n\n4.\tCompact representation of states, that do not need to process the student’s entire learning sequence\n\nWeakness\n1.\tThe limitations specified in the introduction “they often rely on ID-based embeddings, which neglects the semantics of question”—is not correct. The first DKT paper was on question ID-based embeddings. But after that, the weakness has been addressed by the community. Most DKT algorithms rely of embeddings of KC, such as AKT [1]. \nEmbeddings of questions has been explored by researchers’ way before the LLM era, see a 2018 paper: EERNN[2], a 2020 paper: RAKT[3], a 2022 paper: SRL-KT [4].\n\n2.\tThe KC representation in Section 4.1 is a problematic, in my opinion. Prior deep learning era, KC annotation was performed by domain experts with multiple rounds of review.\nThe proposed approach annotated KC by a sampling method through LLM with no expert intervention. The step can be performed by a machine-teaching or human-in-the loop fashion.\nSimilarly, the solution step generation is done by LLM, but no quality assurance is performed.\n\nReferences\n\n1.\tGhosh, A., Heffernan, N. and Lan, A.S., 2020, August. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2330-2339).\n\n2.\t. Su, Y., Liu, Q., Liu, Q., Huang, Z., Yin, Y., Chen, E., Ding, C., Wei, S. and Hu, G., 2018, April. Exercise-enhanced sequential modeling for student performance prediction. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).\n\n3.\t Pandey, S. and Srivastava, J., 2020, October. RKT: relation-aware self-attention for knowledge tracing. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 1205-1214).\n\n4.\tFarhana, E., Rutherford, T. and Lynch, C.F., 2022, June. Predictive student modelling in an online reading platform. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 11, pp. 12735-12743).", "was_revised": true, "final_justification": "I have read the authors' rebuttal. The authors have addressed by concern and also added additional experimental results.\n\nThis work has promise in personalized learning. I have raised my score.", "rebuttal": "Thank you very much for your thoughtful review and for recognizing the potential of our work!\n \nYour decision to support this submission gives us the opportunity to bring a foundational contribution to the community. **We strongly believe that ExRec opens up a scalable and modular pathway for future research in personalized education**, enabling researchers to plug in their own datasets, KT models, or RL policies and push the boundaries of exercise recommendation systems. We truly appreciate your feedback in helping us refine and strengthen this vision.\n\nBelow, please find our clarifications to the points you raised.\n\n**W1. The limitations specified in the introduction “they often rely on ID-based embeddings, which neglects the semantics of question”—is not correct. The first DKT paper was on question ID-based embeddings. But after that, the weakness has been addressed by the community. Most DKT algorithms rely of embeddings of KC, such as AKT [1]. Embeddings of questions has been explored by researchers’ way before the LLM era, see a 2018 paper: EERNN[2], a 2020 paper: RAKT[3], a 2022 paper: SRL-KT [4].**\n\nThank you very much for your thoughtful comment! We agree that there has been meaningful progress in incorporating question and KC semantics into KT models, and we appreciate your references to EERNN, RAKT, and SRL-KT. However, we respectfully believe that the broader problem of relying on ID-based embeddings is **far from solved**, especially when viewed in the context of **exercise recommendation** or more generalizable KT frameworks.\n\nIndeed, our framework differs from prior approaches not just in using semantic information, but in **how** we use it. The cited models typically rely on shallow integration, such as pre-trained word embeddings or static text encoders. In contrast, **we train our question representations end-to-end via a contrastive learning module that explicitly aligns questions (and their solution steps) with corresponding knowledge concepts.** This yields richer, more interpretable embeddings that better support both generalization and downstream tasks like RL-based recommendation.\n\nFurther, even the most **recent and advanced KT models** [e.g., 1,2,3,4,5,6] still primarily rely on ID-based embeddings or even only KC embeddings. This highlights that, while promising work exists, the field has not fully moved beyond this limitation. We appreciate the opportunity to clarify this distinction and will revise our framing accordingly.\n\n[1] Li, Qing, Zhijun Huang, Jianwen Sun, Xin Yuan, Shengyingjie Liu, and Zhonghua Yan. \"HKT: Hierarchical structure-based knowledge tracing.\" Information Processing & Management. 2025.\n\n[2] Cheng, Weihua, Hanwen Du, Chunxiao Li, Ersheng Ni, Liangdi Tan, Tianqi Xu, and Yongxin Ni. \"Uncertainty-aware Knowledge Tracing.\" AAAI. 2025.\n\n[3] Bai, Youheng, Xueyi Li, Zitao Liu, Yaying Huang, Mi Tian, and Weiqi Luo. \"Rethinking and improving student learning and forgetting processes for attention based knowledge tracing models.\" AAAI. 2025.\n\n[4] Hou, Mingliang, Xueyi Li, Teng Guo, Zitao Liu, Mi Tian, Renqiang Luo, and Weiqi Luo. \"Cognitive Fluctuations Enhanced Attention Network for Knowledge Tracing.\" AAAI. 2025.\n\n[5] Bai, Youheng, Xueyi Li, Zitao Liu, Yaying Huang, Teng Guo, Mingliang Hou, Feng Xia, and Weiqi Luo. \"csKT: Addressing cold-start problem in knowledge tracing via kernel bias and cone attention.\" Expert Systems with Applications. 2025.\n\n[6] Shen, Xiaoxuan, Fenghua Yu, Yaqi Liu, Ruxia Liang, Qian Wan, Kai Yang, and Jianwen Sun. \"Revisiting knowledge tracing: A simple and powerful model.\" ACM International Conference on Multimedia. 2024.\n\n\n**W2: The KC annotation in Section 4.1 is a problematic, in my opinion. Prior deep learning era, KC annotation was performed by domain experts with multiple rounds of review. The proposed approach annotated KC by a sampling method through LLM with no expert intervention. The step can be performed by a machine-teaching or human-in-the loop fashion. Similarly, the solution step generation is done by LLM, but no quality assurance is performed.** \n\nThank you for raising this important point! We respectfully believe that the absence of expert intervention in our KC annotation process is not a drawback, but a key strength. Our aim is to reduce reliance on time-consuming and potentially inconsistent expert annotations to enable scalable and adaptable pipelines for new datasets and domains. That said, our framework is modular by design: teams with access to high-quality expert-labeled KCs can bypass this step and proceed directly to the subsequent modules.\n\nRegarding quality assurance, we build upon recent studies [1,2] that rigorously evaluated the use of LLMs for KC annotation. These works conducted both automated and human evaluations, comparing LLM-generated annotations with those provided in standard KT datasets. In both cases, the LLM-based annotations were consistently preferred, particularly for their clarity, generalizability, and semantic coherence: they often outperformed the dataset-provided expert labels.\n\nFinally, we agree that quality control remains important, and our approach naturally accommodates optional human review or refinement where needed. Our contribution lies in demonstrating that high-quality KC annotations can be generated automatically and reliably. This way, we significantly expand the accessibility and applicability of personalized educational systems.\n\n\n**Q1: Do the authors have any plan how to ensure QA of the generated solutions by LLMs?**\n\nThank you very much for raising this important point! While enhancing the mathematical reasoning abilities of LLMs is beyond the scope of our work, we fully agree that **ensuring the quality of generated solution steps is essential** for maintaining the validity of our framework.\n\nTo this end, we leveraged the fact that the XES3G5M dataset includes ground-truth answers. After generating solution steps via the LLM, we conducted a second round of LLM-based evaluation, comparing the final computed answer within the generated steps against the correct answer. Since extracting the answer from free-form output is non-trivial, this LLM-assisted comparison proved both **scalable and reliable**. We observed a correctness rate of over 98%, with most discrepancies involving minor rounding issues.\n\nCrucially, in our framework, what truly matters is **not the final numeric accuracy**, but the **verbal description of the strategy** expressed in the solution steps. This is because our contrastive learning and semantic embedding modules compute representations based on the **textual and conceptual content** of entire sentences, where numerical tokens contribute minimally to the embedding. Therefore, as long as the reasoning is sound and well-articulated, minor arithmetic deviations do not undermine the learning signal.\n\nStill, for flagged errors, we prompted the LLM again (this time providing the correct answer) to encourage better alignment. We further validated the QA process by manually reviewing 100 random samples marked as correct by the evaluator, and confirmed that they were indeed accurate.\n\nWhile not central to our core contribution, we will describe this procedure in detail in the implementation section. We appreciate your suggestion to include it!\n\n\n**L1:The authors have used only one dataset. They can also explore relevant K-12 math dataset on their proposed model, such as the ASSISTments datasets.** \n\nThank you for the suggestion. We initially considered using ASSISTments, which exists in several versions across different years. However, in the versions accessible to us, the question content was either **completely unavailable** or embedded in **highly noisy HTML**, often rendering the items **unreadable even for human annotators.** Given that our framework critically relies on understanding the semantic content of questions for KC annotation and exercise representation, ASSISTments was unfortunately not a viable option for a meaningful evaluation.\n\nThat said, we took your suggestion seriously and extended our evaluation to **another large-scale online math learning dataset, EEDI**, which offers clean and verified textual content. This dataset includes 47,560 students, 4,019 unique questions, and a total of 2,324,162 interactions. It shares many real-world characteristics with the XES3G5M dataset and is provided by the same organization behind the NIPS34 benchmark. Its questions were OCR-transcribed and reviewed by EEDI’s internal team, making it well-suited for semantic analysis.\n\n**Task 1: Global Knowledge Improvement. (Reported: % Max. Improvement as in paper)**\n\n|Random Policy|Historical Data|DDPG|DDPG w/ MVE|SAC|SAC w/ MVE|TD3|TD3 w/ MVE|TRPO|PPO|Discrete SAC|C51|Rainbow|DQN|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|-2.31|-0.21|16.65|18.87|8.97|20.08|9.62|16.85|1.78|2.83|18.81|20.83|37.00|30.51|\n\n*Note:* The reported improvements on the EEDI dataset appear lower than those on XES3G5M because the maximum possible absolute improvement is much larger (0.56 vs. 0.21). As a result, students must achieve larger absolute knowledge gains to reach full mastery. In absolute terms, the improvements on EEDI are comparable to those on XES3G5M, even though the percentages appear smaller.\n\nThanks to the flexibility of our framework, we were able to seamlessly apply our entire pipeline to EEDI, and we report the results in the revised version."}]

Author rebuttal:

Meta-review: The paper presents a novel framework for personalized exercise recommendation with semantically-grounded knowledge tracing. All reviewers agree that it produces interpretable learning trajectories, generalizes well to unseen questions, and demonstrates strong empirical performance across multiple evaluation tasks, with a clear and technically sound design.
Decision: Accept (poster)