# Personalizing the Sequencing of Learning Activities by using the Q-Learning and the Bayesian Knowledge Tracing

Amel Yessad

<figure><img src="image_1.png" /></figure>

## To cite this version:

Amel Yessad. Personalizing the Sequencing of Learning Activities by using the Q-Learning and the Bayesian Knowledge Tracing. 17th European Conference on Technology-Enhanced Learning, Sep 2022, Toulouse, France. hal-03710500

HAL Id: hal-03710500

https://inria.hal.science/hal-03710500v1

Submitted on 24 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Personalizing the Sequencing of Learning Activities by using the Q-Learning and the Bayesian Knowledge Tracing *

Amel Yessad¹[0000-0001-7575-6433]

Sorbonne Université, CNRS, LIP6, Paris, France
amel.yessad@lip6.fr

**Abstract.** In this paper, we present an approach for personalizing the sequencing of learning activities that relies on the Q-learning. The Q-learning learns a sequencing policy to select learning activities that aims to maximize the learning gain of students.

On the one hand, the core of this approach is the use of the Bayesian knowledge tracing (BKT) to model the student knowledge state and to define the Q-Learning reward function. On the other hand, we defined with experts rules to generate simulated students. These simulated data were used to initialize the Q-table of the Q-Learning and answer its “cold start” problem.

We present empirical results showing that the sequencing policy learned from the expert-based initialization of the Q-table provides the ITS with an efficient strategy to improve the students’ knowledge states in comparison with the Q-table randomly initialized. We further show that Q-Learning approach based on the knowledge states of the students inferred by the BKT are promising way for adaptive instruction in intelligent tutoring systems.

**Keywords:** Adaptive instruction · Q-Learning · Bayesian knowledge tracing.

## 1 Introduction

In intelligent tutoring systems (ITS), Curriculum Sequencing has been widely studied and consists on the planned sequence of learning activities (definitions, examples, questions, problems, etc.) that are most suitable according to the student characteristics [1].

Several research [6, 5, 7] have shown the interest of reinforcement learning (RL) for instructional policies, as RL models can learn complex and latent relationships between instructional tasks, students actions, and knowledge outcomes. In particular, the problem of sequencing the learning activities in ITS according to the student characteristics fits well a RL problem [5].

In this paper, we tackle the issue of the personalized sequencing of learning activities. To do this, we propose an approach that relies on the Q-Learning [4],

<footnote>* Supported by organization x.</footnote>
a RL algorithm. The sequencing policy learned in the Q-Learning is guided by the student knowledge state that is inferred by the bayesian knowledge tracing (BKT) [3].

We first formalise the problem of sequencing of learning activities by defining the main elements of the Q-Learning: the Q-table and the reward function. Then, we propose to initialize the Q-table with experts in order to answer the “cold start” problem of the Q-Learning, to fast its convergence and maximize the student learning gain.

To evaluate our approach, we carried out a first empirical study. The obtained results show that the sequencing policy learned from the expert-based initialization of the Q-table provides the ITS with an efficient strategy to improve the students' learning gains in comparison with the Q-table randomly initialized. We further show that Q-Learning approach based on the knowledge states of the students inferred by the BKT are promising way for adaptive instruction in ITS.

We will use the term Knowledge component (KC) in this paper, others refer to the KC as a skill, a concept or even a competency.

The paper is organized as follows: first, the principle of the Q-Learning and the BKT model are presented. The proposed approach and our contributions are summarized in Section 4. Then, the learning process and the experimental study are described in Section 6. Finally, the main conclusions are given in Section 7.

## 2 Q-Learning: Q-table and Q-function

Q-Learning is a RL algorithm where an agent learns the make decisions (actions) in different situations (states) through trial and error. It is relied on (1) a table, named Q-table that associates observed states $s$ with actions $a$ and (2) a function, named Q-function that maximizes a "value function" $Q(s, a)$ of an action $a$ for a state $s$.

In our case, the Q-table is the data structure used to calculate the maximum expected future rewards for each learning activity at each student knowledge state. This table will guide the Q-Learning agent to select the "best" learning activity for each student to maximize her learning gain. Each value of the Q-table is first initialized randomly and then learned via the following Q-function (or the Bellman equation):

$$ \underbrace{\text{NewQ}(s, a)}_{\text{New Q-Value}} = Q(s, a) + \alpha \left[ \underbrace{R(s, a)}_{\substack{\text{Immediate reward} \\ \text{learning rate}}} + \gamma \underbrace{\max Q'(s', a')}_{\substack{\text{Discount rate} \\ \text{Maximum predicted reward, given} \\ \text{new state s' and all its possible actions a'}}} - Q(s, a) \right] $$

Where $R(s, a)$ is the immediate reward received when selecting the action $a$ in the state $s$, $\alpha$ is the learning rate ($0 < \alpha \le 1$) and $\gamma$ is the discount rate
$(0 < \gamma \le 1)$, reflecting the importance of the immediate reward comparing to
the future rewards.

The reward function $R(s, a)$ is detailed in the section 5.2.

# 3 BKT model

The BKT model [3] is a two state hidden Markov model that is used to infer students' mastery of a knowledge component (KC) and at every practice opportunity, a student who has not mastered the KC has some probability of attaining mastery (the parameter $T$). If a student has mastered a KC, they will answer a question correctly unless they “slip” with some probability (the parameter $S$), and if the student has not mastered the KC, they can only guess correctly with some probability (the parameter $G$). In BKT, each KC is modeled separately without considering any relationships between KCs. Other work on individualized BKT models [2] describes different approaches for defining and learning student-specific parameters but we focus our approach only on the original BKT model.

<figure><img src="image_3.png" /><figcaption>Fig. 1. The BKT model and its parameters L, S and G</figcaption></figure>

# 4 Our approach and contributions

We propose an approach that learns how to assign learning activities to students in order to maximize their learning gains. This approach is based on connecting the Q-Learning to the BKT. Our contributions are:

1. We formulate the problem of sequencing of learning activities in order to maximize the student learning gain,

2. We propose a method based on simulated students to initialize the Q-table with "acceptable" values and answer the "cold start" problem of the Q-Learning algorithm. The simulated students were defined with experts.

3. We carried out a first experiment in order to evaluate the performance of the implemented system
# 5 Problem formulation

In this section, we formalize the problem of learning a sequencing policy for maximizing the student learning gain. It implies to define the mains components of the Q-Learning algorithm : the states and the actions of the Q-table and the reward function that maximizes the student learning gain.

## 5.1 Student knowledge state and learning activities

In accordance with the BKT model, each knowledge component (KC) is either in the learning state or in the unlearned state. Thus, we consider a knowledge state of a student as a vector of the mastering of each KC by the student (1 if the KC is learned by the student, 0 otherwise). The size of the vector is the number of the KC considered in the ITS. Thus, if we consider N KCs then we have 2^N possible knowledge states.

In Q-Learning, the Q-table is used to associate actions to states. In our case, we consider each student knowledge state as a state of the Q-table and each action as a learning activity (definition, example, demonstration, etc.). Each time a learning activity is proposed to the student, an associated testing activity is also proposed. The testing activity is mandatory and the student has to perform it before passing to new learning activity. That is serves us for updating the mastery of the KCs in the BKT model of the student.

## 5.2 Reward function

In each step for each student, the Q-Leaning selects a learning activity to present to the student, based on the Q-table and the ϵ-greedy exploration/exploitation strategy. Once, the student performs the testing activity associated to the selected learning activity, the BKT model infers the new mastery of the KCs worked on the learning activity. After converting the mastery probabilities to a binary values, the binary knowledge state is communicated to the Q-Learning agent for determining the next state in the Q-table. Finally, the agent recieves the immediate reward corresponding to the move from the knowledge state s to the knowledge state s' and updates the Q table entry according to the Bellman equation (cf. 2).

The reward function is defined as the following:

$$R(s, a) = \sum_{i=1}^{N} (s'_{i} - s_{i}) \quad \text{if } s'_{i} > s_{i}$$

where s' is the new knowledge state of the student inferred by the BKT model after selecting the learning activity a and N the number of KCs in the ITS. The underlying idea of this reward function is that the more new KCs are mastered, the greater the reward. The cumulative rewards quantify the learning gains of the students.
# 6 Learning process and experiment

In education, it is quite critical to initialize the Q-table randomly because the RL agent, before learning enough a good sequencing policy, can recommend activities that are not well adapted to the students who may thus have to complete more activities than later students, may spend more time to improve their knowledge state and may be demotivated to use the ITS. In order to address this concern—known as the “cold start” problem— we initialize the sequencing policy using simulated students. We focus this paper on this issue and aim to verify that the learning of the RL policy can be speeded up if the Q-table is initialized with data similar to the target data. Thus, we have implemented a three-step process:

1. An initializing step: it consists on initializing the Q-table with simulated students. Rules were defined with experts to generate simulated students (cf. 6.1). These simulated data were used to train for the first time the sequencing policy and initialize the Q-table.

2. A training step: in this step, the RL agent interacts with real students either by exploiting optimal decisions or by exploring other activities and updating the Q-table based on the expectation of the future rewards. These successive updates would allow the RL agent to converge to a good sequencing policy that maximize efficiently the students’ learning gains.

3. A testing step: when the RL agent has converged to a good sequencing strategy, it is time to use it to teach other students. These students will achieve their knowledge goals in the best way the RL agent has learned.

## 6.1 Simulated students

We model three student classes (strong, medium or weak) based on information provided by human experts about the number of attempts to answer correctly a testing activity and the prerequisite links between KC worked on in the ITS. Two rules were defined:

- R1: a student cannot answer correctly a testing activity on a KC without having first mastered all its prerequisite KC.

- R2: the probability that a student in the strong class answers correctly a testing activity is much higher than that of a student in the medium class and that of the latter is much higher than that of a student in the weak class.

These two rules were used to generate simulated data (500 simulated students in each class). These simulated students were used to learn a first sequencing policy in order to initialize the Q-table. We notice that these rules are only used to generate simulated data and not to learn to select learning activities.
## 6.2 Experimental study

We carried out a first experiment with fifty students who interacted with the ITS [anonymized], all of them are high school students starting to learn programming with python. The knowledge domain is composed of the following KCs: "variable", "sequential execution", "conditional structure" and "repetition structure". To evaluate the performance of connecting the Q-Learning to the BKT model, we compared our method against a random method of sequencing of learning activities. In the random method, we asked the students to complete learning activities in an order prescribed randomly. We randomly assigned 40 of participants to the Q-Learning sequencing condition (QS), and 10 to the random sequencing condition (RS). The objective of this first study is to answer the following two questions:

1. Is the initialization of the Q-table with simulated students allows real students to achieve efficiently the mastering of all the KC in comparaison with the Q-table randomly initialized?

2. Is the time spent by the students interacting with ITS in QS condition is less than the time they spent in the RS condition, before mastering all the KCs?

For the first question, we compared the convergence of two sequencing policies: the first policy learned with simulated students from a random-initialized Q-table and the second one learned with real data from the Q-table obtained after the first policy. In addition, we simplified the measure of efficiency as the average of the number of the learning activities proposed to the students. The time having no meaning for the simulated data.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** (A) Average number of learning activities displayed to the students in the case of sequencing policy learned from the Q-table initialized with simulated students and in the case of the sequencing policy learned from the Q-table initialized randomly (B) Average time spent by students interacting with the ITS in the case of random sequencing condition and the case of Q-Learning sequencing condition</figcaption></figure>
## 6.3 Results

On one hand, Fig. 2A shows the number of learning activities required by the students to master the content. The x axis shows the number of students that have interacted with the ITS. Initially, the Q-Learning needs around 30 learning activities to allow the simulated students to master all KCs. After the first 300 simulated students, the sequencing policy is tuned, obtaining a performance of less than 10 learning activities. However, once the Q-table is initialized thanks to the simulated students, the sequencing policy allows the real students to master all KCs with less than 10 learning activities.

On the other hand, in Fig. 2B we can observe how the students interacting with the Q-Learning sequencing policy spent less time to acquire all KCs than the students of the random condition, even when the Q-Learning is still tuning the sequencing policy. This result shows that the Q-Learning sequencing policy is useful for the students.

# 7 Conclusion

Personalized sequencing of learning activities is crucial for improving students’ learning gains in ITS.

This paper establishes connections between the Q-learning and the BKT and show that this mixed approach provides a potential solution.

The obtained preliminary results need to be further tested with other experiments by controlling variables such as the initial level of the students and even by using other individualized BKT models. There are several research directions for future work to provide evidence about the scaling-up of the approach.

## References

1. Aleven, Vincent, et al. "Instruction based on adaptive learning technologies." Handbook of research on learning and instruction (2016): 522-560

2. Yudelson, Michael V., Kenneth R. Koedinger, and Geoffrey J. Gordon. "Individualized bayesian knowledge tracing models." International conference on artificial intelligence in education. Springer, Berlin, Heidelberg, 2013

3. Corbett, Albert T., and John R. Anderson. "Knowledge tracing: Modeling the acquisition of procedural knowledge." User modeling and user-adapted interaction 4.4 (1994): 253-278.

4. Watkins, Christopher John Cornish Hellaby. "Learning from delayed rewards." (1989)

5. Bassen, Jonathan, et al. "Reinforcement learning for the adaptive scheduling of educational activities." Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020

6. Efremov, Aleksandr, Ahana Ghosh, and Adish Singla. "Zero-Shot Learning of Hint Policy via Reinforcement Learning and Program Synthesis." International Educational Data Mining Society (2020).

7. Doroudi, Shayan, Vincent Aleven, and Emma Brunskill. "Where's the reward?." International Journal of Artificial Intelligence in Education 29.4 (2019): 568-620.