# Learning Feedback Policy from Historical Data: An Offline Approach within Pyrates

Badmavasan Kirouchenassamy, Amel Yessad, Sébastien Jolivet, Matthieu

Branthôme, Sébastien Lallé, Vanda Luengo

► To cite this version:

Badmavasan Kirouchenassamy, Amel Yessad, Sébastien Jolivet, Matthieu Branthôme, Sébastien Lallé, et al.. Learning Feedback Policy from Historical Data: An Offline Approach within Pyrates. International Conference on Artificial Intelligence in Education, Jul 2025, Palerme, Italy. pp.165-178, 10.1007/978-3-031-98414-3_12. hal-05158735

HAL Id: hal-05158735

https://hal.science/hal-05158735v1

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Learning Feedback Policy from Historical Data:
An Offline Approach within Pyrates

Badmavasan KIROUCHENASSAMY<fnref n="1" />[0009-0003-6502-154X],
Amel YESSAD<fnref n="1" />[0000-0001-7575-6433],
Sébastien JOLIVET<fnref n="1" />,<fnref n="2" />[0000-0003-3915-8465],
Matthieu BRANTHÔME<fnref n="3" />[0000-0002-9795-0760],
Sébastien LALLÉ<fnref n="1" />[0000-0003-4460-8336], and
Vanda LUENGO<fnref n="1" />[0000-0001-8978-0944]

**Abstract.** The expense and logistics of organizing experiments to train and evaluate teaching policies, as well as the potential negative impacts of these policies on the very first students, are significant challenges in the field of education. In this paper, we explore the feasibility of using offline reinforcement learning (RL) to learn an adaptive feedback policy that improves student progress in *Pyrates*, a programming platform. Leveraging an existing dataset of teacher-provided feedback and student interactions and codes, we have developed an offline RL model capable of learning an optimal feedback policy without direct interaction with students. The trained policy is then evaluated to assess its effectiveness in maximizing student progress and its suitability for online deployment in real-world educational settings. Our evaluation yields promising results, demonstrating that offline-trained policies can significantly enhance student progress, highlighting their potential for scalable deployment in programming platforms. However, challenges remain in ensuring robustness and adaptability when transitioning to online use.

**Keywords:** Feedback Policy · Computer science Education · Programming Platform · Offline Reinforcement Learning

## 1 Introduction

Feedback is one of the most powerful influences on learning and achievement [10, 21]. It is crucial in K-12 programming education as it helps students identify, understand and correct their errors, thereby enhancing their problem-solving skills [11, 17, 5]. However, a major challenge on platforms designed for young learners is the capacity of teachers to provide personalized feedback across different types of feedback [16, 30]. In crowded classrooms, teachers often lack

<footnote n="1">¹ Sorbonne Université, CNRS, LIP6, F-75005 Paris, France
{badamavsan.kirouchenassamy,amel.yessad,sebastien.lalle,vanda.luengo}@lip6.fr</footnote>

<footnote n="2">² IUFE & TECFA, Université de Genève, Suisse
sebastien.jolivet@unige.ch</footnote>

<footnote n="3">³ Université de Rennes, IRISA, CNRS, Lannion, France
matthieu.branthome@univ-rennes.fr</footnote>
the time and resources to thoroughly review students' codes and offer person-
alized feedback [26, 23, 3]. Automating feedback further ensures that students
receive immediate, consistent, and scalable guidance while reducing the burden
on teachers, especially in large-scale online courses [27]. The lack of mechanisms
to automate and adapt feedback significantly limits the student's progress, high-
lighting the need for intelligent systems capable of providing automated adaptive
feedback in real time [28].

The process of determining adaptive feedback involves decision-making under uncertainty for two key reasons. First, it's unclear which feedback type is best in a given situation since the system receives noisy, incomplete, and uncertain data about the student and their activity. Second, the impact of feedback on the student is also uncertain. Moreover, feedback policy may vary based on different objectives. For example, a policy designed to minimize the time it takes for a student to solve a problem could differ significantly from a policy that aims to improve the student's understanding of the underlying concepts. Based on these observations, Is it possible to train an AI model to automatically decide an adaptive feedback in such an uncertain context?

Among AI learning paradigms, Reinforcement Learning (RL) models are par-
ticularly well-suited for optimizing decisions via a trial-and-error strategy with-
out any supervision from experts, with delayed rewards capturing the cumulative
effect of sequential decisions [24, 25]. Several research [29, 7, 2, 6] have shown the
interest of reinforcement learning (RL) for instructional policies, as RL models
can learn complex and latent relationships between instructional tasks, students
actions, and knowledge outcomes.

Online reinforcement learning (RL) approaches rely on direct interaction with students to learn an effective feedback decision model. However, conducting such experiments requires significant logistical effort and resources, making it costly and complex to organize. Moreover, during the initial training phase, students may receive suboptimal feedback until the policy converges to an optimal one, potentially impacting their learning experience.

An alternate approach is to use offline RL algorithms that effectively leverage previously collected experiences without requiring online interaction. In theory, these approaches would allow learning an optimal feedback policy from data generated with any other feedback policy. However, offline RL remains a largely underexplored avenue in educational research, particularly regarding the autonomous selection of appropriate feedback across diverse feedback categories.

Therefore, in this paper, we address the following research questions:

- **RQ1:** To what extent can a feedback policy be learned from historical data using offline reinforcement learning to effectively maximize student progress?

- **RQ2:** How does the learned feedback policy compare to the policy used to generate the historical data in terms of effectiveness and decision-making patterns?

To answer these two research questions, we use the logs generated by the pro-
gramming platform named Pyrates [3] (Cf. section 3). Our approach focuses on
leveraging offline RL to learn a feedback policy that aims to maximize students'
progress using Pyrates' historical data.

In this article, we begin by reviewing related work, followed by an introduc-
tion to the context of Pyrates and our data collection process. We then formalize
the feedback decision problem before presenting and discussing the study results.

2 Related Work

2.1 Single vs. Multiple Feedback Types

In the early works, automated feedback systems in educational settings have increasingly leveraged Markov Decision Processes (MDPs) which is also a foundational framework for reinforcement learning. In the context of programming education, a prominent application of MDPs has been in automated hint generation [22, 18, 19, 20, 15]. These approaches typically formulate the feedback decision problem as selecting the most appropriate hint to guide students toward a correct solution. However, these methods often operate within a constrained decision space, limiting choices to the “best” feedback within a predefined type rather than considering fundamentally different feedback types. In contrast, our study aims to develop an system capable of dynamically selecting from multiple feedback types.

## 2.2 Online vs. Offline Reinforcement Learning

In the context of educational feedback, online reinforcement learning has gained traction, with models deployed directly within learning platforms to learn from real-time student interactions [9, 8]. These online methods aim to continuously adapt feedback policies based on immediate student responses. However, as highlighted in [6] and further emphasized by the challenges of real-world educational deployments, online RL faces significant limitations. These include the extensive interaction time required for convergence, the risk of exposing early students to suboptimal policies, and the potential for inconsistent feedback delivery due to continuous model updates.

To address these shortcomings, particularly the issues of suboptimal early
feedback and the need for efficient learning, offline reinforcement learning presents
a promising alternative [14, 13]. [6] discusses the application of offline RL in edu-
cational settings, demonstrating its potential to train models offline and deploy
them online, mitigating the risks associated with online learning. Offline RL,
which learns policies from previously collected datasets without further online
interaction, has achieved significant success in other domains, notably robotics
[12]. Another benefit of offline RL approaches is the ability to learn different
policies that optimize different criteria from the same dataset generated with
any policy.

The application offline RL techniques, such as Conservative Q-Learning (CQL)[13],
to address the feedback decision in educational settings has not been explored
to our knowledge. Therefore, this work is dedicated to bridging this gap, by
applying CQL in feedback decision context.
## 3 Presentation of Pyrates

Pyrates<fnref n="1" /> is a serious-game-based programming learning platform, engaging students in writing Python code to control the movement of an avatar [3]. In the context of Pyrates, there are 8 game levels, and each level has 5 types of feedback. For each level, experts have predefined feedback corresponding to the different types of feedback. The feedback types are incremental, offering detailed help: **TR** where the level's goal and relevant control functions are explained, **CO** where the relevant programming concept (e.g., for loop) is revealed, with a reference to the programming memo, **PI** where a guidance on implementing the concept in Python is presented, and **CR** where a sample solution is provided and finally **OTHER** where the teacher provided customized feedback that is different from the other four types of feedback. However, due to the very limited instances of this type of feedback and the lack of data on its specific content, we have chosen to exclude it from our study. Apart from this, all other types of feedback were selected by the teacher through the Pyrates interface in response to student requests. These feedback entries were systematically recorded alongside the students' activity traces, forming the dataset <fnref n="2" /> used for our analysis. More details on this dataset and its collection can be found in [4].

As discussed earlier, to avoid deploying online learning models that learn through interaction or rely on random feedback selection, we positioned the teacher as the decision-maker for feedback types. This approach allowed us to collect a reliable dataset without affecting students' learning experience. We gathered data from eight high school teachers, each deciding the type of feedback whenever students requested it. Upon exploring the collected data, we report in Table 2 the amount of feedback delivered per game level. Levels 6 to 8 are not included because too many students ran out of time. Level 2 also required less feedback, as sown in Table 2, most likely because it targets the same concept (loop) as Level 1. Thus in this paper, we focus only on Levels 1, 3, 4 and 5, i.e., the levels where feedback was the most needed and in which a sufficient amount of data was collected to train the RL feedback policy.

<table><caption>Table 1: Sample size and distribution of the feedback received per level.</caption><thead><tr><th rowspan="2">Level</th><th rowspan="2">#students</th><th rowspan="2">#feedback</th><th colspan="4">Feedback type distribution</th></tr><tr><th>TR</th><th>CO</th><th>PI</th><th>CR</th></tr></thead><tbody><tr><td>Level 1</td><td>215</td><td>176</td><td>22 (13%)</td><td>91 (52%)</td><td>51 (29%)</td><td>12 (7%)</td></tr><tr><td>Level 2</td><td>215</td><td>35</td><td>10 (29%)</td><td>5 (17%)</td><td>13 (37%)</td><td>7 (20%)</td></tr><tr><td>Level 3</td><td>215</td><td>408</td><td>69 (17%)</td><td>136 (33%)</td><td>134 (33%)</td><td>69 (17%)</td></tr><tr><td>Level 4</td><td>209</td><td>338</td><td>22 (7%)</td><td>97 (29%)</td><td>122 (36%)</td><td>97 (29%)</td></tr><tr><td>Level 5</td><td>187</td><td>164</td><td>18 (11%)</td><td>58 (35%)</td><td>53 (32%)</td><td>35 (21%)</td></tr></tbody></table>

<footnote n="1">¹ https://py-rates.fr/</footnote>

<footnote n="2">² https://github.com/Matthieu-BRANTHOME/PyratesIA/tree/L%40S_2025</footnote>
# 4 Learning Feedback Policy Using Offline RL

In this study, student progress refers to the degree of completion of the current game level relative to predefined objectives, which we assume directly reflects their in-game progression.

Within Pyrates, teachers choose the feedback type provided to students; nevertheless, around 40% of the feedback appears ineffective in producing immediate progress, based on the observed differences in student progress between consecutive submissions. We observe that even feedback type CR, which provides the solution to the student, does not always lead to the student's progress (16.3% of the provided feedback does not induce students' progress). Several reasons may explain this outcome. Teachers may adopt a long-term strategy, using feedback to promote student reflection, establish foundational understanding, or assess comprehension. Another reason is that the selection of feedback may not always align with the student's current stage, potentially limiting its effectiveness.

Thus, we frame feedback selection as a sequential decision process, where each feedback decision affects the student's subsequent state, with the overall impact determined by multiple interdependent choices over time. The 40% of teacher feedback that did not lead to immediate progress may have served as stepping stones for long-term learning. Following prior work on hint decision-making [7, 1], we adopt the same sequential decision framework and optimize the sequential decision making via RL approaches.

In RL, the **behavior policy** is the strategy that generates the training data, while the **target policy** is the one learned through optimization to maximize rewards. In our study, the behavior policy refers to the teacher's feedback policy used to generate the data, and the objective is to learn a target policy from the teacher's decisions that better optimizes student progress.

## 4.1 Problem Formulation

In RL, the model (or agent) interacts with an environment, which is typically modeled as a Markov Decision Process (MDP). In the context of Pyrates, the MDP is defined by the tuple $M = (S, A, P, R)$ as follows:

- Each state $s \in S$ represents the environment at a given time step with which the RL agent interacts. Although various pieces of information are available from the Pyrates student log data, we chose to represent the state using only two attributes—student progress and attempt count (i.e., the number of feedback instances provided). This simplification is driven by the necessity to maintain sufficient data coverage and improve learning performance, considering the limited availability of data, a lower-dimensional state representation leads to better data coverage and improved learning performance. However, it may fail to capture nuanced feedback strategies. Using many features would create a large number of unique states, each with fewer data points, thus limiting the exploration of the state space. Student progress provides insights into the types of errors made, while the attempt count reflects
the level of student struggle; together, they form a concise yet informative state representation. In our implementation, we experimented with state representations based solely on student progress, solely on attempt count, and a combination of both to analyze the impact of state dimensionality on model learning outcomes.

- Each action $a \in \mathcal{A}$ corresponds to a specific type of feedback provided by teachers. The set $\mathcal{A}$ is composed of the 5 types of feedback.

- The dynamics are governed by the probability transition function: $P(s' | s, a) = \text{Pr}\{s_{t+1} = s' | s_t = s, a_t = a\}$, where $s'$ denotes the subsequent state of the student after receiving feedback $a$ in state $s$. In our setting, these dynamics are not known a priori; however, we aim to learn them through offline RL agent learning.

- The reward function $R$ plays a crucial role in guiding policy learning and must be carefully designed. In the Pyrates context, feedback types vary in the amount of information they reveal to students. Notably, one feedback type, CR, containing significantly more information than other types. To avoid biasing the learning policy towards this feedback, we introduce a reward framework that normalizes rewards based on the degree of information revealed in each action (noted as $I(a)$). Therefore, the reward for each feedback action $a$ is given by:

$$R(s, a, s') = \begin{cases} \frac{1}{I(a)}, & \text{if the feedback } a \text{ improves the student's progress,} \\ 0, & \text{otherwise.} \end{cases}$$

The goal is to iteratively learn the target policy $\pi^*: S \to \mathcal{A}$ that maximizes students' progress over time. We choose student' progress in the game as the primary metric to track improvement over time.

## 4.2 Data Collection and Q-Learning for Offline RL

Formally, the offline dataset is given by: $D = \{(s_i, a_i, r_i, s'_i)\}_{i=1}^N$, where: $s_i \in S$ is the state (student's state) at time step $i$, $a_i \in \mathcal{A}$ is the feedback type provided by the teacher, $r_i = R(s_i, a_i, s'_i)$ is the observed reward, $s'_i \in S$ is the next state after provision of the feedback. This dataset is generated by a behavior policy $\mu(a | s)$ that may differ from the target policy we want to learn.

As illustrated in Figure 1, the distribution of actions per state is markedly uneven, with certain states being explored much more frequently than others. This imbalance in state exploration poses a challenge for conventional Q-learning, particularly in an offline training context, where many state-action pairs remain under-sampled or unexplored. To address this issue, we adopt a conservative Q-learning approach [13], which incorporates a penalty for actions associated with high uncertainty or sparse data, reducing the overestimation bias in Q-values and stabilizing learning in offline RL.

Essentially, a Q-value estimates the benefit of performing an action in a given state. In standard Q-learning, the Q-value is updated via the Bellman equation
<figure><img src="image_2.png" /><figcaption>Fig. 1: Distribution of Actions counts across different states</figcaption></figure>

as follows: $Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma \max_{a' \in \mathcal{A}} Q(s', a')(1 - d) - Q(s, a)]$, where $r$ is the immediate reward received when transitioning from $s$ to $s'$ after taking action $a$, $d$ is 1 if $s'$ is terminal (0 otherwise), $\gamma$ is the discount factor and $\alpha$ is the learning rate which determines how much new information overrides old information.

In the conservative Q-learning variant, a penalty term is subtracted from the target to discourage overestimation for uncertain or rare state-action pairs: $Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma (\max_{a' \in \mathcal{A}} Q(s', a') - \beta C(s', a'))(1-d) - Q(s, a)]$, where $C(s', a')$ quantifies the uncertainty or rarity of the state-action pair, and $\beta \ge 0$ is a parameter controlling the strength of the penalty. This modification helps ensure that overestimated Q-values for out-of-distribution actions are effectively suppressed. In essence, conservative Q-learning "plays it safe" by tempering the Q-value estimates for actions that are poorly represented in the dataset, thereby reducing the risk of overestimation. This adjustment is crucial in our scenario, as it ensures that the learned policy remains robust and reliable, even when faced with the uneven distribution of explored states.

In our implementation, we trained the model using mini-batches of size 32 from the offline dataset, repeated over 1,000 iterations, ensuring stable and efficient parameter updates while balancing computational cost. To minimize random variability, we repeated the experiment 10 times. We applied a decaying discount factor, gradually reducing the importance of future rewards over time. This transition shifted the model's focus from long-term rewards ($\gamma$ close to 1) to more immediate rewards ($\gamma$ approaching 0), ensuring adaptability and improved decision-making as training progressed.

## 4.3 Evaluation Metrics

Evaluating policies derived from offline training is a challenge in itself. Unlike online methods, offline training relies on pre-collected datasets, which can introduce biases due to distributional shifts and limited state-action coverage. This setting prevents direct interaction with the environment, making it essential
to employ robust evaluation techniques that can accurately assess policy performance without real-time feedback. In our study, we adopt a three-pronged evaluation approach:

## 1. Offline Policy Evaluation Metrics

- **Importance Sampling (IS):** We use IS to evaluate a new policy using offline data. It works by reweighting each observed reward according to how likely the new policy would have taken that action compared to the behavior policy. The sum of these weighted rewards estimates the expected return of the new policy. The overall estimate varies between 0 and 1. For example, if the overall IS estimate is 0.5, it means that—on average—the new policy is expected to yield a reward of 0.5 per decision. This helps to gauge how much better (or worse) the new policy might perform compared to the data-generating policy. A low IS estimate does not mean the target policy is performing poorly. It simply means the target policy is already close to the behavior policy, and there may not be much room for improvement.

- **Fitted Q Evaluation (FQE):** FQE metric quantifies the expected cumulative reward that a target policy will accumulate over time, starting from a given state. Although individual rewards are bounded between 0 and 1, the FQE value can exceed 1 because it represents the sum of rewards over many time steps. In this context, a higher FQE estimate indicates superior policy performance, as it reflects the policy's ability to accrue a greater total reward in the long run. The exact value of the FQE metric is influenced by factors such as the discount factor(cf. Section 4.2) and the length of the decision horizon, providing a comprehensive measure of the policy's long-term efficacy.

2. **Policy Convergence Metrics:** Ensuring Q-value stabilization is key in offline training. We track Q-values during training, and their stabilization indicates that further iterations yield minimal changes. This convergence suggests the policy has captured the dataset's underlying dynamics, increasing confidence in its reliability.

3. **Comparison Analysis Between the Teacher and the RL Policies:** We introduce a novel comparison approach to evaluate whether the target policy outperforms the teacher's policy (which is the behavior policy used to generate the data). First, a teacher-weighted Q-value is calculated for each state by averaging the teacher's action frequencies with the corresponding model Q-values. This is compared to the best Q-value, which is the highest Q-value among the model's actions. A paired statistical test is conducted to determine if there is a significant difference between the teacher-weighted Q-values and the best Q-values, indicating whether the teacher's decisions systematically lead to suboptimal outcomes. Additionally, Cohen's kappa quantifies the agreement between teacher decisions (highest frequency action) and model decisions (highest Q-value). A permutation test generates a null distribution to assess the significance of this agreement.
# 5 Results

In this section, we present the results of our study. First, in Sections 5.1 and 5.2, we analyze the convergence and performance of the trained policy to address RQ1. This includes evaluating how effectively the policy learns and adapts over time. Next, in Section 5.3, we conduct a comparative analysis between the model's policy and the teacher's policy to answer RQ2, providing insights into their similarities, differences, and potential advantages.

## 5.1 Policy Convergence Metrics

Figure 2 shows that most models' Q-values converge during training, indicating the policy effectively learns the dataset's underlying dynamics and reward structure. A single feature state (student progress or attempt counts) stabilize faster, as they help the model quickly identify effective actions. In contrast, the combined feature state (student progress + attempt count) converges more slowly due to higher complexity, increased dimensionality, and potential noise. These results highlight that simpler state features promote faster convergence. Therefore, based on the shape of these curves, we can partially answer **RQ1**, concluding that it is possible to learn an optimal policy without knowing the policy used to generate the data.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Q-value variations across training episodes</figcaption></figure>
<table><caption>Table 2: Offline Policy Evaluation Performance Metrics: Analysis Across State Features and Levels</caption><thead><tr><th>Level</th><th>State Feature</th><th>IS estimate</th><th>FQE</th><th>IS Weighted Sum</th></tr></thead><tbody><tr><td rowspan="3">All Levels</td><td>Student Progress</td><td>0.3204</td><td>1.5577</td><td>1332.0</td></tr><tr><td>Attempt Count</td><td>0.2649</td><td>1.5725</td><td>1332.0</td></tr><tr><td>Student Progress, Attempt Count</td><td>0.2607</td><td>1.9094</td><td>1332.0</td></tr><tr><td rowspan="3">Level 1</td><td>Student Progress</td><td>0.4988</td><td>1.7925</td><td>212.0</td></tr><tr><td>Attempt Count</td><td>0.5532</td><td>1.7707</td><td>212.0</td></tr><tr><td>Student Progress, Attempt Count</td><td>0.4973</td><td>2.2579</td><td>212.0</td></tr><tr><td rowspan="3">Level 3</td><td>Student Progress</td><td>0.2456</td><td>1.6952</td><td>476.0</td></tr><tr><td>Attempt Count</td><td>0.3132</td><td>1.5921</td><td>476.0</td></tr><tr><td>Student Progress, Attempt Count</td><td>0.2240</td><td>2.2735</td><td>476.0</td></tr><tr><td rowspan="3">Level 4</td><td>Student Progress</td><td>0.2702</td><td>1.5858</td><td>404.0</td></tr><tr><td>Attempt Count</td><td>0.1382</td><td>1.5716</td><td>404.0</td></tr><tr><td>Student Progress, Attempt Count</td><td>0.1641</td><td>1.7941</td><td>404.0</td></tr><tr><td rowspan="3">Level 5</td><td>Student Progress</td><td>0.2805</td><td>1.2562</td><td>196.0</td></tr><tr><td>Attempt Count</td><td>0.2236</td><td>1.2256</td><td>196.0</td></tr><tr><td>Student Progress, Attempt Count</td><td>0.2321</td><td>1.7656</td><td>196.0</td></tr></tbody></table>

## 5.2 Offline Policy Evaluation Metrics

Across all levels and state representations (cf. Table 2), the IS estimates are positive. This consistent positivity suggests that, on average, the target policy is yielding positive returns when adjusted against the behavior policy. In practical terms, a positive IS estimate indicates that the re-weighted returns are not only non-negative but also potentially signal an improvement over the baseline behavior (ie. The teachers' policy).

In Level 1, the IS estimates range from 0.4988 to 0.5532, showing that the target policy performs significantly better than the behavior policy once we adjust for each action's likelihood. In contrast, Levels 3, 4, and 5 have lower IS estimates (typically between 0.1382 and 0.3132), meaning the target policy's advantage is not as evident in these levels.

The IS weighted sum reflects the total number of samples used for the estimate. Although Level 1 has a weighted sum of 212.0, the repeated sampling of the same state-action pair results in a high IS estimate. On the other hand, Level 3's weighted sum is larger at 476.0 as its data come from a broader range of state-action pairs. This broader coverage makes the IS estimate more conservative, as it does not rely heavily on a single frequently observed state-action pair.

Several factors may explain why Level 1 shows better metrics. First, Level 1 may be inherently easier or more predictable, allowing the target policy to clearly outperform the behavior policy. Second, even though there are fewer samples, the data in Level 1 might be more representative of optimal behavior, consistently yielding high returns. Third, the target policy may have learned the optimal actions for Level 1 more effectively, as indicated by the higher IS estimates and FQE values (up to 2.2579) that signal strong long-term value.
While Level 3 has the largest dataset, its increased diversity dilutes the performance gains; the behavior policy in Level 3 may also be well-tuned, leaving less room for improvement by the target policy to show significant improvements.

Across all levels, we see that FQE values are generally higher when both student progress and attempt count are combined. For example, at All Levels, FQE is 1.5577 for Student Progress, 1.5725 for Attempt Count, and 1.9094 for the combined state representation. This pattern is echoed at the individual level granularity: at Level 1, while the IS estimate is fairly similar across representations, the combined representation yields the highest FQE (2.2579), and similar trends appear for Levels 3, 4, and 5. This suggests that the long-term q-values of the policy are enhanced when both state features are considered, even if the immediate (IS) estimates do not always show a big difference. But FQE values vary across levels and combinations, highlighting the importance of feature selection and model tuning to improve policy evaluation accuracy.

Inconsistent performance of state features across levels suggests that the relevance and effectiveness of these features are context-dependent. In some levels, one state feature may capture critical aspects of the environment more effectively than in others, leading to better policy performance. In levels with simpler dynamics, the model might rely more on straightforward features like “Student Progress” whereas in levels with complex dynamics, the model might struggle to leverage simple features effectively, requiring more nuanced features like “Attempt Count”. This variability could be due to differences in the underlying dynamics, complexity, or structure of each level. Essentially, if one state feature allows the model to outperform another in a given level, it means that the feature is more informative or better aligned with the key decision-making elements in that specific context. Conversely, the lack of consistent superiority across all levels highlights the importance of tailoring state representations or combining features to account for the diversity in environmental challenges.

Our findings support **RQ1**, demonstrating that a feedback policy can be learned from historical data using offline RL to maximize student progress. Higher IS and FQE estimates, particularly with combined state features, indicate that the learned policy outperforms the behavioral policy in both immediate returns and long-term value. This confirms the potential of offline RL for optimizing feedback strategies. However, the approach depends on the quality and coverage of historical data. If key aspects of student behavior are underrepresented, the learned policy may struggle to generalize.

## 5.3 Teacher Policy vs Model Policy

The results of the paired-t test reveal that, in 12 out of 13 comparisons, the expected rewards of the teachers' policy are significantly different from the expected rewards of the model (with p-values ranging from 0.0000 to 0.0258), indicating that the policy of the teachers can systematically lead to suboptimal results relative to the optimal decisions learned by the model in regard to the objective of improving student progress. In parallel, Cohen's kappa values, which mostly range between 0.8082 and 1.0000 (with significant p-values except for one
Level 5 instance), suggest a high degree of agreement between the teacher's most frequent action in each state and the model's highest Q-value action in the same state. This apparent contradiction implies that, while the teacher and model generally concur on the preferred actions, the teacher's approach tends to yield lower expected rewards compared to the model's best feedback choices. These findings indicate that while teacher feedback may be noisy—reflecting variability across multiple teachers and even inconsistent choices by a single teacher—the model reliably captures the underlying decision tendencies by filtering out such noise. As a result, the learned policy generally agrees with the teachers in its actions, yet it identifies subtle improvements in reward outcomes, reinforcing its potential to outperform the teacher's strategy in practice. As for **RQ2**, our findings confirm that the learned policy is statistically significantly different from the teacher's feedback policy in terms of expected rewards and, as a result, students' progress, despite a high overall agreement in action decisions.

# 6 Conclusion & Future Work

Our results demonstrate that a feedback policy can be effectively learned from historical data, provided there is sufficient coverage of state-action pairs, ensuring diverse contextual representation. This feasibility study confirms that the learned policy aligns better with the current goal of improving student's game progress than the teacher-generated policy, enabling its deployment for automated feedback decision at scale. If certain model actions fail to induce game progress, this may indicate gaps in the current-state representation, emphasizing the need for more comprehensive data and feature enhancements. Ultimately, this iterative approach enables increasingly fine-grained decisions, and with additional metrics such as pre- and post-tests, it could extend beyond the goal of improving students' game progress to improving students' mastery concepts.

Importantly, our findings directly address the research gap regarding the feedback decision-making especially on cold start as the model has proven to be viable to be deployed in the online environment. While prior work has focused extensively on using RL to optimize overall instructional policies, our study shows that an effective feedback policy can still be learned through historical data. This not only confirms the viability of using offline data for policy induction in educational contexts but also suggests that further enhancements — such as richer state representations and additional performance metrics — could lead to even more personalized and adaptive feedback strategies.

Future work should explore deploying the learned feedback policy to provide real-world evidence of whether the policy leads to measurable learning gains compared to traditional teacher feedback. Additionally, future research could consider a richer pool of feedback, where each instance is characterized by multiple features (e.g., tone, format, and content). This raises an open question: can conservative Q-learning handle the complexity and noise of multi-dimensional feedback, or are further changes needed? Exploring this could refine our approach and boost reinforcement learning in education.
**Acknowledgments.** This study was funded by Generation5 as part of Badmavasan Kirouchenassamy's CIFRE PhD. We thank the students and teachers who participated in the Pyrates experiments and allowed us to collect data.

## References

1. Tiffany Barnes and John Stamper. Toward automatic hint generation for logic proof tutoring using historical student data. In *International conference on intelligent tutoring systems*, pages 373–382. Springer, 2008.

2. Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games, Ethan Fast, and John C Mitchell. Reinforcement learning for the adaptive scheduling of educational activities. In *Proceedings of the 2020 CHI conference on human factors in computing systems*, pages 1–12, 2020.

3. Matthieu Branthome. *Apprentissage de la programmation informatique: analyses et ressources pour accompagner la transition collège-lycée*. PhD Thesis, Université de Bretagne occidentale-Brest, 2023.

4. Matthieu Branthôme and Sébastien Lallé. Learning from Teachers: AI-Driven Feedback for a High School Python Serious Game. In *Proceedings of the 12th ACM Learning @ Scale Conference (L@S '25)*, Palermo, Italy, 2025.

5. Tyne Crow, Andrew Luxton-Reilly, and Burkhard Wuensche. Intelligent tutoring systems for programming education: a systematic review. In *Proceedings of the 20th Australasian Computing Education Conference*, pages 53–62, 2018.

6. Shayan Doroudi, Vincent Aleven, and Emma Brunskill. Where's the reward? *International Journal of Artificial Intelligence in Education*, 29(4):568–620, 2019.

7. Aleksandr Efremov, Ahana Ghosh, and Adish Singla. Zero-shot learning of hint policy via reinforcement learning and program synthesis. *International Educational Data Mining Society*, 2020.

8. Fahmid Morshed Fahid, Jonathan Rowe, Yeojin Kim, Shashank Srivastava, and James Lester. Online reinforcement learning-based pedagogical planning for narrative-centered learning environments. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 23191–23199, 2024.

9. Siyong Fu. A reinforcement learning-based smart educational environment for higher education. *International Journal of e-Collaboration (IJeC)*, 19(6):1–17, 2022.

10. John Hattie and Helen Timperley. The power of feedback. *Review of educational research*, 77(1):81–112, 2007.

11. Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. *ACM Transactions on Computing Education*, 19(1):1–43, January 2019.

12. Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials, 2023.

13. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. *Advances in Neural Information Processing Systems*, 33:1179–1191, 2020.

14. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. *arXiv preprint arXiv:2005.01643*, 2020.
15. Ali Malik, Mike Wu, Vrinda Vasavada, Jinpeng Song, Madison Coots, John Mitchell, Noah Goodman, and Chris Piech. Generative grading: near human-level accuracy for automated feedback on richly structured problems. arXiv preprint arXiv:1905.09916, 2019.

16. Chutaphon Masantiah, Shotiga Pasiphol, and Kamonwan Tangdhanakanond. Student and feedback: Which type of feedback is preferable? *Kasetsart Journal of Social Sciences*, 41(2):269–274, 2020.

17. Marcus Messer, Neil CC Brown, Michael Kölling, and Miaojing Shi. Automated grading and feedback tools for programming education: A systematic review. *ACM Transactions on Computing Education*, 24(1):1–43, 2024.

18. Chris Piech, Mehran Sahami, Jonathan Huang, and Leonidas Guibas. Autonomously generating hints by inferring problem solving policies. In *Proceedings of the second (2015) acm conference on learning@ scale*, pages 195–204, 2015.

19. Thomas Price, Rui Zhi, and Tiffany Barnes. Evaluation of a data-driven feedback algorithm for open-ended programming. *International Educational Data Mining Society*, 2017.

20. Kelly Rivers and Kenneth R Koedinger. Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. *International Journal of Artificial Intelligence in Education*, 27:37–64, 2017.

21. Valerie J. Shute. Focus on Formative Feedback. *Review of Educational Research*, 78(1):153–189, March 2008.

22. Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. Automated feedback generation for introductory programming assignments. In *Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation*, pages 15–26, Seattle Washington USA, June 2013. ACM.

23. Lihui Sun, Zhen Guo, and Linlin Hu. Educational games promote the development of students’ computational thinking: a meta-analytic review. *Interactive Learning Environments*, 31(6):3476–3490, 2023.

24. Richard S Sutton and Andrew G Barto. *Reinforcement learning: An introduction*. MIT press, 2018.

25. Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018.

26. Betül Tonbuloğlu and Ismail Tonbuloğlu. The effect of unplugged coding activities on computational thinking skills of middle school students. *Informatics in Education*, 18(2):403–426, 2019.

27. Ke Wang, Rishabh Singh, and Zhendong Su. Data-driven feedback generation for introductory programming exercises, 2017.

28. Husam Yaseen, Abdelaziz Saleh Mohammad, Najwa Ashal, Hesham Abusaimeh, Ahmad Ali, and Abdel-Aziz Ahmad Sharabati. The impact of adaptive learning technologies, personalized feedback, and interactive ai tools on student engagement: The moderating role of digital literacy. *Sustainability*, 17(3):1133, 2025.

29. Amel Yessad. Personalizing the sequencing of learning activities by using the q-learning and the bayesian knowledge tracing. In *European Conference on Technology Enhanced Learning*, pages 638–644. Springer, 2022.

30. Zhe Victor Zhang and Ken Hyland. Fostering student engagement with feedback: An integrated approach. *Assessing Writing*, 51:100586, 2022.