Proximal Policy Distillation
Authors: Giacomo Spigler
Venue: Accepted by TMLR ()

Abstract: We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that integrates student-driven distillation and Proximal Policy Optimization (PPO) to increase sample efficiency and to leverage the additional rewards that the student policy collects during distillation. To assess the efficacy of our method, we compare PPD with two common alternatives, student-distill and teacher-distill, over a wide range of reinforcement learning environments that include discrete actions and continuous control (ATARI, Mujoco, and Procgen). For each environment and method, we perform distillation to a set of target student neural networks that are smaller, identical (self-distillation), or larger than the teacher network. Our findings indicate that PPD improves sample efficiency and produces better student policies compared to typical policy distillation approaches. Moreover, PPD demonstrates greater robustness than alternative methods when distilling policies from imperfect demonstrations. The code for the paper is released as part of a new Python library built on top of stable-baselines3 to facilitate policy distillation: <Anonymized GitHub Repository> .

Published in Transactions on Machine Learning Research (06/2025)
Proximal Policy Distillation
Giacomo Spigler
g. y.siiglerrHiibbrrpubbersrssy.
AI for Robotics Lab (AIR-Lab) Department of Cognitive Science and Artificial Intelligence Tilburg University
Reviewed on OpenReview: https: // openreview. net/ hor//ormm?? id- KfVXe88888
Abstract
We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that integrates student-driven distillation and Proximal Policy Optimization (PPO) to increase sample efficiency and to leverage the additional rewards that the student policy collects during distillation. To assess the efficacy of our method, we compare PPD with two common alternalives, student-distill and teacher-distill, over a wide range of reinforcement learning environments that include discrete actions and continuous control (ATARI, Mujoco, and Procgen). For cach environment and method, we perform distillation to a set of larget student neural networks that are smaller, identical (self-distillatiinn), or larger than the teacher network. Our findings indicate that PPD improves sample efficiency and produces better student policies compared to typical policy distillation approaches. Moreover, PPD demonstrates greater robustness than alternative methods when distilling policies from imperfect demonstrations. The code for the paper is released as part of a new Python library built on top of stable- baseliness to facilitate policy distillation: https: //github. cow.aau//cco///egg////eeeeeee distill
1
Introduction
Deep Reinforcement Learning (DRL) has been used to autonomously learn advanced behaviors wide range of challenging lasks, spanning from difficult games (Silver et al. 2017 Vinyals et al. 2019 Berner et al.
2019) to the control of complex robot bodies (Akkaya et al. 2019 Liu et al. 2022b
Haarnoja et al. 2024]. and advanced generalist agents [Reed et al. 2022 Abramson et al. 2020 Octo Model Team et al. 2024).
However, optimizing the performance of DRL agents typically requires extensive trial-and-error, requiring many iterations to identify effective reward functions and appropriate neural network architectures. As such, considerable research effort has been made to improve the sample efficiency of reinforcement learning. A particularly effective approach is to leverage prior knowledge -such that obtained from previous training runs AAgaral et al. 2022 through policy distillation (PD) (Rusu et al. 20166. Policy distillation inspired by knowledge distillation (KD) in supervised learning (Hinton et al. 2015). 1,, focuses on transferring knowledge from a 'teacher' neural network to a 'student" network that usually differs in architecture or hyperparameters.
Despite the conceptual similarity between KD and PD, their use and implementation are very different. KD is mainly usod for model compression, whereby functions learned by a high-capacity neural network are transferred smaller network. This helps to reduce inference time, which is crucial to run the final model embedded hardware. This setting is not common in DRL, the neural networks are typically kept small to shorten the long training times.
In contrast, PD is utilized differently in DRL. For example, a smaller model that is cheap to train can be 'renncarnated' into a larger one that is more expressive, possibly with different hyperparameters (Agarwal ct al. 2022), achieve better performance while limiting the computational resources required. Another effective use of PD in DRL to use teacher model(s) as 'skills priors', e.g., useful component behaviors that solve sub-tasks of problem, to make it easier for student agent to learn to solve more complex tasks

---

Published in Transactions on Machine Learning Research (06/2025)
(Merel et al. 2019 Liu et al. 2022b; Zhuang et al. 2023 In this case, PD can be used as regularizing term to bias towards behaviors that are expected to be useful for the larger, more complex lask, or to simply combine number of single-task teachers into a single multitask agent (Rusu et al. 2016). Finally, policy distillation can be used to re-map the types of inputs that a neural network receives. This is useful for example in robotics, where policies trained on state-based (or even privileged) observations can be distilled into vision-based policies (Akkaya et al. 2019 Liu et al. 2022ah.
Regarding implementation, KD typically involves adding distillation loss to the problem-specific (dataset) loss function, to train student network to reproduce the same output values the teacher model. While several recent works have explored combining policy distillation with reinforcement lcarning objectives
[Rajeswaran et al. 2017 Weihs et al. 2021: Nguyen et al.) 2022: Shenfeld et al., 2023), many policy distillation methods still use only a distillation loss, training the student through supervised learning rather than in a reinforcement learning setting. However, this overlooks the valuable information that can be gained from the rewards that the student collects during distillation. Exploiting these rewards could accelerate the distillation process, potentially enable the student to outperform the teacher, and reduce the risk of overfitting to imperfect teachers.
We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that combines student- driven distillation and Proximal Policy Optimization (PPO) (Schulman et al. 2017). PPD enhances PPO by incorporating a distillation loss to either perform traditional distillation, or to act as skills prior. Specifically, PPD enables the student to accelerate learning through distillation from a teacher, while potentially surpassing the teacher's performance. While PPD is designed and evaluated primarily with fully-observable students, it can also be applied in partially-observable tasks, for instance by using student policy with memory (e.g., LSTM-based), provided that the teacher policy has access to the full environment state.
The main contributions of this paper are:
1. We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that improves sample efficiency and final performance by combining Student-driven distillation with policy updates by PPO.
2. We perform thorough evaluation of PPD against two common methods for policy distillation, student- distill (on-policy distillation) and teacher-distill (Czarnecki et al., 2019), over wide range of RL environments spanning discrete and contimuous action spaces, and mun-oo-distribulaie generalization. We assess the performance of the three methods with smaller, identical (self-distillation) and larger student network sizes, compared to the teacher networks.
3. We analyze the robustness of PPD compared to the two baselines in a scenario with 'imperfect teachers', whose paramcters are artificially corrupted to decrease their performance.
4. We release new Python library, sb3-distill which implements the three methods within the baselneass framework (Raffin et al. 2019 to improve access to useful policy distillation methods.
2 Proximal Policy Distillation
Problem setting. We consider a reinforcement learning setting based on the Markov Decision Process A. r. Pu. (Sutton & Barto 2018). An agent interacts with the environment in discrete timestops == O,..,, that logether make up episodes. On the first timestep of each episode, the initial state of the environment sampled from the initial state distribution so ~ po(s). Then, at each timestep the agent selects action a E A using the policy function To S + A, represented by neural network with parameters 0 that lakes stales 8, E S as input. The environment dynamics are determined by the transition function p: S x A + S (i..., 8t41 p(st, at)). The agent receives rewards at each timestep depending on the previous state transition according to a reward function r r:S S x A + R. The objective of the reinforcement learning agent is to find an optimal policy x² that maximizes the expected sum discounted rewards
(S, p.
https://github. wtceaauub.cmmm////eeeeeeeeeeeeee wtw.spigl//eeceeceeeececco http://spperrrebbddiieeee sb__tistildddd1
2

---

Published in Transactions on Machine Learning Research (06/2025)
x² = arg max EYao Σ r'rela ~ (55)),50 ~ po(s), s441 ~ p(sc, oi)
In the context of policy distillation, we have access a teacher agent that has been trained solve a reinforcement learning task within the MDP framework. We particularly focus on teachers trained via actor-critic methods. Policy distillation then starts with Our objective is to transfer the knowledge from the teacher policy to the student while simultaneously training the student on a task using reinforcement learning. The new task can be the same as the teacher's (e.g., to implement reincarnating RL (Agarwal et al. 2022), or to perform model compression), or a new one (e.g., using the teacher as skills prior (Merel et al. 2019 Liu et al., 2022b Tirumala et al. 2022 ) The student should learn as efficiently as possible, and idcally be robust imperfect teachers, so that the performance of the student can in principle be higher than the teacher's.
student policy that is randomly initialized.
Approach. We build on the framework developed by Czarnecki al. (2019) since in its general form it already allows for the inclusion of environment rewards during distillation. The general form of the distillation gradient in this framework is
VoL(0) = Eg
- Vo log To(a: 1 + Vol(To, T()
(1)
t=1
policy-gradicn term distillation term
Where ! are the parameters of the student policy ne that we wish to train (possibly in conjunction with its value function V,,,, and {$1, 01, r1, 82, 02, 7711 are trajectories sampled by interacting with the target environment using control policy R₁ is a reward signal for timestep t, which can take many forms in vanilla Policy Gradient (e.g., sum of episode returns Σ/rl Tt. (discounted) sum of returns-to-go Eka Tr, the advantage function (s at), etc). Finally, f(TO, Vag Tr) is an auxiliary term for the chosen distillation method (for example, KL-divergence between teacher and student policies).
We focus on the setting where the student itself acts as the control policy, i.c., q = ma, since typically yields better performance after distillation than using the teacher. In fact, relying on the teacher for exploration biases the data toward its limited itate-visitation distribution, leaving many states unexplored (Czarnecki et al., 2019). In this paper, refer to distillation methods that use the student as the control policy student- driven, and those that use the teacher as teacher-driven.
To obtain Proximal Policy Distillation (PPD), we extend the formulation from Eq. by replacing the vanilla policy-gradient term with the Proximal Policy Optimization (PPO) surrogate (Schulman et al.) 2017). This requires three modifications.
(i) We improve sample efficiency by extending the original formulation re-use data from rollout buffer for several epochs, instead of using the sampled trajectories only once per update. As such, the control policy used to collect data for the k PPD update is the student policy from the previous step q = Téz and we need to introduce apportance-samllinn weights pr(0) introduce clipping to implement a proximality constraint to improve training stability, in PPO-clip. (iii) We choose KT.-divergence as distillation loss, which we also clip
=e(e.ls))
(ii) We
meş (asla.)
e Va. TI) =
(- (1) || T st)) max ((0),
Note that clipping the KL term is not required when using forward KL-divergence. However, simultaneous distillation from multiple teachers may benefit. from using reverse KT-divergence, for which clipping is necessary. We thus keep the clipping term to retain a more general formulation.
Rather than write another gradient, we package these ingredients into an implicit optimization problem which is iteratively updated by mini-batch SGD:
3

---

Published in Transactions on Machine Learning Research (06/2025)
To(a 1-c))
B arg max Bs 8
Lppo(s, a, --LL
AKI(TTeaccerr (Teachee (s) (s)) max
TT, s)
To(a 8) A(s.a). olc.)) g(c,A) Tex(a] s)
()A
A20 0
Ls a. ) min
(2)
- ()- A<0
where the proximality constraint is enforced by the hyperparameter e. LpPo is the PPO-clip loss, À is a hyperparameter that balances the relative strength of the PPO and distillation losses. Re, is the policy from the previous PPO iteration that was used to collect the rollout buffer, *0x+1 is the resulting policy at the end of the iteration, which will be used to collect the next rollout, and Tteacher is the teacher policy that we wish to use to guide learning. The critic is trained from scratch using a regression loss, as PPO.
The full algorithm is reported in Appendix A (algorithm
3 Empirical Evaluation
We compared PPD against two policy distillation methods: 'student-disuill" and "leacher-distil.. Student- distill (SD) is similar to 'on-policy distill' from Czarnecki et al. (2019), where policy distillation treated as a supervised learning problem over trajectories collected using the student as sampling (control) policy. Teacher-distie (TD) performed in the same way as student-distill, but the teacher policy is used to sample environment. trajectories instead of the student's. ss Pseudocode of the two baseline methods is included in Appendix algorithms and 3
Evaluation was performed on environments from Atari, Mujoco and Procgen because they span three axes that most affect performance in reinforcement learning and policy distillation: (i) state-space complexity low-dimensional states (Mujoco) vs. high-dimensional pixel observations (Atari & Procgen); (ii) action spaces discrete (Atari & Procgen) vs. continuous (Mujoco); (iii) dusto--distti generalization identical train test environment (Atari & Mujoco) vs. procedurally generated train/tess splits (Procgen). All tasks were episodic but discounted, thus differring from the strictly undiscounted formulation in (Czarnecki et al. 2019
Namely, we used the following environments:
subset of 11 games from the Atari suite (Atlantis, BeamRider, CrazyClimber, DemonAttack, Enduro, Freeway, MsPacman, Pong, Qbert, Seaquest, and Zaxxon).
4 Mujoco environments (Ant, HalfCheetah, Hopper, and Humanoid).
4 environments from the Procgen bechmark (coinrun, jumper, miner, and ninja).
We first trained teacher models for each environment using PPO, repeating the process over five random seeds. Full details of the training procedure, hyperparameters, and network architectures are provided in Appendix A.2
We then trained student models using the three policy distillation methods, PPD, SD, and TD, applied to three different student network architectures. These include a smaller network with approximately 25% of the teacher network's parameters, an identical network in self-distillation scenario (Furlancllo et al. 2018), and a larger network containing about 3-7 times more parameters than the corresponding teacher. We repeated all the experiments for cach random seed. Exact network sizes and architectures are included in Appendix A.3
Sample efficiency. We first examined the training curves during distillation to evaluate the sample efficiency of the different methods. Figure shows the case of distillation larger student networks. Similar figures for the other student network sizes are included in Appendix B.2
4

---

Published in Transactions on Machine Learning Research (06/2025)
We found that PPD typically learns faster and reaches better performance levels, except in the Mujoco environments, where student-distill showed faster learning pace, although final performance was comparable in both methods.
ppd
student-distill
teacher
4000
human-nomallizdd score
0.6
3000
reward
reward
0.4
2000
1000
steps (millions)
steps (millions)
steps (millions)
Training curves of PPD and student-distill in the setting of distillation onto a larger student network, showing the episodic returns collected during distillation. Distillation was performed over five random seeds and all environments in each of three suites 'atari', 'mujoco', and 'procgen'. The final training performance of the teacher models shown as a dashed line. Results for procgen are calculated over training levels. We observe that PPD students achieve higher rewards than student-distill ones, and are generally more sample efficient. The teacher-distill trace is omitted here because reflects performance on teacher-generated trajectories, leading to misleadingly high training scores; the complete curve is shown and discussed Supplementary Figure g Each curve is the interquartile mean over all environments in the suite, with 95% confidence intervals.
Figure 1:
Distillation performance. We then assessed the performance of student models trained with the three distillation methods. Evaluation was executed a lest setting (which the case of 'procgen' corresponds using different set of levels), where the distilled students were used interact with the environment. Actions chosen deterministically, instead of the stochastic action selection used during training, except for 'procgen', where observed that deterministic policies were prone to getting stuck, leading to lower performance for all agents. Results for Atari environments are reported as human- normalized scores, using base values from Badia et al. (2020).
Table D shows the results with respect to the corresponding teacher's performance. We found that PPD outperfoms both student-distill and teacher-distill across all environments. Additionally, we observed that, like in Figure [ the models perform close to each other on Mujoco environments.
Furthermore, noted that distilling into larger student networks generally results in better student performance after distillation, often surpassing that of the teacher models, especially for PPD.
We further show that PPD substantially outperforms a baseline with only PPO (without distillation) in Appendix B.4
Distillation from imperfect teachers. Since the goal of policy distillation is to reproduce the behavior of teacher policy into student network, the performance of the student is generally limited by the performance of the teacher. However, we found that PPD often achieves performance superior to its teacher. We investigated this aspect of the method further with a simple experiment where we deliberately corrupted the performance of the teacher models by perturbing their parameters with Gaussian noise
0: x- +
5

---

Published in Transactions on Machine Learning Research (06/2025)
Performance of student models of three sizes (smaller, same, larger), trained using the three distillation methods (PPD, student-distill, and toacher-distill). Evaluation is performed in test sctting (which in the case of "proogen' corresponds to using different set of levels). Results are calculated as fraction of teacher score for each environment and random seed, and then aggregated by geometric interquartile mean (95% confidence intervals are included in brackets).
Table 1:
env
smaller sd
td
ppd
atari
0.86x [0.81, 0.9]
0.94x [0.89, 1.0]
0.97x [0.94, 1.01]
mujoco
0.99x 00.98, 1.0
0.99x (0.98, 1.0]
0.99x
(0.97, 1.0]
procgen
0.72x
[0.69, 0.77]
0.78x [0.74, 0.83]
0.93x
[0.87, 1.01]
same-size
td
sd
ppd
atari
1.0x [0.97, 1.04]
1.07x [1.03, 1.12]
1.08x [1.04, 1.13]
mujoco
0.99x [0.98, 1.0]
1.0x 0.99, 1.0]
1.0x [1.0, 1.02]
procgen
0.84x [0.78, 0.89]
0.88x
[0.84, 0.94]
1.03x [0.97, 1.08)
larger
td
sd
ppd
atari
1.07x [1.03, 1.13]
1.09x [1.05, 1.15]
1.1x [1.05, 1.16)
mujoco
0.99x
[0.97, 1.0]
1.0x 110, 1.02]
1.0x [0.99, 1.01]
procgen
0.91x
[0.86, 0.98)
0.96x [0.93, 1.0]
1.04x [0.98, 1.08]
c~N(00.²)
with o = 0.05. We found that it can be challenging to obtain teachers with degraded performance that still manage to perform effectively in their target environments. For this reason, we focused on a subset of the environments (four Atari environments (BeamRider, CrazyClimber, MsPacman, Qbert} and four Procgen environments (miner, jumper, coinrun, ninja}}) for which reasonable imperfect teachers could be generated.
We then performed policy distillation using the three methods, and compared their performance relative to that of the original non-corrupted teachers. Evaluation was limited to students with larger networks. For Procgen, we evaluated the trained agents using test-levels (results for training levels are available in the Appendix, Table 9). Each experiment was repeated five times with different random seeds.
As shown in Table 2 PPD was found to consistently achieve better performance than both student-distill and teacher-distill, which instead fared only marginally better than their respective corrupted teachers.
Table 2:
Performance of student models, trained using the three distillation methods (PPD, student-distill, and teacher-distill) using 'imperfect teachers' that are artificially corrupted to decrease in performance. Results are calculated as a fraction of the original teacher score for each environment and random seed, and then aggregated by geometric interquartile mean (95% confidence intervals are included in brackets). Distillation is performed on four Atari and four Procgen environments, and onto larger student networks.
env
corrupted teacher
larger sd
score
td
ppd
atari
0.34x 0.233 0.51]
0.4x [0.28, 0.54]
0.46x [0.3, 0.64]
0.64x [0.48, 0.8]
procgen
0.65x 00663, 0.69]
0.63x [0.59, 0.67] 0.64x [0.61, 0.67]
0.71x (0668, 0.74]
Effect of hyperparameter We finally explored the impact the hyperparameter A, which balances the PPO and distillation losses, on distillation performance. The analysis was limited the same subset of Atari environments as in the study of imperfect teachers, and focused on distillation onto the larger network architecture. The value of the hyperparameter was varied over four values À E (0.5, 1, 5).

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 2 shows that increasing the value of speeds up distillation, due to the higher weight given to the distillation loss. Nevertheless, the impact of this hyperparameter is relatively small.
Test-time evaluation of the PPD students with different À showed that lower values allow students to learn better policies than the corresponding teacher, as expected since high prioritizes the distillation loss against the PPO loss, leaving less margin for the student to deviate from the teacher's actions. The relative final test. performance of students compared to the teacher are shown parentheses in the legend of Figure 2
0.5
0.4
0.3
1.0
5.0
0.2
90 teacher
0.1
0.0
steps (1e6)
Figure 2: Performance of student models trained using PPD with different values of X E {0.5, 1,2,5), over random seeds. Distillation is performed on four Atari environments and onto larger student networks. Dashed lines indicate 100% (black) and 90% (gray) of the performance of the teacher models. All curves denote interquartile means with 95% confidence intervals. A circle on each curve marks the last training step where the student's performance below 90% of the teacher's. All curves were collected during training, in Figure
4 Related Work
Many current policy distillation methods overlook the rewards obtained by the student during the distillation process. Instead, they frame policy distillation solcly as a supervised learning task. In this approach, a dataset of state observations and leacher outputs collected using either the student or teacher policies. The data then stored a distillation dataset from which training mini-batches are sampled.
A significant body of recent work focuses distillation from teachers with privilegod information full observability to partially-observable students (Nguyen et al. 2022 Weihs et al. 2021). where pure supervised learning cannot fully capture the teacher's behavior. In these scenarios, the combination of RI. with distillation becomes crucial for effective learning. While PPD was primarily designed for fully-observable settings, its integration of environment rewards with distillation makes it potentially suitable for such scenarios, as suggested by robustness imperfect teachers (albeit a different. context).
Meanwhile, PD form of supervised learning can be easily adapted to transfer functions between different types of models, such value- teachers (e.g., trained with DQN) and policy-based students. This proves beneficial in scenarios like reincarnating reinforcement learning (Agarwal et al. 2022). allowing continued training with alternate RL methods after distillation. For example, Green et al. (2019) showed that teachers trained using DQN can be distilled into "PPO students,", allowing for subsequent fine-tuning with Proximal Policy Optimization (PPO) Green et al.) 2019). However, unlike our work, PPO was not used during distillation, and the trajectorics for the distillation loss were collected in a teacher-driven way, with the teacher interacting with the environment instead of the student. This method closely resembles the 'teacher distill' strategy in baseline, which we demonstrated to be less effective than PPD.

---

Published in Transactions on Machine Learning Research (06/2025)
An alternative way to perform policy distillation is through imitation learning (IL), where demonstrations from teacher models provide specific actions as hard targets, rather than using the typical soft targets (e.g.. output probabilities). For example, ZZhuang et al. (2023) use DAgger (Dataset Aggregation) (Ross et al.) 2011) to distill five behaviors for a robot quadruped (climb, leap, crawl, tilt, run) into a single control policy, by selecting the appropriate teacher for each part of a training obstacle course. The advantage of DAgger over naive Bchavior Cloning stems from its ability to query teacher actions for new states encountered by the student, which is always possible in policy distillation, rather than relying solely on teacher-collected demonstrations. The combination of IL with RL has also been explored to improve the sample efficiency of RL methods and to bias behaviors learned with RL towards natural movements (Rajeswaran et al. 2017).
Most relevant to our approach are works that combine policy distillation with reinforcement learning in different ways. COSIL (Nguyen et al. 2022) and TGRL (Shenfeld et al. 2023) incorporate the distillation loss in the reward function, while ADVISOR (Weihs et al. 2021] adds it to a policy gradient objective like in PPD, although without clipping. All three methods, however, focus exclusively on settings with privileged teachers and partially observable students, using different. methods to dynamically trade off between teacher guidance and environment rewards. Similarly, (Schmitt et al. 2018) propose using Population Based Training (Jaderberg et al. 2017) to dynamically update this relative weighting. The main difference with Proximal Policy Distillation is that PPD extends PPO by applying proximal constraints to both objectives and enabling effective student-driven exploration, while these approaches rely on balancing schemes and, in the case of COSIL and TGRL, are based on different RL algorithms (SAC and IMPALA respectively). ADVISOR, like PPD, builds on PPO but differs in how it combines the objectives. Additionally, PPD is explicitly evaluated in settings with architectural differences between teacher and student, and shows robustness to imperfect. teachers even without dynamic loss balancing. While these prior works focus on the specific challenge of partial observability, we evaluate PPD across a wide range of standard environments including discrete and continuous control tasks, demonstrating its effectiveness as general policy distillation framework.
5 Discussion and Conclusions
We introduced Proximal Policy Distillation (PPD), a new method for policy distillation that combines reinforcement learning by PPO with a suitably constrained distillation loss.
Through a thorough evaluation over a wide range of environments, we showed that PPD achieves higher sample efficiency and better final performance after distillation, compared to two popular distillation baselines, student-distill and teacher-distil.. We suggest that this is due to the reuse of samples from the rollout buffer, together with the capacity to exploit environment rewards during distillation. We also confirmed that using the student policy to collect trajectories during distillation effectively reduces overfitting to teacher demonstrations (Czarnocki et al. 2019).
In theory, the sample efficiency of teacher-distill can be improved by first collecting a dataset of trajectories using the teacher and then applying offline supervised learning. Instead, our evaluation of teacher-distill was performed online, by immediately using and then discarding teacher trajectorics. We note however that reusing demonstrations collected using the teacher policy may exacerbate overfitting to the teacher. This is shown in the Appendix (Figure g while teacher-distill can quickly match the teacher performance during training, that is on trajectories generated by the teacher itself, its performance drops significantly when the distilled student is used to interact with the environment (as shown in Table 1
We also found that using larger student neural networks during distillation correlates with improved perfor- mance, even surpassing that of the original teacher. Our results thus validate the advantages of 'rinccarnating' reinforcement learning agents that are first trained simpler networks to reduce training time into larger and more capable networks [Agarwal et al. 2022 Schmitt et al. 2018). However, if the student is to contimue learning using any actor-critic RL method, it becomes necessary to also distill the critic of the teacher together with its policy netw (actor). In the case of PPD, this is not necessary, since the student agent can learn its value function from scratch while interacting with the environment during distillation.
We further investigated the robustness of PPD compared to the two basclines in a special case where teachers were corrupted to perform suboptimally. This setting draws from the challenging, unresolved issue of learning
8

---

Published in Transactions on Machine Learning Research (06/2025)
from imperfect demonstrations (Gao et al. 2018 which stems from the fact that successful replication of teacher's 8 behavior would include both its good and bad actions. In our tests, we found that PPD students managed to regain a large fraction, although not all, of the original uncorrupted performance in the environments tested, outperforming both student-distill and teacher-distill. Most notably, PPD students consistently obtained higher rewards than the imperfect leachers they were learning from. Further analysis of the performance of the distilled models on training versus test levels on Proegen further suggests that both student-distill and teacher-distill remain limited by the performance of the (imperfect) teachers.
Finally, we released the 'sb3-distill" library to aid reproducibility of the work and facilitate the application of policy distillation methods, particularly Proximal Policy Distillation. The library implements the three methods PPD, student-distill, and teacher-distill, within the stable- baselies framework (Raffin et al. 2019 by introducing a new PolicyDiisillationAAggrtttt interface to extend 'OnPolicy Algorithm' classes.
Future work should focus on extending PPD use teachers as skills priors, especially in the case of multitask policy distillation. To make that practical, and to further help PPD students to achieve higher performance than their teacher, it will be beneficial to implement dynamic balancing in the trade-off between the PPO
and individual teacher distillation losses, as for example done by Schmitt et al. (2018) Weihs et al. (021))
Nguyen et al. 2022 : Shenfeld et al. (2023).
6 Acknowledgments
This research was supported by SURF grant EINF-5635. We gratefully acknowledge SURF (https://www surf.n] .nl) for providing access to the National Supercomputer Snellius. We also thank Murat. Kirtay and Bosong Ding for their valuable support and stimulating discussions, and the anonymous reviewers for their insightful feedback, which substantially improved this manuscript.
References
Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brusscc, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoe, Andrew Dudzik, el al. Imilating interactive intelligence. arXie preprinl 2rXiu:2012.06672 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Beyond tabula. rasa: Reincarnating reinforcement. learning. arx preprinl arXiv:2206. 0162,, 2022.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint ar 1xXxx:9100001 910.071133 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari human benchmark. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXie preprint arXiv: 10.2. 060600 2019.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiu preprint Xii:11912.01588, 2019.
Wojciech M Czarnecki, Razvan Pascamu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1331 1340. PMLR, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures In International conference on machine learning, pp. 1407 -416. PMLR, 2018.
htttps///github con/ hetppiglrrg/ebdddaeeeee distill

---

Published in Transactions on Machine Learning Research (06/2025)
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference. on machine learning, pp. 1607-1616, 2018.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. In 6th International Conference on Learning Representations, ICLR, Workshop Track Proceedings, 2018.
Sam Green, Craig M Vineyard, and Cetin Kaya Kog. Distillation strategies for proximal policy optimization. arXiv preprint arxiv: 1erXi:11001000 101.00020.. 2019.
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyasuvunakool, Noah Y Siegel, Roland Hafner, et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Science Roboties, :89):00088022, 2024.
Gooffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowlodge in a neural network. In NIPS Deep Learning and Represenlation Learning Workshop, 2015.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint ar Xxx::11111000000 09846,, 2017.
I-Chun Arthur Liu, Shagun Uppal, Gaurav S Sukhatme, Joseph J Lim, Peter Englert, and Youngwoon Lee. Distilling motion planner augmented policies into visual control policies for robot manipulation. In Conference on Robot Learning, pp. 641 650. PMLR, 2022a.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Ali Eslami, Daniel Hennes, Wojeiech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. Science Robotics, 7(69))eab,0235, 2022b.
Josh Morel, Leonard Hasenelover, Alexandre Galashov, Arun Ahuja, Vu Pham, Grog Wayne, Yee Whye Tch, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):5229 533, 2015.
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. arXiv preprint arXic:2211.0190) 2022.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Jocy Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Antonin Raffin. RI baselines3 zoo.
2020.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable basclines3, 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXio preprint or Xixi:: 779. 10087, 2017.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions Machine Learning Research, 2022.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Sourleenth international conference on arlificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülgehree Guillaume Desjardins, James Kirkpatrick, Razvan Pascamu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. In 4th International Conference on Learning Representalions, ICLR, 2016.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Docrsch, Wojciech M Czarnecki, Joel 7. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint 1r: 038.53 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint 107::17770000 06347, 2017.
Idan Shenfeld, Zhang- Wei Hong, Aviv Tamar, and Pulkit Agrawal. Tgrl: An algorithm for teacher guided reinforcement learning. In International Conference on Machine Learning, pp. 31077 31093. PMLR, 2023.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 350(7676):334--33) 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. The Journal of Machine Learning Research, 23(1)99989 10056, 2022.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
https://zeeoda: http:://eeoooooorrr//////////// org/record/8127025
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement. learning. Nature, 575(7782)3350 354, 2019.
Luca Weihs, Unnat Jain, Tou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing, Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134-19146, 2021.
Ziwen Zhuang, Zapeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In Conference on Robot Learning, 2023.
11

---

Published in Transactions on Machine Learning Research (06/2025)
A
Supplementary Methods
A.1 Algorithm listings and baseline methods
We include full algorithm listings for the three distillation methods compared in this work. PPD is shown in Algorithm 1 student-distill in Algorithm 2 and teacher-disulll in Algorithm
Algorithm Proximal Policy Distillation
Input: teacher policy Teeacher
Initialize student policy Th and value function Vg- for ....... do
Collect trajectories by running the student policy mo in the environment to fill rollout buffer Dj- {ss, d2 Ta s{)} with environment steps.
Compute returns ft and then advantage estimates, Å..
for epoch = 1,2, Repochs.... do
Shuffle rollout buffer.
for m = 1,2, Rmiiibatchess... do
Extract the m-th mini-batch Dr from Dj-
Update the policy and value functions by maximizing the combined objectives via gradient descent over mini-batches
0 arg max
Σ (s,a)EDT
(s a, 0) a e (8, a, ALPrD(S,
Dr
Σ
(vols)-1)
Φc+1 arg min
Drr
(s,a,R)ED
where
To(als) To, (a|s) Als, a). gle A(s.a)))
(1 (1+E)A,
A20 0
Lp a, = min
g(e,
(1 (1-
A
To(a|s) Te (a|s)
LPpD(s, a, (KL(Treacher(s) (s) max
end for
end for end for
12

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm Student-Distill
1: Input: teacher policy Tteacher and value function Vieacher Initialize student policy Ty and value function Vg.
for == 1,2,... do
Collect trajectories by rumning the student policy in the environment to fill rollout buffer
DR = {(*;, ai, (i, with environment steps.
5: Perform a step of gradient descent to obtain new parameters DR+1 and øk + 1
1 0k+1 = 17 'DDx
(Tteachherll [T) LeLetroppy s,
(s
0k+1 = Φx |Da
1
Σ
(Vols) Veeacher (s))
(s.a)ED)
6: end for
Algorithm Peacher-Distill
Input: teacher policy Treacher and value function Vieacher
Initialize student policy The and value function Vg.
for k 1,2.... do
Collect trajectories by rumning the teacher policy Tteacher in the environment to fill rollout buffer
D; = (st, (1,)),))) +{)} with environment steps.
5: Perform a step of gradient descent to obtain new parameters 02+1 and Φk +
1
OKt1 11 'DD
Σ
KL(Tteacher|| [T) Leenrropy (0,
(s.a)ED
Φk+1 Φb 'IDrk
Σ (s,a)ED
(Vo(s) Vrenceer (s))
6: end for
13

---

Published in Transactions on Machine Learning Research (06/2025)
A.2 Training of the teacher models
We train teacher models for each environment over five random seeds {100, 200,300, 400, 5003. We used the
following environments from Gymnasium Towees et al. 2023 and Procgen (Cobbe et al. 2019
Atari (11 environments):
AtlanttisNoFrrmmskiipvvs EnduroNoFrammesiip-vy
SeaquestNooFramssklp-- FreewayloFrrmmeskipry
laaPacmannooFramestti-r-.. Ponglooframeskipp-v4 @berthoFrameskiiprgg laxxonllofraneskip-vv.
DemonAttrccMMoFPamssit---- CaazyCllamberNo77rrasssll....
Mujoco (5 environments): Ant-v4, HalfCheetah-v4, Hopper-v4, Swimmer-v4, Humanoid-v4.
Procgen (4 environments): miner, jumper, ninja, coinrun.
Training on Atari and Procgen enviromments was performed using an IMPALA-CNN architecture with {16,30 32, 32} convolutional filters, a 256-unit, fully connected layer, and ReLU activations (Espeholt et al., 2018). Separate neural networks were used for policy and value networks. Training on Atari was performed using PPO for a total of million environment sleps, while Procgen environments were trained for 30 million steps. Atari environments were wrapped terminate on the first life loss, without frame skipping, and using frame stacking with the 4 most recent environment frames. For Procgen, used 200 'casy' mode unique levels during training.
Mujoco enviromments used a MultiLayer Perceptron (MLP) backbone with two hidden layers of 256 units and ReLU activation. Separate networks were used to represent the policy and value functions. Training was performed for 10 million environment steps.
Rewards were always normalized using running average, and observations were normalized when training Mujoco environments. Final normalization statistics for Mujoco teachers were then fixed and reused during distillation, to guarantee that both student and teacher received the same inputs. This was not strictly required, and it was only chosen for simplicity.
The PPO hyperparameters are shown in Table 3. Simple hyperparameter tuning was initially performed,
starting from parameter values suggested from the stable basel--a model zoo (Raffin 2020
Table 3: PPO hyperparameters used for training the teacher models. Hyperparameter
Value
n envs
18
n steps batch size
256 (Atari, swimmer, hopper), 512 (Procgen, Mujoco)
512
0.995
0.9
Ir e_echshs ent ent__oeff total env steps
3e-4
4
0.01 (Atari, Procgen, swimmer, hopper), 0.0 (Mujoco) 10 million (Atari and Mujoco), 30 million (Procgen)
A.3 Policy distillation experiments
We performed policy distillation of all teachers, across all random seeds, for all PD methods (PPD, student- distill, teacher-distill) onto three differrntt student network sizes: smaller, same (same architecture as the student, i.e., self-distillationn, and larger.
The smaller network for Atari and Procgen was a Nature- CNN MMiih et al. 2015 with convolutional filters (2, (8s4, 4s2,3s1) and fully connected layer of 128 units, resulting 0.25x the number of parameters of the teacher. The larger networks for Atari and Procgen were IMPALA-CNNS with (32, 632,64,66 64)
14

---

Published in Transactions on Machine Learning Research (06/2025)
convolutional filters and 1024 units in the fully connected layer (~ 7.5r the number of parameters of the base teacher nerwork).
The smaller network for Mujoco was an MLP with two hidden layers of sizes 128 and 64 (~> 0.25x the number of parameters of the teacher), while the larger network had two hidden layers of size 512 (~> 3r the number of parameters of the base network)
Table reports the number of parameters for all student networks.
The hyperparameters of PPD related to PPO were the same for the teacher training, except used 0.999 during distillation (7 0.995 for swimmer and hopper), and_coef(p) and shorter rollout trajectories (n steps-64 for PPD, and n steps-5 for student-distil and teacher-distill). Distillation was performed for 2 million environment steps for all methods.
Evaluation of distilled students was performed with deterministic actions for Atari and Mujoco, and stochastic actions in Proegen. Evaluation Procgen environments was always on unseen lest levels, rather than the training levels used for training the teacher and for distillation. Results from Atari games reported with human-normalized scores, using base values from Badia et al. (2020). Rewards were averaged over 50 randomized episodes for each trained or distilled agent and random seed, for Atari and Mujoco, and 200 randomized episodes for Procgen.
Table 4: Number of parameters of the three networks used (smaller, same, larger) for each of the three domains (Atari, Mujoco, Procgen). Values in parentheses denote the relative size of each network with respect to the original teacher.
domain
smaller
samo-size
larger
atari
279.3k (0.25x)
1.1M (1x)
8.3M (7.54x)
mujoco
22.7k (0.24x)
95.8k (1x)
322.7k 7k (3.37x)
procgen
142.5k (0.23x)
626k (Ix)
4.6M (7.35x)
15

---

Published in Transactions on Machine Learning Research (06/2025)
B Supplementary Results
B.1 Teacher training
Training curves for all teachers, averaged over five random seeds, are shown as follows: Figure (Atari), Figure (Mujoco), and Figure (Procgen).
ScyusstW/fomcck...
B.arnN Rom.cko vd
Rowwsy/k/trmcck p
3000
000
4000
1500
500
3000
1000
400
2000
500
1000
steps (1.6)
steps (1e6)
steps (1e6]
steps (1eb)
Torghsttaranddzet
Cbetthotrrmaddpet
4000
20-
12000
8c00
2300
7000
reward
6400
2000
4000
eco0
1000
10
10
stops (3.6)
steps (1e6)
steps (1e6)
stops (1c6)
300000
150000
3000
40000
E50000
100000
22000
10
steps (1x6)
Steps (106)
steps (1o6)
Figure 3: Training curves for all Atari teachers, averaged over 5 random seeds. Shaded areas denote standard deviation.
0020
7503
6002
5003
5000
2520
2002
41003
1300
2000
2000
1002
1000
1020
10
steps (1e6)
steps (3c6)
steps (2e6)
stops (1.eG)
Figure 4: Training curves for all Mujoco teachers, averaged over 5 random sceds. Shaded arcas denote standard deviation.
B.2 Policy distillation
We include figures with the training performance during distillation different sizes of student networks, extending Figure from the main text. Figure 7 shows the training curves while distilling into the smaller
16

---

Published in Transactions on Machine Learning Research (06/2025)
reward
20
25
20
25
30
32
25
20
23
70
10
15
20
23
30
10
13
20
25
steps. (1e6)
steps (1e6)
steps (1e6)
steps (1.66
Figure 5: Training curves for all Procgen teachers, averaged over 5 random seeds. Shaded areas denote standard deviation.
student network, Figure 7 for the same-sized network, and Figure for the larger network. The latter is thus like Fig. from the main lext, but with the inclusion of the training curve for teacher-distill.
Note that while teacher-distill approaches the performance of the teacher model quickly, the result is misleading, since performance teacher-distill is measured over teacher trajectorics. Indoed, the final performance of teacher-distill evaluated using the distilled student's trajectories (see Table T is worse than both other models, suggesting significant overfitting.
We then report individual scores for all environments and students in Table and the individual, per- environment training trajectories for distillation to larger student networks for PPD and student-distlle in Figures
9 10 and
that is, the training curves that are combined to form Figure. in the main text..
4000
human-ramaa leed score
0.5
3000
reward
0.3
2000
1000
0.7
0.0
0.5
1.0
1.5
0.0
0.5
1.5
2.0
0.0
0.5
1.0
15
2.
steps (millions)
stops (millions)
stops (millions)
Figure Same as Figure
but with distillation smaller student.
17

---

Published in Transactions on Machine Learning Research (06/2025)
0.3
humannormalizse score
3000
reward
reward
pad Seacher
0.5
1.0
0.0
7.5
1.0
1.5
7.5
steps (millions)
steps (millions)
steps (millions)
Figure 7: Same as Figure but with distillation to same-sized student.
numan-nomallzee score
0.0
3000
roward
reward
2000
1000
Kiddat-diull touchsrdaill
(millions)
steps (millions)
steps (millions)
Figure 8: Same as Figure 1 showing distillation to larger student, but also including teacher-distill. Note that while teacher-distill approaches the performance of the teacher model quickly, the result is misleading, since performance in teacher-distill is measured over teacher trajectories. Indeed, the final performance of teacher-distil evaluated using the distilled student's trajectories (see Table [ is worse than both other models, suggesting significant overfitting.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Table The table extends Table 1 from the main text by reporting results for cach environment separately, averaged over 5 random seeds. Atari games are reported as human-normalized scores.
env
leacher score
smaller
same-size
larger sd
td
sd
ppd
td
sd
ppd
td
ppd
atari
atlantis
12.85
21.1
16.27
23.43
18.07
14.07
20.42
18.07
19.64
28.03
seaquest
0.04
0.04
0.04
0.04
0.04
0.04
0.04
0.04
0.04
0.04
beamrider
0.2
0.12
0.16
0.2
0.2
0.23
0.26
0.22
0.22
0.25
enduro
0.69
0.57
0.61
0.58
0.79
0.8
0.88
0.85
0.85
0.87
freeway
0.99
0.73
0.69
0.98
0.95
0.97
1.0
0.98
0.96
0.99
mspacman
0.35
0.33
0.34
0.31
0.34
0.38
0.34
0.39
0.36
0.38
pong
1.17
0.98
1.13
1.14
1.16
1.17
1.17
1.17
1.17
1.17
qbert
0.58
0.54
0.58
0.39
0.66
0.73
0.67
0.76
0.76
zaxxon
0.69
0.51
0.59
0.58
0.57
0.58
0.68
0.59
0.6
0.64
demonattack
4.45
5.62
11.08
11.61
9.64
10.25
8.71
9.37
7.59
9.55
crazyclimber
2.08
1.53
2.83
2.23
2.19
2.92
2.33
2.36
2.35
2.62
mujoco 3574
ant
3529
3505
3279
3466
3543
3549
3530
3630
3490
half-cheetah
6664
6588
6530
6549
6595
6621
6638
6567
6620
6631
hopper
2510
2451
2521
2501
2489
2496
2534
2454
2540
2527
humanoid
4540
3747
4226
4443
4070
4591
4804
3737
4416
4618
procgen-train
coinrun
7.29
5.39
5.55
6.76
5.94
6.12
6.99
6.63
6.91
7.25
jumper
8.39
6.98
7.07
7.89
8.22
8.29
8.41
8.29
8.43
8.43
miner
6.64
1.24
1.47
1.76
2.34
2.8
4.39
3.08
4.11
5.61
ninja
6.38
3.44
4.21
4.61
4.55
4.73
5.18
5.23
5.12
5.65
procgen-test.
coinrun
6.28
5.2
5.08
6.17
5.06
5.55
6.36
5.7
5.74
6.45
jumper
5.34
3.89
4.53
5.4
5.57
5.47
5.75
5.6
5.9
5.73
miner
4.57
1.22
1.46
1.49
1.87
2.51
3.66
2.37
3.34
4.43
ninja
4.44
3.34
3.83
4.33
4.1
4.13
4.68
4.29
4.58
4.57
19

---

Published in Transactions on Machine Learning Research (06/2025)
Sequertkiifomeapsss pad
human normllzes score
bumun-tommlizee score
Ruman normaltzed score
human nomeiiten score
0.
0.5
0.0
8.03
0.02
0.4
0.03
0.5
0.3
0.00
0.)
steps (millions
(million)
<Taps (millionss)
(milions)
rstaemanteermmmstt put
Chartlofimmet p-tt
Numma nonmar score
Tammat---ppmm maaa---eemallleeeeeee score
Tumor norma toboo score
0.00
0.0
12.0
0.4
022
staps (miliins)
50055 (millons)
stops (millions
Stops (million)
CrayCirbe Merronskipst
human- Numan--rrrnaa ppeec scoe
Humaan--ornnaa Srand s-come
ped
0.4
1.0
student clcil
0-
0.5
2/2
15
20
1.0
0.5
3.0
steps [millins]
Steps (million))
stegs (millions
Figure 9: Individual per-enviromment (Atari) training curves during distillation with PPD and student-distill onto larger student networks. Figure | (left) combines this data obtained by interquartile averaging, Curves shown are the interquartile mean over 5 random seeds. Shaded areas denote 95% confidence intervals.
Artuv
Kullchosah-ee
Hosporee
Humoro divd
5000
2000
2000
rewerd
2005
2002
1590
2002
2003
1000
1.0
0.5
2.0
0.5
1.0
2.2 0.3
2.0
stops (millions
stegs (millians)
stops (millions)
stops (million))
Figure 10: Individual per-environment (Mujoco) training curves during distillation with PPD and student- distill onto larger student networks. Figure [ (left) combines this data obtained by interquartile averaging. Curves shown are the interquartile mean over random seeds. Shaded areas denote 95% confidence intervals.
20

---

Published in Transactions on Machine Learning Research (06/2025)
reward
(millons)
(million)
(millios)
(millinn))
Figure 11: Individual per-environment (Procgen) training curves during distillation with PPD and student- distill onto larger student networks. Figure Curves shown are the interquartile mean over 5 random seeds. Shaded arcas denote 95% confidence intervals.
I (left) combines this data obtained by interquartile averaging.
21

---

Published in Transactions on Machine Learning Research (06/2025)
B.3 Distillation: imperfect teachers
We report the results of distillation from imperfect teachers for each environment in Table 8 The results from this table are aggregated and shown in Table from the main text.
Table Full results for each environment, extending Table 2 from the main text. We show the performance of student models, trained using the three distillation methods (PPD, student-distill, and teacher-distil) from 'imperfect teachers' that are artificially corrupted to decrease performance. Results are calculated as fraction of the original teacher score for each environment and random seed, and then aggregated by geometric interquartile mean (95% confidence intervals in brackets). Distillation is performed on four Atari (human-normnllized scores) and four Procgen environments, and onto larger student networks.
original teacher corrupted teacher
larger sd
score
score
td
ppd
BeamRider
0.2
0.15x
0.14x
0.16x
0.19x
TrazyClimber
2.08
0.22x
0.22x
0.3x
0.41x
MsPacman
0.35
0.38x
0.43x
0.47x
0.66x
Qbert
0.58
0.52x
0.65x
0.59x
0.8x
atari
0.42x
[0.33, 0.57]
0.48x
[0.38, 0.61]
0.52x
[0.38, 0.68)
0.69x
[0.55, 0.82]
miner
6.64
0.69x
0.36x
0.48x
0.69x
jumper
8.39
0.56x
0.6x
0.63
0.63x
coinrun
7.29
0.8x
0.71x
0.77x
0.83x
ninja
6.38
0.64x
0.69x
0.65x
0.72x
proegen
0.65x [0.63, 0.69]
0.63x
[0.59, 0.67]
0.64x [0.61, 0.67]
0.71x
[0.68, 0.74]
22

---

Published in Transactions on Machine Learning Research (06/2025)
B.4 PPD versus PPO
We analyze the elfectiveness of combining distillation and reinforcement learning in PPD by comparing against training student policies with pure PPO (without the distillation loss). We perform this comparison using larger student networks on the same subset of environments used in the imperfect teachers experiments (four Procgen and four Atari environments).
The results, shown in Table 7 demonstrate that PPD substantially outperforms pure PPO training, high- lighting the value of incorporating teacher guidance through our proximal distillation approach.
Table 7: Bascline comparison of PPO (without distillation) and the three distillation methods compared in our work: teacher-distill, student-distill, and PPD. Results calculated fraction of teacher score for each environment and random seed, and then aggregated by geometric interquartile mean (95% confidence intervals are included in brackets).
cnv
teacher
larger
score
ppo
id
sd
ppd
atari 0.48
0.38x [0.27, 0.51]
1.13x [1.02, 1.28)
1.13x [1.03, 1.29)
1.16x [1.05, 1.33]
proegen (train)
7.07
0.44x
[0.38, 0.5]
0.86x
[0.83, 0.89]
0.86x
[0.83, 0.9]
0.95x
[0.92, 0.98)
procgen (test)
5.07
0.71x
[0.63, 0.79]
0.91x (0.86, 0.98]
0.96x [0.93, 1.0)
1.04x
00.97, 1.08]
23

---

Published in Transactions on Machine Learning Research (06/2025)
B.5 Procgen: train test levels
We look into the performance of leachers and students on Procgen environments, where evaluation is performed on the original training levels instead of the unseen test levels used for evaluation in the main text.
Table extends the main results Table while Table shows results for the case of imperfect teachers.
Table 8:
Performance of students models of three sizes (smaller, same, larger), trained using the three distillation methods (PPD, student-distill, and teacher-distill), Evaluation performed separately on the train and test. levels of Procgen, contrary to Table I that only shows performance evaluated on unscen levels. Results are calculated as a fraction of tcacher score for each environment and random sced, and then aggregated by geometrie interquartile mean (95% confidence intervals in brackets).
teacher score
smaller sd
td
ppd
procgen (train)
7.07
0.62x [0.6, 0.66]
0.71x [0.66, 0.76)
0.81x [0.77, 0.87]
procgen (test)
5.07
0.72x [0.69, 0.77]
0.78x (0.74, 0.83)
0.93x (0.87, 1.01]
teacher
same-size sd
score
td
ppd
procgen (train)
7.07
0.76x [0.74, 0.79]
0.79x [0.75, 0.83)
0.88x [0.84, 0.91]
procgen (test)
5.07
0.84x [0.78, 0.89
0.88x (0.84, 0.94
1.03x
00.97, 1.08
teacher score
larger sd
ppd
procgen (train)
7.07
0.86x
[0.83, 0.89]
0.86x (0.83, 0.9]
0.95x
[0.92, 0.98)
proegen (test.) 5.07
0.91x [0.86, 0.98)
0.96x (0.93, 1.0]
1.04x [0.98, 1.08)
Table 9: Performance of student models, trained using the three distillation methods (PPD, student-distill, and teacher-distill) from 'imperfect teachers' that are artificially corrupted to decrease in performance. Results are calculated as fraction of the original teacher score for each environment and random seed, and then aggregated by geometric interquartile mean (95% confidence intervals brackets). Distillation is performed on four Atari and four Procgen environments, and onto larger student networks. Evaluation performed on training levels.
env
original score
corrupted score
larger sd
td
ppd
miner
6.64
0.82x
0.41x
0.56x
0.82x
jumper
8.39
0.89x
0.87x
0.88x
0.91x
coinrun
7.29
0.88x
0.86x
0.88x
0.92x
ninja
6.38
0.78x
0.78x
0.75x
0.83x
proegen (train)
0.86x [0.79, 0.91]
0.79x [0.77, 0.83)
0.79x [0.75, 0.83)
0.87x
[0.84, 0.89)
24


Reviews:
[{"review_id": "YATqrWpNaj", "reviewer": "Reviewer_buAe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ novel approach unifying policy distillation and PPO\n+ wide empirical evaluation wrt sample efficiency, performance under different teacher policy quality\n\nWeaknesses:\n- clarity and readability of some sections still needs improvement\n- results could be analysed in more depth in order to better understand the scope and strenghts of the method", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces Proximal Policy Distillation (PPD) a novel method that augments Proximal Policy Optimisation (PPO) with policy distillation, to allow a student agent to also leverage additional knowledge from the direct interactions with the environment, next to the distillation process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Section 2:*\n\nMy main remark regards the clarity of *Section 2*, more precisely the presentation of the PPD approach. I propose the following updates, in order to improve the presentation and allow for a self-contained contribution:\n\nEq. 1 presents the overall distillation framework introduced by Czarnecki et al. (2019), but it is currently impossible to grasp, lacking sufficient supporting text to explain the meaning and the various components. \n\t\n- First of all, I advise to clearly explain and define the meaning of \"student-driven distillation\". This would also allow to explain what the control/exploration policy is in your case: $q_\\theta = \\pi_\\theta$. Secondly, please explain the remaining components and notations: $\\mathscr{l}, \\hat{R}$.\n- In this text, \"We then extend the formulation by incorporating elements from Proximal Policy Distillation (PPO)\", I suppose you mean optimisation instead of distillation. \n- Finally I find it hard to conceptually make the transition from Eq. 1 to Eq. 2. Can you clarify how exactly you instantiate the framework proposed by Czarnecki et al. (2019)?\n\n*Section 3:*\n\n- It is appreciated that experiments were carries out on three suites of environments, but can you also motive and characterise this choice in terms of type of state/action spaces? Do you, similar to Czarnecki et al. (2019), only consider undiscounted objectives and episodic tasks? \n- In Figure 1, why are there no curves for teacher-distill? I see it is presented in Figure 8 in the appendix, so I am not sure why it is not part of the main text. Also do I understand correctly that results are averaged over all environments? It would be interesting to get insights per environment suite, for example understanding in more depth the Mujoco or procgen environments.\n\n*Section 1:*\n- Could you also clarify for which settings is PPD appropriate? You do mention in the related work that it targets fully-observable environments?", "broader_impact_concerns": "There is no Broader Impact Statement section in the work, but I do no think one is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UO8a0Eivl9", "reviewer": "Reviewer_Ffw1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the method is simple and intuitive. in addition to the PPO RL term, they add a PPO clipping loss between the teacher and student policy, just like how PPO does it to constrain the current policy and updated policy in its minibatch update.\n- the paper is well written.\n- there is some evidence that the method does well over pure policy distillation approaches.\n\nWeaknesses:\n\n- the method is not very novel, and the experimental comparison is not convincing enough to justify the method. many, many other methods combine RL and imitation learning / expert distillation objectives, see the references below for just a sampling.  **methodological novelty is not a dealbreaker to me** as simple methods that do well are good contributions. however, I am not convinced that this method is good due to the lack of baselines in the comparison.\n- you only compare against pure distillation approaches. you should really compare against other methods that also combine RL and IL / distillation, as those are the most competitive baselines. it's common knowledge  that RL + distillation outperforms pure distillation, as seen by the long line of prior work.\n- why not compare against PPO, which can be seen as an ablation / baseline that only uses the RL term instead of RL + distillation?  \n- missing ablations: why clip the distillation term with the PPO style clipping? it would be nice to show an ablation here to justify the choice.\n- the imperfect teacher experiments aren't really interesting in their current state, as we would expect pure distillation approaches do worse compared to RL + distillation approaches. a more fair comparison would be to compare your method against other RL + IL approaches in this regime. \n\n### References on methods that combine RL and imitation objectives:\n\nThese are just a few off the top of my head - see their related work for many more.\n\n- Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations\n- Bridging the Imitation Gap by Adaptive Insubordination\n- TGRL: An Algorithm for Teacher Guided Reinforcement Learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a policy distillation algorithm that combines the PPO RL objective and a clipped distillation term. \nThey benchmark it on some Atari, mujoco, and procgen tasks, and show gains over pure distillation approaches that don't optimize with RL rewards. They also show their method deals with imperfect teacher policies better than pure distillation objectives, because the RL objective rewards the policy for optimal behavior.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above, a more rigorous comparison is necessary for my recommendation.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "51qmO5sWCs", "reviewer": "Reviewer_AzDK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The paper is written well and presents the motivation and the technical contributions in a clear and easy-to-follow way.\n- The experimental section covers a wide array of environments, model sizes, and teachers (optimal and sub-optimal).\n- Very interesting results where distillation to a model of the same size results in 10% improvement in performance. This point to the usefulness of the method as a kind of regularization, although further analysis is needed to understand it properly.\n\n*Weaknesses*\n- The paper does not acknowledge multiple prior works that also dealt with a combination of distillation loss and reward maximization:\nNguyen, Hai, et al. \"Leveraging Fully Observable Policies for Learning under Partial Observability.\" Conference on Robot Learning. 2022.\nWeihs, Luca, et al. \"Bridging the imitation gap by adaptive insubordination.\" Advances in Neural Information Processing Systems 34 (2021): 19134-19146.\nShenfeld, Idan, et al. \"Tgrl: An algorithm for teacher guided reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n- Given these papers, the novelty of the proposed method is unclear and should be clarified.\n- One of the differences between PPD and prior work is the clipping of the KL loss. However, the paper doesn't include an explanation or intuition of why this change is beneficial, nor ablation regarding this part. Unlike policy gradient, KL minimization does not have to be 'on-policy' and can be minimized over arbitrary sequences.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines the problem of policy distillation in reinforcement learning. While policy distillation usually only tries to learn from a teacher, the paper suggests also performing reward maximization in cases where you have access to a reward function. By combining PPO loss and distillation loss the policy can benefit from both objectives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Please update the introduction\\related work with the relevant literature I mentioned above and clarify the difference between your work and theirs. This should also include Agrawal et al. 2022, which also uses a combination of RL and distillation.\n- Please add 95% CI for all results in the paper. In general, I advise using the metrics proposed in the following paper as they are currently the standard in the community:\nAgarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n- The proposed method outperforms the baselines in two out of three environments. Please add some analysis or discussion about why the method fails to outperform baselines in mujoco. I believe this can help in understanding the limitations of the method and be of interest to the community.\n- I am missing a baseline of only performing PPO.", "broader_impact_concerns": "The paper does not contain or require a Broader Impact Statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)