Relative Entropy Pathwise Policy Optimization
Authors: Claas A Voelcker
Axel Brunnbauer
Marcel Hussing
Michal Nauman
Pieter Abbeel
Radu Grosu
Eric Eaton
Amir-massoud Farahmand
Igor Gilitschenski
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
parallel simulation
value function
ppo
policy gradients
policy optimization
Primary area: reinforcement learning
TL;DR: REPPO uses straight-through gradient estimation via a surrogate Q function to obtain more accurate policy gradients.

Abstract: Score-function based methods for policy learning, such as REINFORCE and PPO, have delivered strong results in game-playing and robotics, yet their high variance often undermines training stability. Improving a policy through state-action value functions, for example by differentiating Q with regard to the policy, alleviates the variance issues. However, this requires an accurate action-conditioned value
function, which is notoriously hard to learn without relying on replay buffers for reusing past off-policy data. We present Relative Entropy Pathwise Policy Optimization, an algorithm that trains Q-value models purely from on-policy trajectories, unlocking the use of Q function derivatives to compute policy updates in the context of on-policy learning. We show how to combine stochastic policies
for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning. This results in an efficient on-policy algorithm that combines the stability of Q-based policy gradients with the simplicity and minimal memory footprint of standard on-policy learning. Compared to state-of-the-art on two standard GPU-parallelized benchmarks, REPPO provides strong empirical performance at superior sample efficiency, wall-clock time, memory footprint, and hyperparameter robustness.

Published as a conference paper at ICLR 2026
RELATIVE ENTROPY PATHWISE POLICY OPTIMIZATION
Claas A Voelcker1.2.x Axel Brunnbauer Marcel Hussing', Michal Nauman Pieter Abbeel Eric Eaton", Radu Grosu", Amir-massoud Farahmand 7 Igor Gilitschenskil.2
University of Toronto, Vector Institute, STU Wien, *University of Pennsylvania University of Warsaw, 'UC Berkeley, "Polytechnique Montréal, Mila Quebec Al Institute Authors contributed equally, correspondence cvoelckerecs ooootoo.ooommmooooooo edu, axel www.nbauereeeeee..m...bbbuue..ee twwienn ac.at
ABSTRACT
based methods for policy learning, such as REINFORCE and PPO, have delivered strong results game-playing and robotics, yet their high variance often undermines training stability. Improving policy through state-action value functions, for example by differentiating Q with regard the policy, alleviates the variance issues. However, this requires accurate function, which is notoriously hard to learn without relying on replay buffers for reusing past off-policy data. We present Relative Entropy Pathwise Policy Opti- mization, an algorithm that trains Q-value models purely from on-policy trajec- tories, unlocking the use of Q function derivatives to compute policy updates the context of on-policy learning. We show how to combine stochastic policies for exploration with constrained updates for stable training, and evaluate impor- tant urchitectural components that stabilize value function learning. This results in an efficient on-policy algorithm that combines the stability of Q-based pol- icy gradients with the simplicity and minimal memory footprint of standard on- policy learning- Compared to state-of-the-are on two standard GPU-parallelized benchmarks, REPPO provides strong empirical performance at superior sample efficiency, wall-clock time, memory footprint, and hyperparameter robustness.
condiitond value
REPPO Training
Algorithm Components
Evaluation Environments
66 00fs.a) Bezfols) da Policy gradients
On-policy Data
Q Learning
&
Cittic
Entropy Constraint
Kl. Constraint
Actor
Figure 1: Overview of the Relative Entropy Pathwise Policy Optimization algorithm. REPPO combines four core components to achieve stable and fast actor-critic training across a wide variety of reinforcement learning tasks. Isaac and Atari results can be found in the appendix. Implementa- tions in different popular RI. frameworks can be found at https: //g///// //githb.b.. com/reppo-rl.
INTRODUCTION
Most modern on- on-policy algorithms, such as TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017), use score-based gradient estimator to update the policy. These methods have proven ful for robotic control (Rudin et al., 2022; Kaufmann et al., 2023; Radosavovic et al., 2024), and mongl fine-tuning (Ouyang al., 2022; Touvron al., 2023; Gao et al., 2023; Liu al.,

---

Published as a conference paper at ICLR 2026
Value Approximation
Pathwise estimation
Eyrug Vof(o(s 0)]
By
Voflo(u, 0)
V pr [f())]
Importance Sampling
V Taps (2)
Ea-po (olgoo))))))
Egup
f(z)
Scure-bused estimation
az)
Gradient Estimation
Practical Computation
Figure 2: Overview of the strategies used by REPPO and PPO to obtain policy gradient estimators. Computing the gradient requires a mathematical transformation that allows for efficient estimation from samples, and additional steps that make the computation tractable in practice.
2024), but are often plagued by training instability. Zeroth-order, score-based gradient approxima- tion exhibits high variance (Greensmith et al., 2004), which leads to unstable learning (Ilyas et al., 2020; Rahn al., 2023), especially in high- high-dimensional continuous spaces (Li al., 2018). In ad- dition, it requires importance sampling to allow sample reuse, which exacerbates the high variance.
train parameterized state-action value function (Lillicrap al., 2016; Fuji- moto et al., 2018; Haarnoja et al., 2018), and use it to improve the policy, for example by using a pathwise policy gradient (Silver al., 2014), i.c. taking the derivative of the Q function wrt the action. Using a parameterized function to improve the policy often leads to faster and more stable learning learning by reducing the score-based estimators variance (Mohamed et al., 2020) and by allowing us to remove importance sampling corrections.
Alternatively
However, the effectiveness of these approaches bounded by the quality of the approximate value function (Silver et al., 2014). As such, algorithms that use state-action value function usually rely on improving value learning through off-policy training (Fujimoto et al., 2018; Haarnoja al., 2018). Unfortunately, off-policy training requires the use of replay buffers. Storing these replay buffers can be a challenge when the collected samples cannot in memory. addition, training with past data introduces various challenges for value function fitting (Thrun & Schwartz, 1993; Baird, 1995; Van Hasselt, 2010; Sutton et al., 2016; Kumar et al., 2021; ;; Nikishin et al., 2022; Lyle et al., 2024; Hussing et al., 2024; Voelcker et al., 2025). This raises our question:
Can we train a sufficiently correct value function approximation and effectively use it for policy improvement in fully on-policy setting without large replay buffers?
Building on the progress in accurate value function learning (Sutton, 1988; Haarnoja et al., 2019; Schwarzer et al., 2021; Hussing al., 2024; Farebrother et al., 2024), present an efficient on- policy algorithm, Relative Entropy Puthwise Policy Optimization (REPPO), which uses the pathwise gradient estimator with accurate value estimation trained on-policy data. REPPO builds on the maximum entropy framework (Ziebart et al., 2008) to encourage exploration. It combines this with KL regularization scheme, inspired by the Relative Entropy Policy Search method (Peters al., 2010), which prevents aggressive policy updates from destabilizing the optimization.
Furthermore, we evaluate several prominent advances in neural network architecture design sta- bilize learning: categorical Q-learning (Farebrother et al., 2024), normalized neural network archi- tectures (Nauman al., 2024a; Hussing et al., 2024), and auxiliary tasks (Jaderberg et al., 2017). These components feature in many recent variants (Schwarzer et al., 2021; 2023; Nauman et al., 2024a; Hussing et al., 2024; Gallici et al., 2024; Lee et al., 2025a;b; Nauman et al., 2025; Fujimoto et al., 2024) of common value learning algorithm such SAC (Haarnoja et al., 2018). We find that categorical Q-learning and normalization have a strong impact on the performance, while auxiliary tasks only show small impact, but become more relevant when reducing the amount of samples.
We test our approach in a variety of locomotion and manipulation environments from the Mujoco Playground (Zakka et al., 2025) and ManiSkill (Tao et al., 2025) benchmarks, and show that REPPO competitive with tuned on on-policy baselines in terms of sample efficiency and wall-clock time, while using significantly smaller memory footprints than comparable off off-policy algorithms. Fur-
2

---

Published as a conference paper at ICLR 2026
thermore, find that the proposed method is robust the choice of hyperparameters. To this end, our method offers stable performance across more than 30 tasks spanning multiple benchmarks with single hyperparameter set. In introducing REPPO, work makes the following contributions:
We showcase that using a state-action value function and a pathwise policy gradient can be effective in on-policy RI., as it allows on-policy action resampling, forgoing importance corrections. However, this requires learning a highly accurate state-action value function.
2. We show how a joint entropy and policy deviation tuning objective can address the twin problems of sufficient exploration and controlled policy updates.
3. We evaluate architectural components such as cross-entropy losses, layer normalization, and auxiliary tasks for their efficacy in pathwise policy gradient-based on-policy learning.
We provide sample implementations in both the JAX (Bradbury et al., 2018) and PyTorch (Paszke com com/reppo-rl..
et al., 2019) frameworks at https::/ggttie
2 BACKGROUND, NOTATION, AND DEFINITIONS
We consider the setting of the Markov Decision Process (MDP) (Puterman, 1994). defined by the tuple (X' A., P, 7, Y, Po). where X' is the set of states, A is the set of actions, P(x'|æ,, a) is the transition probability kemel,, r(æ, a) is the reward function, and y E [0, 1) is the discount factor. We write P= (x:) for the molicy-conditioned transition kernel and (yllar for the n-step transi- tion kernel. An agent interacts with the environment via a policy (a) which defines a distri- bution over actions given a state. The objective is find policy that maximizes the expected discounted return, ((T) = EE 12r=0 7'r(I,, at)], where Ho Po is the initial state distribu- tion, and at T--(2))- The state-action value function associated with policy и are defined as
(=(x, E E 7'r(21, a+) Zo I, ag
We use (y(x) to denote the discounted sta- tionary distribution over states y when starting in state x. When us (-19), y Po. will simply write (x) to denote the probability state under the discounted occupancy distribution."
2.1 POLICY GRADIENT LEARNING
A policy gradient approach (Sutton & Barto, 2018) is general method for improving (parame- terized) policy тy by estimating the gradient of the policy-return function ((Te) with regard to the policy parameters 0. The policy gradient theorem states that
(oo())) = Rerprr (- (Q (I, a) Vo log mo(a[z))-
This identity is particularly useful both the Q value and the stationary distribution can be estimated by samples obtained from following the policy for sufficiently many steps in the environment.
An alternative approach is the deterministic policy gradient theorem (DPG) (Silver et al., 2014) 4). The estimator for the DPG relies on access to differentiable state-action value function and deterministic differentiable policy zdet (1:). While access to the true value function is an unrealistic
assumption, we can use a trained surrogate model, Q, to obtain a biased estimate of the gradient
VoJ(To)
Egpoop V (Ir, a) a))a=x)((( Vonget (1:))-
Finally, the DPG can be expanded to reparametrizable stochastic 2 We term this the pathwise policy gradient, following Mohamed al. (2020). but the formulation has been used prominently in prior work such as SAC (Haarnoja et al., 2018), just without proper name. The gradient estimator can be obtained from the following expectation
V (mg) R Ex=px..-ppe) [V [V.Q00" (x, a) a))a==
V g
rep (x, e)].
(3)
where T <) is a reparameterization of To (a(x). To avoid notational clutter we will write To(alz) from now on to always mean the appropriate reparameterization.
Many policy gradient methods the undiscounted empirical state occupancy for optimization (Nota & Thomas, 2020). While this could considered an error, is nonetheless a common approximation. REPPO similarly uses empirical samples without accounting for the discount factor in the objective.
We discuss extension non-reparametrizahle, discrete policies in Appendix C.
3

---

Published as a conference paper at ICLR 2026
Comparison of Policy Optinizzion Medhods
Men DPG
Sense Gradess
Phutoricc Goulio:
Sumnggan
Strogaa PPG
Strong Sammggu DEG
Strvag Saringar
Optinxen
300 D000 1250
2000
2300
Ptoration
(a) Achieved returns (left) path of policies trained different gradient estimation methods. compare a score-functies based policy gradient estimator (blue) with three variants of pathwise gradient es- timators: using the ground truth objective function (orange); an inaccurate surrogate model (green), and an accurate surrogate model (red). All PPG based methods show markedly reduced variance in the policy updates.
REINFORCE
PPO
Pstoai
8.0
(b) Gradient path over eight steps in the middle of the trajectory, visualized per algorithm for 8 steps. For Reinforce and PPG, samples drawn every step. For the importance sampling based algorithms, set samples sampled the beginning and subsequent steps are conducted using importance sampling.
Figure 3: Visualization of gradient paths on 2D example function.
2.2 UNDERSTANDING SOURCES HARMFUL VARIANCE IN GRADIENT ESTIMATION
To build additional intuition the differences between different policy gradient estimators, we conduct an illustrative experiment. Implementation details can be found in Appendix D.
We initialize four Gaussians and update their parameters to maximize J [(u, Σ) Ez [o(x)] on test function g(x) with four different methods: score- based policy gradient (using Equa- tion 1), a pathwise PG with the ground truth objective function. and two pathwise PGs using learned approximations, one accurate and one inaccurate (all using Equation 3). We visualize the returns and the path of the mean estimates Figure 3a. In addition, we zoom on the gradient paths of the score-based estimator. We visualize 100 different eight step paths from the middle of the trajectory. Here, in addition to the vanilla score-based estimator, we also show an importance sampling and a clipped importance sampling estimator. These paths visualized in Figure 3b.
The experiment shows that score-based gradient estimators have high variance, and can lead to unstable policies which fail to optimize the target. In addition, while importance sampling increases the sample efficiency of the algorithm, it greatly exacerbates these variance issues. We find that clipping the ratio estimate, as proposed by Schulman et al. (2017), prevents catastrophic instability, but does not reduce the variance substantially. On the other hand, using a pathwise gradients is remarkably stable and exhibits small variance. However, it either requires access the gradients of the objective function, or a strong surrogate model.
To pathwise gradients in on-policy learning, our goal is thus to learn suitable value function that allows us to estimate a low variance update direction without converging to a suboptimal solution.
3 RELATIVE ENTROPY PATHWISE POLICY OPTIMIZATION
Naively, we could take an off-policy algorithm like SAC and train solely with data from the current policy. However, Seo al. (2025) recently showed, this quickly lead to unstable learning.
4

---

Published as a conference paper at ICLR 2026
To succeed in the on-policy regime, need be able to continually obtain new diverse data, and compute stable and reliable updates. Combining a set of recent advances in both reinforcement learning well neural network value function fitting, can satisfy these requirements. We first introduce the core RI. algorithm, and then elaborate on the architectural design of the method.
At its core, REPPO proceeds similar to other on-policy actor-critic algorithms through three distinct phases: data gathering, value target estimation, and value and policy learning (see Algorithm 1). To obtain diverse data, REPPO uses a maximum-entropy formulation, adapted multi-step TD- (Subsection 3.1), to encourage exploration. Finally, to ensure that policies do not collapse and policy learning is stable, REPPO uses KL-constrained policy updates with a schedule that balances entropy-driven driven exploration and policy constraints (Subsection 3.2).
3.1 VALUE FUNCTION LEARNING
Off Off-policy PPG methods like TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018) mostly use single step Q learning, i.c. they use only immediate rewards for value function updates. This paired with large replay buffers to stabilize learning. While on-policy algorithms cannot past policy data, they can instead use low bias multi-step TD targets for stabilization (Fedus et al., 2020). Therefore, multi-step TD-À targets form the basis for our value learning objective. Note that REPPO is more closely related to SARSA than to Q-learning (Sutton & Barto, 2018), due to being on-policy.
In addition to multi-step returns, diverse data is crucial. To achieve a constant rate of exploration, and prevent the policy from prematurely collapsing to deterministic function, we leverage the maximum entropy formulation for RL (Ziebart et al., 2008; Levine, 2018). The core aim of the maximum entropy framework is to keep the policy sufficiently stochastic by solving a modified policy objective which not only maximizes rewards but also penalizes the loss of entropy in the policy distribution. The maximum-entropy policy objective (Levine, 2018 8) can be defined as
(ME(TT) = END
Σ 1=
(rr(Ena))) +HH[TT(
(4)
where H(TT(()) is the entropy of the policy evaluated at x, and a is hyperparameter which trades off reward maximization and entropy maximization. REPPO combines the maximum entropy ob- jective with TD-A estimates, resulting in the following target estimate
12
G (k (zt, at)
(r(xx, log n (uxl<xx)) + 7 nt (Tnn an)
(5)
h=
N Σ (x M=U
G²(z,a)=
(6)
EN
where the maximum length the future trajectory we obtain from the environment for the state- action pair (xr, a). Our implementation relies on the efficient backwards pass algorithm presented by Daley & Amato (2019). Crucially, the targets are computed on-policy after a new data batch is gathered, and the targets are not recomputed before gathering new data. Our Q learning loss is
CREPPO (0ll,,)1)) CB
1 Σ HT. [Qo(ar, a:), (1r, + Laxx (ols,,,, B
where of refers to the next state sample starting from I; and HT. is the HL.-Gauss loss (see Subsec- tion 3.3 and Subsection D.2), and Laux is presented in Subsection 3.3 and Subsection D.3.
Using purely on-policy targets allows us to remove several common off-policy stabilization compo- nents from the value learning setup. REPPO does not require pessimism bias, we can forgo the clipped double Q learning employed by many prior methods (Fujimoto et al., 2018). Tuning pes- simistic updates carefully allow for exploration is a difficult task (Moskovitz et al., 2021), this simplification increases the robustness our method. We also do not need a target value function copy, since do not recompute the target each step and therefore remains on-policy.
5

---

Published as a conference paper at ICLR 2026
3.2 POLICY LEARNING
A core problem with value-based on-policy optimization is controlling the size of the policy update, as the value estimate is only accurate on the data covered by the prior policy, A large policy update can therefore destabilize learning (Kakade & Langford, 2002). This problem has led to the develop- ment of constrained policy update schemes, where the updated policy prevented from deviating too much from the behavioral (Peters et al., 2010; Schulman et al., 2015). To control the deviation, (KI. divergence, also called the relative entropy (Peters et al., 2010), as it can be justified theoretically through information geometry (Kakade, 2001; Peters & Schaal, 2008; Pajarinen et al., 2019), and is easy to approximate using samples.
we use the Kullback-Leibb
Some works in the literature (Neumann, 2011; Sokota et al., 2022) claim that the reverse mode might preferable for policy constraints, as is mode-seeking, and the forward mode is mode- averaging. However, this intuition does not cleanly translate to setting. As our policies unimodal tanh-squashed Gaussian, the main impact of the KL direction is that the reverse- KL is entropy reducing. As we explicitly aim to increase the policy's ' entropy using the maximum entropy formulation, using forward-mode KL makes the optimization more stable.
Policy Optimization Objective Our policy updates derive from a constrained optimization prob- lem which includes both entropy and the KL constraint, and where o' is the behavior policy, and ēKI. and eœ the respective KL. and entropy constraints
max 0 Eawppgy
[Ba-to(112) (Q(x,a))
(8)
subject to Bar [DKL. (TT (-1x) || mo(+l>))) S EKI.
(9)
[H[[To (-x=]]] 2 EH.
(10)
similar combination of maximum entropy and KL divergence bound has been explored in vari- ous forms (Abdolmaleki al., 2015; Pajarinen et al., 2019; Akrour et al., 2019). However, while previous approaches use complex solutions to this problem, such as approximate mirror descent, line search, heuristic clipping, we take simpler approach. We relax the problem, which intro- duces two hyperparameters, a for the entropy, and / for the KL. Inspired by Haarnoja et al. (2019), REPPO automatically adapts these constraints when the policy violates them.
Policy Updates and Multiplier Tuning In the constrained objective, we introduce two hyper- parameters, EH and εKl, which bound the entropy and KL divergence. The goal of the Lagrangian parameters is to ensure that the policy stays close to these constraints. As we need to ensure that they remain positive, we update them log space with a gradient based root finding procedure
f "
Exmpprr [(H[To(-1x)) - Ex4]
(11)
B x 3 n E [(DKI, (To (-1x) [T (/oo-//)))))))))) (x:)) - EKI.))
(12)
Finally, to ensure our KL constraint is (approximately) maintained, we clip the actor loss based whether the constrained is currently violated. The full policy objective for REPPO is now
on
-Q(x; a) + e log TO (a(os)) C CREPPO (0|z.)
if & Σラ= log (ar(a)) < EKL
(13)
(a(a)))) e E/= log (o(ay)=))
otherwise
where is sampled from (-(a) and aj from the past behavior policy To (-ra))))). and l denotes how many samples are used to approximate the KL. As with the critic, the optimized loss is a mean over a minibatch from the rollout data. Note that contrary to other on-policy algorithms like PPO and TRPO, we are not forced to use actions sampled from the behavior policy in the policy gradient estimator, which removes the need for importance sampling correction. We will show that this greatly improves the performance of REPPO in Subsection 4.1.
Jointly tuning the entropy and KL multipliers is a crucial component of REPPO. As the policy entropy and KL are tied, letting the entropy of the behavior policy collapse means the KL constraint prevents any policy updates. Furthermore, the entropy and KL terms are balanced against the scale of the returns in the maximum entropy formulation. As the returns increase, keeping the multipliers fixed causes the model to ignore the constraints over time, accelerating collapse. However, as we tune both in tandem, our setup ensures steady, measured update rate of the policy.

---

Published as a conference paper at ICLR 2026
3.3 STABLE REPRESENTATION AND VALUE FUNCTION ARCHITECTURES
While the RL algorithm offers strong foundation obtain strong surrogate values, we also draw on recent off-policy advances in value function learning that improve training through architecture and loss design. We incorporate three major advancements into REPPO further stabilize training.
Cross-entropy loss for regression The first choice is to replace the mean squared error in the critic update with more robust cross-entropy based loss function. For this, REPPO uses the HL-Gauss loss (Farebrother et al., 2024). This technique was adapted from the distributional C5I algorithm (Bellemare et al., 2017), which can lead to stable learning algorithms. Inspired by this insight and histogram losses for regression (Imani & White, 2018), Farebrother et al. (2024) hypothesize that observed benefits are due to the fact that many distributional algorithms use a cross-entropy loss, which is scale invariant. Palenicek et al. (2025) further investigate and reinforce this claim, showing that stable gradients arise from cross-entropy based losses. We present the mathematical form of the loss formulation in Subsection D.2. We find that categorical loss is crucial addition, ablation experiments show (Subsection E. I), but alternatives like C51 could easily work as well.
Layer Normalization Several recent works (Ball et al., 2023; Yue et al., 2023; Lyle et al., 2024; Nauman et al., 2024a; Hussing et al., 2024; Gallici et al., 2024) have shown the importance of layer normalization (Ba et al., 2016) for stable critic learning. Gallici et al. (2024) provides thorough theoretical analysis of the importance of normalization in on-policy learning, while Hussing et al. (2024) focuses on assessing the empirical behavior of networks in off-policy learning with and without normalization. As we operate in an on on-policy regime where value function targets are more stable, we find that normalization not as critical for REPPO as it is for off-policy bootstrapped methods; yet, we still see performance benefits in most environments from normalization.
Auxiliary tasks Auxiliary tasks (Jaderberg et al., 2017 7) can stabilize features in environments with sparse rewards, where the lack of reward signal can prevent learning meaningful representations via the Q learning objective (Voelcker al., 2024a). For REPPO, auxiliary tasks are especially impactful when we decrease the number of samples used in each update batch (see Subsection E. E.1). We provide a discussion of this auxiliary task setup, including the loss function, in Subsection D.3.
EXPERIMENTAL EVALUATION
We begin by evaluating whether pathwise estimators improve upon score-based estimation in on- policy RL settings. We then compare our approach to baselines, evaluating final performance, sam- ple and wall-clock efficiency, and stability of policy improvement. Our results demonstrate strong performance of REPPO all axes. Additional details on architectures, hyperparameters, and abla- tions are provided in Subsection D.4 and Appendix E. A discrete variant of REPPO, along with its architectural changes and experimental results, is presented in Appendix C.
Environments We evaluate REPPO on two major GPU-parallelized benchmark suites: 23 tasks from the mujoco playground DMC suite (Zakka et al., 2025) and 8 ManiSkill environments (Tao et al., 2025), covering locomotion and manipulation, respectively. These tasks span high- dimensional control, sparse rewards, and chaotic dynamics.
Mean
Modian
IQM
REPPO (pathwise) REPPO (score-based.
PPO
REPPO (score-based GAE)
0.60
0.75
0.90
0.60
0.75
0.90
0.60
0.75
0.90
Normalized returns
Figure 4: Aggregate performance mujoco-ppaygreee We compare REPPO with two ablations: using the score-based gradient estimator with the learned Q function, and using an on-policy GAE estimate with importance sampling and clipping. For additional context, also report PPO results.
7

---

Published as a conference paper at ICLR 2026
Mean
Median
IQM
REPPO
PPO
PPO (Brax)
PPO (Brax, 200M)
FasITD3 (10,000k buf.)
F3 (32k huf.)
SAC (Brax, 5M)
DPO
RPO
0.6
0.7
0.8
0.45
0.60 0.75
0.90
0.60
0.75
0.90
Normalized returns
(a) Aggregate performance mujoco plcccoplaygrundd We compare REPPO to re-tuned PPO baseline, the Brax PPO and SAC implementations provided by Zakka et al. (2025), as well as FastTD3 (Seo et al., 2025), RPO (Rahman & Xue, 2023), and DPO (Lu al., 2022).
Mean
Median
IQM
REPPO PPO
PPO (100M)
0.75
0.80
0.85
0.90 0.80 Success Rate
0.85 0.90 0.95
0.85
0.90
0.95
1.00
(b) Aggregate success maniskill (Tao et al., 2025). We compare REPPO against a PPO baseline provided by Tao et al. (2025) at 50 million environment steps. As some environments take more than 50 million steps for PPO to achieve strong performance. report the final performance at 100 million steps. While the mean confidence intervals are very broad, REPPO performs strongly on the IQM and median metrics.
Figure 5: Aggregate performance comparison on (a) mujoco playground DMC and (b) ManiSkill3.
4.1 SCORE-BASED AND PATHIISE COMPARISON
REPPO offers alternative score-based policy gradient estimation in on-policy RL. However, we also introduce several enhancements, including automated tuning of entropy and KL coefficients, to improve value and policy learning. To assess the benefits of learned values and pathwise gradient estimation over score-based methods, we conduct two experiments. First, we replace the pathwise term (1 a) in Equation 13 with the score function log т (a|xx) (xx, asg. denoted as REPPO (score-based, Q). Second, we replace the gradient estimator with the GAE-based clipped objective from PPO, denoted as REPPO lscore-based, GAE). Aggregate results are presented in Figure
Using the approximate Q function in the policy gradient objective provides strong improvement over a clipped objective. This showcases the benefits of value function learning and removing im- portance sampling. This also shows that the REPPO framework can be used with policy classes that are not amenable to reparameterization, such as diffusion policies (Chi et al., 2024; Celik et al., 2025; Ma et al., 2025), by using bcore-based estimator together with the learned Q function. Inter- estingly, combining the PPO objective with REPPO leads to slightly worse results than vanilla PPO. We find that the high variance complicates the automatic parameter tuning scheme.
4.2 BENCHMARK COMPARISON
We compare REPPO against the PPO and SAC results reported by Zakka et al. (2025) and Tao et al. (2025). We report PPO basclines at 50M environment steps, and at the larger training horizon used in the original papers (Zakka et al., 2025). Results taken from Zakka et al. (2025) are denoted as "PPO/SAC (Brax)". To ensure that PPO not undertuned for the 50m step regime we re-tuned the hyperparameters of the implementation provided by Lu et al. (2022). SAC results are reported at 5m steps as this amounts to similar total runtime as the 200m PPO results (compare results in Zakka et al. (2025). Naively running SAC at a larger sample budget and wall-clock efficiency can lead to instability, as Seo et al. (2025) demonstrates. Furthermore, we include FastTD3 (Seo al., 2025) DMC locomotion tasks, trained under two memory budgets: the default replay buffer (10,485,760 transitions) and a constrained buffer similar in size to on-policy methods (32,768 transitions) to control for the the memory and performance trade-off. Finally, compare against
8

---

Published as a conference paper at ICLR 2026
DMC Comparison
Maniskill Comparison
1.0
1.0
REPPO
Normalized returss
0.8
0.5
Success Rafe
REPPO
0.2
0.2
PS (10,00
0.0
Environment steps
x10
Environment steps
x10'
Figure 6: Aggregate sample efficiency curves for the benchmark environments. Settings are identical to those Figure 5. REPPO achieves higher performance at a faster rate in both benchmarks.
Robust Policy Optimization (RPO) (Rahman & Xue, 2023) and Discovered Policy Optimization (DPO) (Lu al., 2022). However, even with some hyperparameter tuning, were unable to achieve a strong performance improvement beyond the PPO baseline with these approaches.
20 seeds. We 20 seeds for PPO and 5 for FasTTD3', reporting aggregate scores with 95% bootstrapped confidence intervals (Agarwal et al., 2021). To aggregate scores, mujoco mujoco_play playground returns are normalized by the maximum achieved by any algorithm. For ManiSkill we report raw success rates, which are comparable across tasks.
For REPPO, report results aggregated
Final Performance and Sample Efficiency We first investigate the performance of policies trained using REPPO. We report aggregate performance at the end of training on both benchmarks in Fig- ure For both benchmarks, we also provide the corresponding training curves in Figure
The aggregate results shown in Figure 5 and Figure 6 indicate that our proposed method achieves statistically significant performance improvements over PPO, well as similar performance to FasTTD3 despite REPPO being fully on-policy. Although these results most pronounced in locomotion tasks, ManiSkill manipulation results show significant performance benefits over PPO in terms of outlier-robust metrics (Chan al., 2020a; Agarwal et al., 2021).
We find that PPO struggles on sigh-dimensional tasks such as HumanoidRun. Morcover, despite its approximate trust-region updates, PPO suffers from performance drops and unstable training. This erratic behavior closely mirrors the score-based policy gradient instability shown Figure 3a. In contrast, REPPO exhibits more stable improvements and lower variance across seeds.
Wall-clock Time Wall-clock time is an important metric, as it reflects the practical utility of algo- rithm: faster training enables more efficient hyperpa- rameter search and experimentation. However, suring wall-clock time is nuanced, as results heavily depend on implementation details and are difficult to reproduce. We discuss these challenges across different frameworks in Appendix B. In Figure 7, compare the wall-clock performance of our ap- proach against PPO and SAC in JAX. Other base- lines lack JIT-compilable implementations, making direct comparisons less meaningful.
DMC Comparisen
Normalized renums
0.5 0.6
PPO
PPO (Btes, 2000)
S/C (Brax. 500)
Soconds
The computational cost per update is higher for REPPO than for PPO due to larger default networks and gradient propagation through the critic-actor chain. Nevertheless, both algorithms converge on most tasks in roughly 600-800 seconds, with REPPO achieving about 33% higher normalized returns.
Figure 7: Wall-clock time comparison of REPPO against PPO and SAC implementa- tions JAX. REPPO matches other algo- rithms' speed but achieves higher return.
This shows that the sample efficiency
We use fewer seeds for FastTD3 pytorch specific issues discussed in Appendix and because we smaller GPUs for our experiments.
unable to replicate the speed claimed in the paper. This is due

---

Published as a conference paper at ICLR 2026
DMC Comparison (Reliable success)
Maniskill Compurisen (Reliable success)
1.0
BULVO
1.0
150
2P0
0.8
PoaTDX
(1000004 Ner
0.1
0.5
04
of runs with
0.4
0.1
0.2
0.2
0.0
Emironment staps
Eavironment stops
x007
Figure 8: Fraction of runs that achieve reliable performance as measured by our metric for policy stability and reliability. REPPO's immediately starts achieving high performance in some runs and the number gradually increases indicating stable learning. PPO struggles to achieve high perfor- mance initially and to maintain high performance throughout training.
of pathwise gradients can offset their higher per-update cost, yielding improved wall-clock effi ciency compared to score-based PPO. In addition, find that jax-based SAC, which is tuned to trade sample for computational efficiency, slightly outperforms PPO, but does not match REPPO in performance. We note that other, modern SAC implementations (Nauman et al., 2024b; Lee al., 2025a;b), are able to achieve better performance, but at the cost of computational efficiency.
Reliable Policy Success We further investigate the stability of policy improvements using score- based and pathwise policy gradients. Our guiding principle is that such updates should not cause large drops in performance. To capture this, we adopt the "reliable success" metric, as proposed in Chan et al. (2020b). We define an algorithm as reliably performant if, once its performance exceeds fixed threshold т, never drops below this threshold thereafter. At each timestep, we track the number of runs that satisfy this criterion. This metric reflects the practical requirement that a deployed algorithm should not suddenly degrade simply due to continued training. We report the percentage of reliably successful runs for both REPPO and PPO in Figure 8.
On both DMC and ManiSkill benchmarks, REPPO achieves reliable performance improvements quickly. By the end of training, about four out of five runs have reached the threshold of = 0.9, whereas PPO achieves roughly 40 percentage points fewer reliably performant runs. We also find notable differences in sample efficiency: PPO requires 5-10 million interactions before most envs become reliably performant. Overall, these results show that, despite relying on a biased value model, pathwise policy gradients enable stable long-term improvement.
CONCLUSION AND AVENUES FOR FUTURE WORK
In this paper we present REPPO, highly performant yet efficient on-policy algorithm that leverages trained state- action value functions and pathwise policy gradients. By balancing entropic exploration and KL--contrrains, and incorporating recent advances in neural network value function learning, REPPO is able to learn a high-quality approximation sufficient for reliable gradient estimation. As a result, the algorithm outperforms PPO on two GPU-parallelizee benchmarks in terms of final return, sample efficiency and reliability while being on par in terms of wall-clock time. In addition, the algorithm does not require storing large amount of data making it competitive with recent advances in off- off-policy RL while requiring orders of magnitude lower amounts of memory.
As our method opens a new area for algorithmic development, it leaves open many exciting avenues for future work. As Seo et al. (2025) shows. using replay buffers can be beneficial to stabilize learning as well. This raises the question if our Q learning objective can be expanded to use both on- and poolcy data to maximize performance while minimizing memory requirements. Furthermore, the wide literature on improvements on PPO, such as learned objectives (Lu et al., 2022) can be incorporated into REPPO. We also observe that removing the importance sampling step in PPO has a crucial impact on performance, which suggests further research on the trade-off between efficiency and stability in on-policy gradient estimation is needed. Finally, better architectures such Nauman et al. (2024b), Lee et al. (2025a), Otto et al. (2021) might be transferable to our algorithm and the rich literature architectural improvements in off-policy RL can be tested in on-policy value learning.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
CV acknowledges funding through the Ontario Graduate Scholarship. AMF acknowledges the support of NSERC through the Discovery Grant program [2021-03701), Polytechnique Montréal's PIED program, and IVADO IAR* grant. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies spon- soring the Vector Institute. EE and MH's research was partially supported by the DARPA Triage Challenge under award HR001 12420305. Any opinions, findings, and conclusion or recommenda- tions expressed in this material are those of the authors and do not necessarily reflect the view of DARPA or the US government.
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plas- ticity in continual deep reinforcement learning. In Proceedings of the Conference on Lifelong Learning Agents, 2023.
Abbas Abdolmaleki, Rudolf Lioutikov, Jan Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann. Model-bas based relative entropy stochastic search. Advances in Neural Information Pro- cessing Systems, 2015.
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. In Proceedings of the International Conference on Learning Representations, 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Infor- mation Processing Systems, 2021.
Matthew Aitchison and Penny Sweetser. DNA: Proximal policy optimization with dual network architecture. In Advances in Neural Information Processing Systems, 2022.
Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann. Projections for approximate policy iteration algorithms. In Proceedings of the International Conference on Machine Learning. 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ArXiv, volume abs/ (0166607.65000 2016.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning. Springer, 1995.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In Proceedings of the International Conference Machine Learning, 2023.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. Proceedings of the International Conference on Machine Leurning, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations Python+NumPP programs, 2018. URL http:
/ //github. com/ jax-ml/jax
Onur Celik, Zechu Li, Denis Blessing, Ge Li. Daniel Palenicek. Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. DIME: Diffusion-based maximum entropy reinforcement learning. In Proceedings of the International Conference Machine Learning. 2025.
Stephanic Chan, Sam Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Mea- suring the reliability of reinforcement learning algorithms. In Proceedings of the International Conference Learning Representations, 2020a.
Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Measuring the reliability reinforcement learning algorithms. In Proceedings of the Interna- fional Conference on Learning Representations, 2020b.

---

Published as a conference paper at ICLR 2026
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint ariiv:1100007000000000 2019.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Proceed- ings of the 38th International Conference Machine Learning, 2021.
Brett Daley and Christopher Amato. Reconciling X-returns with experience replay. Advances in Neural Information Processing Systems, 2019.
Pierluea D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G. Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Proceedings of the International Conference on Learning Representations, 2023.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Ir- pan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Proceedings of the International Conference on Machine Learning, 2024.
William Fedus, Prajit Ramachandran. Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentalls of experience replay. In Proceedings of the International Conference Machine Learning, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation eтror in actor-critic methods. In Proceedings of the International Conference Machine Learning. 2018.
Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. In Proceedings of the International Con- ference on Learning Representations, 2024.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In Proceedings of the International Conference on Learning Representations, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the International Conference on Machine Learning, 2023.
Ignat Georgiev, Krishnan Srinivasan, Jie Xu, Eric Heiden, and Animesh Garg. Adaptive horizon actor-critic for policy learningin contact-rich differentiable simulation. In Proceedings of the International Conference Machine Learning. PMLR, 2024.
Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5, 2004.
Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In Proceedings of the International Conference Machine Learning, 2022.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference Machine Learning, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar. Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv: 1rXiv:18/2.05903. 2019.
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. In Proceedings of the International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2026
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for con- tinuous control. In Proceedings of the International Conference on Learning Representations, 2024.
Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massouue massoud Farahmand, and Eric Eaton. Dissecting deep RL with high update ratios: Combatting value divergence. In Reinforcement Learning Conference, 2024.
Andrew Hyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. closer look at deep policy gradients. In Proceedings of the International Conference Learning Representations, 2020.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In Proceedings of the International Conference Machine Learning. 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czameeki,, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In Proceedings of the International Conference on Learning Representations, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel- softbex. In Proceedings of the International Conference on Learning Representations, 2017.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In Proceedings of the International Confer- ence on Machine Learning. 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the International Conference on Machine Learning. 2002.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 2001.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620 998-987, 2023.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- under-parrameterizaioon inhibits data-efficient deep reinforcement learning. In Proceedings of the International Confer- ence on Learning Representations, 2021.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In Proceedings of the International Conference Learning Representations, 2025a.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspher- ical normalization for scalable deep reinforcement learning. In Proceedings of the International Conference Machine Learning, 2025b.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 1005..:180000000 00909, 2018
Jiajin Li, Baoxiang Wang, and Shengyu Zhang. Policy optimization with second-order advantage information. In Proceedings of the International Joint Conference on Artificial Intelligence, 2018.
Zechu Li, Tao Chen, Zhang- Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel q-learning: Scaling off- off-policy reinforcement learning under massively parallel simulation. In Proceedings of the International Conference on Machine Learning. 2023.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Pro- ceedings of the International Conference Learning Representations, 2016.
13

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Proceedings of the International Conference on Machine Learning. 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado Van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. In Proceedings of the Conference on Lifelong Learning Agents, 2024.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. Proceedings of the International Conference on Machine Learning, 2025.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relax- ation discrete random variables. Proceedings of the International Conference on Learning Representations, 2017.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 108. 10470, 2021.
Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano- Muñoz, Xinjie Yao, René Zurbrügg. Nikita Rudin, Lukasz Wawrzyniak, Milad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Akinola, Abrar Anwar, Mark T. Carlson, Ji Yuan Feng, Animesh Garg, Renato Gasoto, Lionel Gulich, Yijie Guo, M. Gussert, Alex Hansen, Mihir Kulkarni, Chenran Li, Wei Liu, Viktor Makoviychuk, Grzegorz Malczyk, Hammad Mazhar, Masoud Moghani, Adithyavairavan Murali, Michael Noseworthy, Alexander Poddubny, Nathan Ratliff, Welf Rehberg, Clemens Schwarke, Ritvik Singh, James Latham Smith. Bingjie Tang, Ruchik Thaker, Matthew Trepte, Karl Van Wyk, Fangzhou Yu, Alex Millane, Vikram Ra- masamy, Remo Steiner, Sangeeta Subramanian, Clemens Volk, CY Chen, Neel Jawale, Ash- win Varghese Kuruttukulam, Michael A. Lin, Ajay Mandlekar, Karsten Patzwaldt, John Welsh, Huihua Zhao, Fatima Anes, Jean-Francois Lafleche, Nicolas Moënne-Loccon Soowan Park, Rob Stepinski, Dirk Van Gelder, Chris Amevor, Jan Carius, Jumyung Chang, Anka He Chen, Pablo de Heras Ciechomski, Gilles Daviet, Mohammad Mohajerani, Julia von Muralt, Viktor Reut- skyy, Michael Sauter, Simon Schirm, Eric L. Shi, Pierre Terdiman, Kenny Vilella, Tobias Wid- mer, Gordon Yeoman, Tiffany Chen, Sergey Grizan, Cathy Li, Lotus Li, Connor Smith, Rafael Wiltz, Kostas Alexis, Yan Chang, David Chu, Linxi "Jim" Fan, Farbod Farshidian, Ankur Handa, Spencer Huang, Marco Hutter, Yashraj Narang, Soha Pouya, Shiwei Sheng, Yuke Zhu, Miles Macklin, Adam Moravanszky, Philipp Reist, Yunrong Guo, David Hoeller, and Gavriel State. Isaac lab: A acceleraed simulation framework for multi-modal robot learning. arXiv preprint 1eXiv:2511.04311 2025.
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No repre- sentation, no trust: Connecting representation, collapse, and trust issues in PPO. In Advances in Neural Information Processing Systems, 2024.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti- mation in machine learning. Journal of Machine Learning Research, 21(132); 2020.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In Proceedings of the International Con- ference on Machine Learning, 2021.
Ted Moskovitz, Jack Parker-Holdes Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 2021.
14

---

Published as a conference paper at ICLR 2026
Michal Nauman and Marek Cygan. Decoupled policy actor-critic: Bridging pessimism and risk awareness in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, 2025.
Michal Nauman, Michal Bortkiewicz, Piotr Mitoš, Tomasz Trzcinski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. In Proceedings of the International Conference on Machine Learning, 2024a.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Milos, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024b.
Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regularized, categorical: High-capacity value functions efficient multi-task learners. In The Thirty-minth Annual Conference on Neural Information Processing Systems, 2025. URL https: wpw.review.e..ec////ee net forum? zerummmid-zzuuuure
Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the International Conference on International Conference on Machine Learning, 2011.
Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Ma- hajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self- predictive RL. In Proceedings of the International Conference on Learning Representations, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. Proceedings of the International Conference on Machine Learning, 2022.
Chris Nota and Philip Thomas. the policy gradient gradient? In Proceedings of the Interna- tional Conference on Autonomous Agents and MulliAgent Systems, 2020.
Fabian Otto, Philipp Becker, Vien Anh Ngo, Hanna Carolin Maria Ziesche, and Gerhard Neumann. Differentiable trust region layers for deep reinforcement learning. In Proceedings of the Interna- tional Conference on Learning Representations, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 2022.
Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. Compatible natural gradient policy search. Machine Learning, 108(8), 2019.
Daniel Palenicek, Florian Vogt, Joe Watson, Ingmar Posner, and Jan Peters. Xqe: Well-conditioned optimization accelerates deep reinforcement learning. arXiv preprint 2rXiv:2509.25174 2025.
Matteo Papini, Giorgio Manganini, Alberto Maria Metelli, and Marcello Restelli. Policy gradient with active importance sampling. Reinforcement Learning Journal, 2:645-675, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Jan Peters and Stefan Schaal. Natural actor-critie. In Nearocomputing, volume 71. Elsevier, 2008
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the AAAI Conference Artificial Intelligence, 2010.
Martin L. Puterman. Markoy Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
15

---

Published as a conference paper at ICLR 2026
Iiija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(8)) 2024.
Md Masudur Rahman and Yexiang Xue. Robust policy optimization in deep reinforcement learning, 2023.
Nate Rahn, Pierluca D'Oro, Harley Wiltzer, Pierre-Luc Bacon, and Mare Bellemare. Policy op- timization in noisy neighborhood: On return landscapes in continuous control. Advances in Neural Information Processing Systems, 36:30618-30644 2023.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. Conference on robot learning, pp. 91-100. PMLR, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning- PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint (xXiv:250.110771 2025.
Max Schwarzer, Ankesh Anand, Rishab Goel, R.
Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self seffppredittive representations. In Proceedings of the International Conference on Learning Representations, 2021.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In Proceedings of the International Conference on Machine Learning. 2023.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505 22442 2025.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the International Conference Machine Learning. 2014.
Samuel Sokota, Ryan D'Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal sponse equilibria, and two-player zero-sum games. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Sanghyun Son, Laura Yu Zheng, Ryan Sullivan, Yi-Ling Qiao, and Ming Lin. Gradient informed proximal policy optimization. In Advances in Neural Information Processing Systems, 2023.
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W.
Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Riedmiller, and Matthew M. Botvinick. V-MPO: On-Policy Maximum Posteriori Policy Opti- mization for Discrete and Continuous Control. In Proceedings of the International conference on Learning Representations, 2019.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In Proceedings of the International Conference on Machine Learn- ing, 2022.
Richard S Sutton. Learning to predict by the methods of temporal differences. In Machine learning, volume 3. Springer, 1988.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, 2nd edition, 2018.
16

---

Published as a conference paper at ICLR 2026
Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-po MIT Press, 2016.
learning. In Journal of Machine Learning Research, volume 17.
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Ávila Pires, Yash Chan- dak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, and others. Understanding self- predictiiee learning for reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2023.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang. Roberto Ca- landra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Connectionisl Models Summer School, 1993.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. In Proceedings of the International Conference on Learning Representations, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099 090888, 2023.
Hado Van Hasselt. Double q-learning. In Advances in Neural Information Processing Systems, 2010.
Claas Voelcker, Marcel Hussing, and Eric Eaton. Can hop in general? discussion of benchmark selection and design using the hopper environment. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024a.
Claas Voelcker, Tyler Kastner, Igor Gilitschenski, and Amir- massoud Farahmand. When does self- prediction help? understanding auxiliary tasks in reinforcement learning. In Reinforcemeni Learn- ing Conference, 2024b.
Claas Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand, and Igor Gilitschenski. MAD-TD: Model-augmented data stabilizes high update ratio RL. In Proceedings of the Interna- tional Conference on Learning Representations, 2025.
Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in Artificial Intelligence, 2020.
Zhengpeng Xie, Qiang Zhang, Fan Yang. Marco Hutter, and Renjing Xu. Simple policy optimiza- tion. In Proceedings of the International Conference on Machine Learning, 2025.
Jie Xu, Miles Macklin, Viktor Makoviychuk, Yashraj Narang, Animesh Garg, Fabio Ramos, and Wojeiech Matusik. Accelerated policy learning with parallel differentiable simulation. In Pro- ceedings of the International Conference on Learning Representations, 2022.
Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and bet- ter resolving q-value divergence in off In Advances in Neural Information Processing Systems, 2023.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazaa, Yuval Tassa, and Pieter Abbeel. MuJoCo playground: An open-source framework for GPU-accelerated robot learn- ing and sim-to-real transfer., 2025. URI. https: / thub. com/ googi e- deepm nd/
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence, 2008.
17

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Stabilizing On-Policy RL A fundamental issue with score-based approaches is their instability. Therefore, various improvements to decrease gradient variance have been considered. Some works have noted the difficulty of representation learning and have addressed this via decoupling the train- ing of value and policy (Cobbe et al., 2021; Aitchison & Sweetser, 2022). Moalla et al. (2024) note that feature learning problems can result from representation collapse, which can be mitigated using auxiliary losses. There are also efforts to reduce the variance of gradients, e.g. by finding a policy that minimizes the variance of the importance sampling factor (Papini et al., 2024) or modifying the loss to ensure tighter total variational distance constraints (Xie et al., 2025).
Incorporating ground-truth gradient signal to stabilize training has also been studied, both for dy- namical systems (Son et al., 2023) and differentiable robotics simulation (Mora et al., 2021; Xu et al., 2022; Georgiev et al., 2024). However, access to a ground-truth gradient requires custom sim- ulators, and in contact-rich tasks, approximate models can provide smoother gradients (Suh et al., 2022).
Trust regions and constrained policy optimization Other approaches have used similar KL and trust region constraint as REPPO. Schulman et al. (2015) and Peters et al. (2010) formulate the KL constrained policy update a constrained optimization problem. Peters et al. (2010) shows closed form solution to this problem, while Schulman et al. (2015) uses a conjugate gradient scheme to solve the relaxed optimization problem. Schulman et al. (2017) replaces the Lagrangian formulation with a clipping heuristic. However, clipping can lead to wrong gradient estimates (Ilyas et al., 2020) and in some scenarios the clipping objective fails to bound the policy deviation (Wang et al., 2020). Akrour et al. (2019) propose to project the policy onto the trust-region to sidestep the difficulty associated with clipping. We find that our approach is simpler to implement and more general, as we do not assume direct projection is possible.
Otto et al. (2021) propose o replace the various trust-region enforcement methods such as line- search or clipping with differentiable trust-region layers in the policy neural network architecture. While our method slightly more general, as we make no assumption on the form of the policy (aside from assuming gradient propagation through the sampling process is possible), trust-region layers could easily be combined with REPPO for appropriate policy parameterizations.
Work on GPU-parallelized On-policy RL With the parallelization of many benchmarks on GPUs (Makoviychuk et al., 2021; Zakka et al., 2025; Tao et al., 2025), massively-parall on-policy RL has become quite popular. While these environments provide simulation testbeds, algorithms trained in such environments have shown to transfer to real-robots, allowing us to train them in minutes rather than days (Rudin et al., 2022).
Hybridizing Off-policy and On-policy RL methods Most closely to our work, Parallel Q Net- works (PQN) (Gallici et al., 2024) was established by using standard discrete action-space off policy techniques in the MPS setting. While our work shares several important features with this method, find that our additional insights on KL regularization and tuning is crucial for adapting the con- cept to continuous action spaces. We also evaluate our approach on discrete action spaces (see Appendix C). While PQN performs slightly better, likely owing to tuned exploration techniques, we show that our method works robustly across both discrete and continuous action spaces.
Other methods, such as Parallel Q-Learning (Li al., 2023) and FasITD3 (Seo al., 2025) also attempt to use deterministic policy gradient algorithms in the MPS setting, but still remain off- tfflcollcy. This has two major drawbacks compared our work. The methods require very large replay buffers, which can either limit the speed if data needs to be stored in regular CPU memory, require very large and expensive GPUs. In addition, the off-policy nature of these methods requires stabilizing techniques such as clipped double Q learning, which has been shown to prevent exploration.
KL-based RL Finally, other works also build on top of the relative entropy policy search (Peters et al., 2010). Maximum A Posteriori Policy Optimization (MPO) (Abdolmaleki et al., 2018) and Variational MPO (Song et al., 2019) both leverage SAC style maximum entropy objectives and use KL constraints to prevent policy divergence. However, both methods use off off-po data together with importance sampling, which we forgo, do not tune the KL and entropy parameters, and crucially do not make use of the deterministic policy gradient.
18

---

Published as a conference paper at ICLR 2026
Going beyond relative entropy, the KL-based constraint formulation has been generalized to include the class of mirror descent algorithms (Grudzien et al., 2022; Tomar et al., 2022). In addition, Lu et al. (2022) meta-learns constraint to automatically discover novel RL algorithms. These advancements are largely orthogonal to our work and can be incorporated into REPPO in the future.
Instability in Off-policy RL Our method furthermore adapts many design decisions from recent off-policy literature. Among these are layer normalizations, which have been studied by Nauman al. (2024a); Hussing et al. (2024); Nauman et al. (2024b); Gallici al. (2024), auxiliary tasks (Jaderberg et al., 2017; Schwarzer et al., 2021; 2023; Tang et al., 2023; Voelcker et al., 2024b; Ni al., 2024), and HL-Gauss (Farebrother al., 2024), variants of which have been used by Hafner et al. (2021); Hansen et al. (2024); Voelcker et al. (2025). Beyond these, there are several other works which investigate architectures for stable off-policy value learning, such as Nauman et al. (2024b); Lee et al. (2025a;b). A similar method to our KI. regularization tuning objective has been used by (Nauman & Cygan, 2025) to build an exploratory optimistic actor. While the technique is very similar, we employ it in the context of the trust-region update, and show the importance of jointly tuning the entropy and KL parameters. Finally, there are several papers which investigate the impact of continual learning in off-policy reinforcement learning, including issues such as out-of- distribution misgeneralization (Voelcker et al., 2025), plasticity loss (Nikishin et al., 2022; D'Oro et al., 2023; Lyle al., 2023; Abbas al., 2023). Since many of these works focus specifically on improving issues inherent in the off-policy setting, we did not evaluate all of these changes in REPPO. However, rigorously evaluating what network architectures and stabilization methods can help to further improve the online regime is an exciting avenue for future work.
B WALLCLOCK MEASUREMENT CONSIDERATIONS
Measuring wall-clock time has become a popular way of highlighting the practical utility of an algorithm as it allows to quickly deploy new models and iterate ideas. Rigorous wall-clock time measurement is a difficult topic, as many factors impact the wall-clock time of an algorithm.
We chose to not compare the jax and torch versions head-to-head differences on different hardware, and the different compilation philosophies lead to different ben- efits and drawbacks. For example, jax full jit-compilation trades a much larger initial overhead for significantly faster execution, which can amortize itself depending on the number of timesteps taken. This is the reason why we do not include FastTD3 in Figure 7, as only Py PyTorch implementation of the algorithm exists. FastTD3 and REPPO use similar algorithms and hyperparameters, therefore, barring complexities like those discussed below, we expect them perform at similar speeds.
found significant runtime
More importantly, torch's 's compilation libraries are built to accelerate standard supervised and gen- erative workflows, but do not support RL primitives equally well. As the CPU needs load ker- nels during training which the GPU then executes, the CPU plays much larger role in the speed measurements of the torch-based variant of REPPO. Especially the tanh-squashed log probability computation and the frequent resampling from the action space cannot be offloaded into an efficient kernel without providing one manually, which we have not done. This is likely due to the fact that torch keeps its random seed the CPU. This is not concern for jax, due to the fact that all kernels are statically compiled when the program is first executed, and random seeds are handled explicilly as part of the program state. Therefore, the CPU is under much lower load.
Instead of raw wall-clock time measurements, which can vary massively across framework and hardware, we recommend that the community treat the question of wall-clock time more carefully. While the actual time for an experiment can be of massive importance from a practical point of view, the advantages and limitations of current frameworks can obscure exciting directions for future work. For example REPPO is highly competitive with PPO when implemented in jax. but struggles somewhat in torch due to framework specific design choices.
C DISCRETE REPPO (D-REPPO)
One of the major advantages of PPO in the zoo of RL algorithms is the fact that it can be used in both continuous and discrete action settings. However, as we build on the DDPG/TD3/SAC line of work, the exposition of our algorithm has focused on the continuous setting alone.
19

---

Published as a conference paper at ICLR 2026
Nonetheless, it is easy to adapt approach to the discrete action setting as well. Following the proposal of Christodoulou (2019), we can circumvent the chained critic-actor gradient and compute the value of the current policy, the entropy, and the KL bound in closed form
B
M
D- AD-REPPO (0]
To(ayla;) (Q(#z, az) + el log Ta (a(kkz)))
(14)
n, SKL
B
B
(D-REPPO(0)B)) DD-REPPO (0)
To (2) log To (aj [ii) To To (21)
(15)
B
1=1
==1
CD-REPPO REPPO (0[B), if E log aer(aylas) 1, D-RPEPO TIS.I.
CD REPPE R
molay|zz) < EKL (16)
(0]B), otherwise.
This variant of our algorithm still directly differentiates the full Q function objective, so can still be seen as pathwise implementation. But computing the expectation in closed form circumvents the necessity to use biased estimator for discrete sampling, such as the Gumbel-Softna trick (Maddison al., 2017; Jang et al., 2017; Fujimoto et al., 2024).
To investigate the benefits of our approach in the discrete action setting, we compare it against PQN (Gallici et al., 2024) and PPO. The main benefit of our approach over PQN is that a) a general algorithm that unifies both discrete and continuous action spaces, due to the underlying actor critic architecture, and b) that the principled entropy and KL objectives stabilize updates and encourages continuing exploration without epsilon greedy exploration strategy.
We find that our algorithm able to perform roughly on-par with PQN in the Atari 10 suite of games (ef. Table 1 and Figure 9) with only minor changes to the architecture to adapt the Atari games benchmark. Notably, suitable settings for the KL and entropy target remain consistent even for the discrete action setting. We only find that the value of X 0.65 that is also recommended by Gallici et al. (2024) is superior to our default value 0.95, likely due to the higher variance of the return in the atari games. While the high variance across Atari games makes drawing clear conclusion difficult, we find that PQN seems to achieve slightly better performance. We find that this is most likely due to the fact that the algorithm adds explicit exploration noise, while we rely the entropy and conservative KL terms to pace policy improvement.
Table 1: Aggregated Human-Normalizze Atari- 10 scores with 95% confidence intervals.
Algorithm
Mean [CI]
Median [CI]
IQM [CI]
REPPO
2.98 [2.64, 3.33]
1.68 [1.48, 1.82]
1.64 [1.54, 1,74]
PQN 3.35 [3.00, 3.76]
1.58 [1.48, 1.71]
1.64 [1.58, 1.71]
D IMPLEMENTATION DETAILS AND HYPERPARAMETERS
In the following, we present implementation details on experiments, as well as a hyperparameter overview.
D.1 TOY EXAMPLE
To obtain the gradient descent comparison in Subsection 2.2 we used the 6-hump camel function, standard benchmark in optimization. As our goal was not to show the difficulties of learning with multiple optima, which affect any gradient-based optimization procedure, but rather smoothness of convergence, we initialized all runs close to the global minimum. The surrogate functions were small three layer, 16 unit MLPs. To obtain a strong and weak version, we used differing numbers of samples, visualized in Figure 10. Every algorithm was trained with five samples from the policy at every iteration. Finally, we tested several learning rates. We chose learning rate which allows the ground- -ruth pathwise gradient to learn reliably. a smaller gradient step size is chose, the Monte- Carlo estimator converges more reliably, at the cost of significant additional computation. We also
20

---

Published as a conference paper at ICLR 2026
Amidar-v5
BattleZone-v5
Bowling-v5
1500
1000
10.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
100
x110f
x10
DoublcDunk-v5
Frostbitc-vS5
KungFuMaster-vS
Retum (Normalizd))
100
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
x100
NameThisSimme-v5
Phoenix-v5
Qbert-v5
20000
10000
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
X10
Riverraidvv5
20000
0.0
0.5
1.0
1.5
2.0
Timestops
Algarithms
REPPO
PQN
Figure 9: Per-enviromment results on the Atari 10 suite
tested subtracting running average mean as a control variate from the Monte-Carlo estimate. While this reduced variance significantly, it still very casy to destabilize the algorithm by choosing larger step size or less data samples.
In total, our experiments further highlight well known fact in gradient-based optimization: while MC-based gradient algorithm can be tuned for strong performance, it is often extremely dependent on finding very good set of hyperparameters. In contrast, pathwise estimators to work much more reliably across a wider range of hyperparameters, which corroborates our insights on REPPO hyperparameters robustly transfering across environemnts and benchmark suites.
D.2 HI.-GAUSS EQUATIONS
Given regression target y and function approximation f(x), HL-Gauss transforms the regression problem into a cross-entropy minimization. The regression target is reparameterized into a histogram approximation hist of (3 σ), with fixed chosen heuristically. The number of histogram bins and minimum and maximum values are hyperparameters. Let hist(y); be the probability value of the histogram at the i-th bucket. The function approximation has an h-dimensioonll output vector of
21

---

Published as a conference paper at ICLR 2026
Figure 10: Samples used to train the surrogate function. On the left, we visualize the 32 sample dataset train the weak surrogate function, in the middle the 1024 datapoints train the strong, and on the right the full objective function.
logits. Then the loss function is
exp f(x);
HL(S(z),v))
Σ hist(y); log Σ exp fxx)
The continuous prediction can be recovered by evaluating
ý [[hist(((=))] (fis) vee(min, max, h)),
where vec(min, max, h) is a vector with the center values of each bin ranging from min to max.
D.3 AUXILIARY TASK SETUP
simple yet impactful auxiliary task latent self prediction (Schwarzer et al., 2021; Voelcker al., 2024b; Fujimoto et al., 2024). In its simplest form, latent self- eeffprreiittion is computed by separating the critic into an encoder ø : X' x
Z and prediction head fe Z + R. The full critic can then be computed as Q(z, a) (ø(zz, [z, a)). A self-predictiv auxiliary loss adds a forward predictive model fp + Z and trains the encoder and forward model jointly to minimize
Laux (It, ags It+1, a1-1) = (f(( a4) ((
(17)
As our whole training is on-policy, do not separate encoder into state-dependent and action dependent part as many prior off-policy works have done. Instead we compute the targets on on-policy with the behavioral policy and minimize the auxiliary loss jointly with the critic loss.
Overall, the impact of the auxiliary task is the most varied across different environments. In some, crucial for learning, while having detrimental effect in others. We conjecture that the additional learning objective helps retain information in the critic the reward signal is not informative. In cases where the reward signal is sufficient and the policy gradient direction is easy estimate, additional training objectives might hurt performance. We encourage practitioners to investigate whether their specific application domain and task benefits from the auxiliary loss.
D.4 REPPO MAIN EXPERIMENTS
In addition to the details laid out in the main paper, we briefly introduce the architecture and addi- tional design decisions, as well as default hyperparameter settings.
The architecture for both critic encoder and heads, as well the actor, consists of several nor- malized linear layer blocks. As the activation function, we use silu/swift. As the optimizer, we use Adam. We experimented with weight decay and learning rate schedules, but found them to be harmful to performance. Hyperparameters are summarized in Table 2. We tune the discount factor y and the minimum and maximum values for the HL-Gauss representation automatically for each environment, similar to previous work (Hansen et al., 2024). This makes the hyperparameters, to- gether with the algorithm description, and the source code, complete algorithm specification in
22

---

Published as a conference paper at ICLR 2026
Critic Architecture
critic hidden dim vmin
512
Environment
min r
total time steps
50, 000, 000
envs n steps
1024
vmmax
max
128
num HL-Gauss bins
151
KLaar Optimization n epochs mini batches batch size Ir maximum grad norm Problem Discount
0.1
num critic encoder layers
2
num critic head layers
2
num critic pred layers
2
64
Actor Architecture
envs X n steps batches
2048
actor hidden dim
512
36 4
num actor layers
3
0.5
RL Loss
8 start
0.01
EKI.
0.1
max env steps 0.95
a start
0.01
EX
0.5 x dimA4
aux loss mult
1.0
Table 2: Default REPPO hyperparameters
the sense of Jordan al. (2020), we only vary hyperparameters across environments following simple equations on clear, domain sepcific hyperparameters such as the size of the action space and the length of the experiment.
For all environments, we use observation normalization statistics computed as simple running average of mean and standard deviation. We found this be important for performance, similar as in other on policy algorithms. Since we do not hold data in replay buffer, we do not need to account for environment normalization specialized manner, and can simply use an environment wrapper.
For more exact details on the architecture we refer interested readers to the codebase.
E ADDITIONAL RESULTS
In the following, we provide additional results and further clarification existing experiments in Section 4.
E.1 DESIGN ABLATIONS
We run ablation experiments investigating the impact of the design components used in REPPO. In these experiments, we remove the cross-entropy loss via HL-Gauss, layer normalization, the auxiliary self-predictive loss, or the KL regularization of the policy updates. To understand the importance of each component for on-policy learning we conduct these ablations for two scales of batch sizes the default 131, 072 on-policy transitions, as well as the smaller batch size of 32, 768.
As shown in Figure 11, our results indicate that both the KL regularization of the policy updates and the categorical Q-learning via HL-Gauss are necessary to achieve strong performance independent of the size of the on-policy data used update our model. We find that the KI. divergence is the only component that, when removed, leads to a decrease in performance below the levels of PPO, which clarifies the central importance of relative entropy regularization for REPPO. Removing normalization has minor negative effects on performance which become worse at smaller buffer sizes. This is consistent with the literature on layer normalization in RL. Similarly, the auxiliary -elf-predietive loss has more clearly negative impact on performance when the batch size becomes smaller. We note that auxiliary loss has an inconsistent impact on the training generally, where is strongly beneficial in some environments, but harmful in others.
23

---

Published as a conference paper at ICLR 2026
Mean
Median
IQM
REPPO
REPPO (no aux loss)-
REPPO (no normalization)
REPPO H
REPPO (no KL) REPPO, reverse KL- REPPO. no KL clip
0.6
0.7
0.8
0.60
0.75
0.90
0.60
0.75
0.90
Normalized returns
(a) Large dataset size ablation (128 x 1024).
Mcan
Median
IQM
REPPO
REPPO (32k buf.)
REPPO (32k buf.. no aux loss)
REPPO (32k buf., normalization) REPPO (32k buf., no Hl-Gaass) REPPO (32k buf., KL)
0.60
0.75
0.4
0.6
0.8
1.0
0.45
0.60
0.75
0.90
Normalized returns
(b) Small dataset size ablation (32 x 1024).
Figure 11: Ablation on components and data size on the DMC benehmark. Both values are signifi- cantly smaller than the replay buffer sizes used in standard off-policy RL algorithms like SAC and FasTTD3. The HL-Gauss loss and KL regularization provide a clear benefit at both data scales. The normalization and auxiliary loss become more important when less data is available, highlighting that some stability problems can also be overcome with scaling data.
E.2 MEMORY DEMANDS AND DATA SCALING
Recent advances in off- pflicy algorithms have shown great performance when large buffer sizes are available (Seo et al., 2025). When dealing with complex observations such images, on-policy algorithms which do not require storing past data have large advantage. In terms of data storage requirements, our algorithm comparable with PPO, yet it remains to answer how well REPPO compares to algorithms that are allowed to store a large amount of data. For this, we compare against the recent FastTD3 (Seo et al., 2025) which also uses GPU-parallelized environments but operates off off-policy. We compare REPPO against the original FasITD3 and we also re-run FasiTD3 with access to a significantly smaller buffer equivalent to the REPPO buffer. We report the results in Figure 12b.
The results demonstrate that REPPO is on par or better in terms of performance on mean and IQM with the FastTD33 approach. This is despite the fact that REPPO uses buffer that is two three orders of magnitude smaller. When decreasing the buffer size of FastTDD, the algorithm's 's perfor- mance drops by a large margin while REPPO barely affected by a smaller buffer. We find that Fas(TD3 with a smaller buffer can retain performance on lower dimensional, easier tasks but suffers on harder tasks that may be of greater interest in practice. In summary, REPPO is competetive with recent advances in off off-colicy learning with sigmificantly lower memory and storage requirements.
These results raise the question whether positive scaling with replay buffer size is general feature of on-policy algorithms. In our default configuration, REPPO uses long rollouts and high number of parallel environments, as well as a large number of policy and value function update steps. PPO on the other hand is often implemented with smaller dataset sizes. We therefore set up REPPO and PPO training runs across 4 datasets, varying the rollout length. To keep the total number of gradient steps and the minibatch size the same, we reduced the number of minibatches proportionally to the batch size. The settings are summarized in Figure 12a. Note that in the large settings, the data becomes more off-policy. Both PPO and REPPO have explicit ways to deal with this, clipping and the KL minimization term respectively, but the clipping term in PPO is only a heuristic to prevent large importance sampling ratios.
24

---

Published as a conference paper at ICLR 2026
Num envs
Num steps
Num minibatches
Epochs
Updates per batch
130k buffer
1024
128
64
8
512
65k buffer
1024
32
16
8
128
32k buffer
1024
4
8
32
16k buffer
256
8
(a) Dataset configurations for the data scaling experiment.
Mean
Median
IQM
REPPO (130k buf.)
FasTTD3 (10,000k buf.)
REPPO (6Sk buf.)
FasTTD3 (32k buf.)
0.64
0.72
0.80
0.7
0.8
0.9
1.0
0.7
0.8
0.9
Normalized returns
(b) Comparison of aggregate performance between REPPO and FasiTD3, REPPO is competitive with the large buffer FasTTD3 version and outperforms Fas(TD3 when memory is limited.
Mean
Median
IQM
REPPO (130k buf.)
REPPO (65k buf.)
REPPO (32k buf.)
REPPO (16k buf.)
PPO (130k buf.)
PPO (65k buf.)
PPO (32k buf.)
PPO (16k buf.)
0.45
0.60
0.75
0.4
0.6
0.8
0.4
0.6
0.8
1.0
Normalized returns
(c) Aggregated performance of REPPO and PPO under different batch dataset sizes. The mean performance of REPPO drops monotonically with decreasing batch size, while PPO shows its highest performance with medium and small dataset size.
Figure 12: Experiment to compare the impact of batch datset size on different on-policy algorithms.
25

---

Published as a conference paper at ICLR 2026
Issac- Volocity- Fla---tt--
Isae Velocity- Rough-HI-st
Sioa-VVlocity-Flae Flatiiy---a-------
Velocity- Wew.city-Ruug---OOO- -Rogh--G----
10
30
50
30
40
Return
20
30
20
20
20
10
10
0
2
X10
x10
X 10
Timesteps
Algorithms
REPPO
PPO (esl
FisITD3
Figure 13: Result overview of REPPO on the Isaacl Lab locomotion tasks
Comparing the performance both approaches (see Figure 12c), we observe clear pattern. The mean performance of REPPO drops steeply with decreasing dataset size. PPO on the other hand does best in the medium and small dataset regimes. This highlights the different mechanisms on which both algorithms operate. Larger datasets allow the trained Q function to generalize better, similar to the insight presented in Figure 3a. On the other hand, for PPO the dataset size needs to be large enough to allow for stable gradient estimationn, but not large that too many gradient update steps are necessary. This because clipping can prevent further learning, and many update steps can exacerbate varaince issues with importance sampling.
Note that at some point, REPPO will likely also stop improving with larger datasets and more gradient update steps. We that the performance differences between the medium and the large dataset at not as strong as with smaller datasets. REPPO cannot continue to learn on fixed data forever, by design, as the KL divergence between two consecutive policies constrained. However, we can hypothesize based on the empirical evidence that REPPO is able to scale more gracefully with large amounts of data.
Overall, we can conclude that REPPO scales favorably with larger data buffers, similar to SAC or FasTTD3, but is not strongly reliant on
E.3 ISAACLAB RESULTS
To provide additional value to the robotics community, who often favor specific implementations and APIs for reinforcement learning algorithms, we provide a reimplementation of REPPO in RSL- RL Schwarke et al. (2025). We ran this implementation of REPPO and compare the results the humanoid locomotion environments in the Isaaclab suite Mittal et al. (2025). Results are presented in Figure 13. The FastTD3 results provided by Seo et al. (2025) are not all ran for the full 250 million timesteps.
For IsaacLab, found that the KL constraint needs be set more tightly 0.01, which is the same value used for PPO.
E.4 PER ENVIRONMENT SAMPLE EFFICIENCY CURVES
Finally, we provide sample efficiency curves per environment Figure 14, Figure 15, and Figure 16.
26

---

Published as a conference paper at ICLR 2026
PiokSingllYCB-.1
PoglnsentionSile-1
Uniteeoi ot
1.00
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
x10
100
x10
Unitneel' AlazreeeiPaazzAplllleeeee Polce
UpiighgUii
PokeCaihe-vi
0.8
1.00
1.00
Success Rate
0.6
0.75
0.75
0.50
0.50
0.25
0.25
0
2
2
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
X10
x10'
Timesteps
Algorithms
Figure 14: Per-environment results the ManiSkill suite
27

---

Published as a conference paper at ICLR 2026
AcroborSwingup
AccobotSeingupSpusee
BallinCup
750
750
750
500
500
500
0
x10
CartpoleBallnce
Cutpoll-BalleeeSuue
CurtpoleSwingup
1000
1000
1000
750
750
750
Return
X10
x10?
X10
CartpolsSwingppppure
ChoctallRun
FingerSpin
1000
1000
1000
750
750
250
250
250
x10
1000
x10
TingerTurnEasy
HaraHad
FishSwim
1000
1000
1000
500
500
500
250
250
2
2
x10
x100'
x10
Timesteps
Algorithms
REPPO
PPO
PPO (Brax)
FasTD3 (10.000% buf.)
Figure 15: Per-environment results the mujoco playground DMC suite
28

---

Published as a conference paper at ICLR 2026
HoppetHlop
HopperStand
PendalumSwing wwingup
750
750
750
500
500
500
0
x1007
ReacherEasy
Reacherllard
WalketRun
1000
1000
1000
750
750
750
Retum
X10*
x10
X10
WalkerWall
WallserStand
HumanoidSSand
1000
1000
1000
750
250
250
250
x10
x10
110
WumannidWallk
HumanoidRun
1000
1000
500
500
250
0
2
2
x10
Timesteps
Algorithms
PPO
PPO (Brax)
FarTDD (10.000k bef.)
REPPO
Figure 16: Per-environment results the mujoco playground DMC suite
29

---

Published as a conference paper at ICLR 2026
F PSEUDOCODE
Algorithm 1: Pseudocode for Relative Entropy Pathwise Policy Optimization
Input:
Environment &, actor network Te, critic network Qs. hyperparameters Output: Trained policy x
// Initialize networks
Actor To. behavior policy Ty with 0' = 0, critic Qs with encoder fa. entropy and KI. temperature and
for iteration = to Naterations do Collect rollout with behavior policy for step 1 to Nateps do
// Step
Apply exploration noise scaling
Sample action a, (=((-)2z))
Execute a, environment, observe (1++1, Tt.
Compute approximate V(+1 + Qo(T (441 with 441 ~ (1(44)
Compute 1/ < (6(Ix+1.4++))
// Maximum entropy augmented reward, see Subsection 3.1 f F2 a log (0(
Store transition (Iz; 0t, Pt, 2411, di, V1+1,
end
// Step 2: Compute TD-À targets, see Subsection 3.1
for t T - down to 0 do
G t f. + >((1 deG + (1 X)V++))
end
// Step
Update networks for multiple epochs
for epoch = 1 to Nepoche do
Shuffle data and create mini-batches
for each mini-batch = {(x, a, a, G*, v) do
/1 Categorical critic update, see Subsection 3.3 L B Σ CnossFinropyyQaa(i az), Cat(G?))
11 Auxiliary task, see Subsection 3.3 Lauxx E
Update critic: ø o aq VolLe + (Loux)
// Actor update with entropy and kL regularization, see Subsection 3.1 and Subsection 3.2
Sample action af Te(+|z;)
Sample actions 0; Tor (+/24))
Compute KL divergence: (RL(E;) + E log (3,1a
Policy loss: L. + Qo(r,, at) e" log To (a(a((z)) - e (II(
(Altematively, compute clipped objective)
Update actor: <- 0 + 1/x VoLu
Entropy update: +- a Ja V Vaca Σ H(mo (x:)] cxr)
KL 9 update: 3 +- 3 1)a V 3
B Σ DxI(x:)) SKI.)
end end
Behavior Policy Update + 8
end
return Trained policy me
30


Reviews:
[{"review_id": "iu6LDr3MkZ", "reviewer": "Reviewer_EJRD", "summary": "The paper proposed utilising the pathwise gradient estimator with an accurately learned surrogate value function to construct more efficient on-policy algorithms. The approach is empirically evaluated across multiple datasets to demonstrate its effectiveness.", "questions": "Minors: the meaning of $x’_i$ in equation (6)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- Figure 2 shows Illustrative examples of different estimators (policy gradient estimator vs. Different pathwise gradient estimators).\n- Experiments contain various benchmarks (DMC suite and ManiSkill environment) and baselines.\n- The efficiency of the algorithm is shown in different settings: number of samples to reach a certain performance level, memory, and wall-clock time.", "weaknesses": "See the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MbOgLNs7Nt", "reviewer": "Reviewer_nRqE", "summary": "The authors are motivated by the goal of formulating a modern on-policy reinforcement learning algorithm that dispenses with the large replay buffers characteristic of many state-of-the-art off-policy methods, thereby reducing the associated memory footprint and data management overhead. To this end, they propose Relative Entropy Pathwise Policy Optimization (REPPO), an on-policy actor-critic algorithm that builds upon the maximum entropy framework and utilizes the pathwise policy gradient, similar to Soft Actor-Critic (SAC).\n\nThe core of their contribution lies in adapting these off-policy components to the more challenging (wrt value-function learning) on-policy setting. In particular, they replace the standard single-step TD targets with multi-step TD(λ) value targets. This is a critical choice, as it aims to reduce the bias in value estimates that is particularly severe in the on-policy regime due to the high temporal correlation of the training data. Furthermore, to ensure stable policy updates, the authors employ a KL-regularized trust region, dynamically tuning the Lagrange multipliers using a simultaneous gradient-based approach inspired by SAC. Finally, the algorithm incorporates several recent advances in value function learning to further enhance stability, including the use of a categorical Q-function representation, layer normalization, and auxiliary tasks. The authors provide a thorough empirical evaluation comparing REPPO against several state-of-the-art algorithms, reporting significant performance improvements in terms of final performance and stability.", "questions": "*   A minor but interesting point from Figure 2 is that the pathwise gradient with the ground truth objective appears to converge more slowly than with the strong surrogate. Could the authors elaborate on the potential reasons for this phenomenon? \n*   Regarding the policy update, the authors supplement the automatic multiplier tuning with a heuristic clipping of the actor loss (Equation 12). It would be valuable for the authors to comment on the empirical importance of this clipping mechanism. How does it contribute to the algorithm's overall stability and performance?\n*  I'd suggest improving on the points mentioned under \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "*   The paper presents a compelling and timely approach to on-policy learning, which is of significant interest to the community, particularly in the context of modern, massively parallelized simulation environments where data generation is no longer the primary bottleneck.\n*   The proposed method is a well-motivated synthesis of several powerful, existing algorithmic components. The originality lies in their successful adaptation and integration for the on-policy setting.\n*   The claims are supported by an extensive empirical evaluation, including thorough ablation studies, which demonstrates strong performance against relevant baselines.", "weaknesses": "The primary weaknesses of the paper lie in its exposition, where the mathematical notation could be more precise and consistent to improve clarity.\n*   The notation in Equation (1) is slightly ambiguous. The expectation is written as being over the state distribution $s \\sim \\mu_\\pi$, but the term inside also depends on the action $a$. It should be made explicit that the action is sampled from the policy, i.e., $a \\sim \\pi_\\theta(\\cdot|s)$.\n*   The presentation of the pathwise policy gradient in Equation (2) could be clearer:\n    *   The notation $a = \\pi_\\theta(x)$ is used without introduction. This notation typically implies a deterministic policy, whereas the algorithm employs a reparameterized stochastic policy. This should be clarified to avoid confusion with the Deterministic Policy Gradient theorem.\n    *   Furthermore, the expectation over the noise variable $\\epsilon$, which is essential to the reparameterization trick, is not made explicit in the expression.\n*   There is a notational inconsistency between different parts of the paper. For instance, Equations (8-11) use $\\pi_\\theta(x)$ to denote the policy distribution, while Equation (12) uses the more explicit conditional notation $\\pi_\\theta(a|x)$. Consistent notation should be used throughout.\n\nTypos/Nitpicks:\n*   In the caption for Figure 4, it is stated that \"we report the final performance at 100 million steps.\" However, the corresponding plot in Figure 4b includes a curve for \"PPO (200M)\". Please clarify the exact training horizon being reported.\n*   Line 368: \"perforamnce\" should be \"performance\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WnTnb9GKqC", "reviewer": "Reviewer_uFnn", "summary": "This paper proposes the REPPO algorithm that optimizes the policy by directly differentiating the Q function in a completely on-policy setting.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The problem studied in this paper is motivated well.\n\n+ Presentation is clear and easy to follow.\n\n+ Empirical results are comprehensive and the proposed algorithm performs well across several benchmark environments. It is particularly interesting to see that directly optimizing the $Q$ function is feasible for on-policy methods, which is often believed to rely on off-policy data.", "weaknesses": "I have some questions regarding the design of the REPPO algorithm.\n\n+ Upon checking the code, I found that the critic function does not directly take the state-action pair as the input, but a latent state $\\phi(s, a)$ which is the output of a feature module. That is, when differentiating the policy objective (Equation 12), using the reparameterization trick it yields \n$$ \\nabla_\\theta Q(x_i, a) = \\nabla_\\theta Q_\\eta ( \\phi(s_i, a) ) = \\frac{\\partial Q}{\\partial \\eta} \\frac{\\partial \\phi}{\\partial a} \\frac{\\partial a}{\\partial \\theta}$$\nwhere I denote the critic parameters by $\\eta$ instead of $\\phi$ to avoid abusing notation. I wonder if the feature module $\\phi$ is critical to the algorithm as the value landscape is much simpler in the latent space.\n\n+ Can you provide more insights on why the HL-Gauss loss is better than MSE in critic training?\n\n+ The rollout length in the code is 128, which is longer than horizons commonly used in on-policy methods. I wonder how different lengths affect the performance, especially for the tasks with short horizon such as Maniskill manipulation problems. \n\n+ Training the Q function can be numerically unstable, which has motivated many tricks in off-policy methods such as double Q learning, employing target networks, etc.. In Section 3.3, the paper introduces three methods to enhance the stability. However, as shown in the ablation study (Figure 10), removing any of them does not lead to da significant performance drop. Can you give me some intuition on why the critic training in REPPO is more robust than its off-policy counterparts beyond these three points?\n\n+ Typically, directly differentiating the Q function requires it to have a nice first-order approximation to the local value landscape in the state-action space so that its gradient with respect to the action yields a correct updating direction. To achieve this, off-policy methods use very short horizons for the TD estimation, such as one step in TD3 and SAC and five steps in TDMPC, under the consideration that longer trajectories would introduce more noise into the estimation and affect the accuracy of the gradient estimation. As mentioned earlier, REPPO uses a very long TD horizon, and I wonder why its Q function can still effectively capture the gradient direction in this case.\n\nI would be happy to increase my score if the authors can address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTCRZnNz9S", "reviewer": "Reviewer_XoJ7", "summary": "The authors propose an on-policy algorithm that utilizes a pathwise gradient estimator with a learned Q-function, along with specific design choices to stabilize the proposed algorithm. The proposed algorithm only involves computing policy updates on on-policy actions, which avoids importance sampling and instability in the traditional PPO method.", "questions": "- In Figure 2: why does PPG with learned surrogate optimize better than the ground truth, even if the contour lines of the objective look quite simple and one would expect following the gradient should be the best direction? Also, what do you use as Q(s, a) for the score-based estimator in this case (should the score-based estimator also have different variants using different surrogates for Q(s, a))?\n\n- Figure 3: The fact that the performance difference between REPPO (pathwise) and REPPO (score-based, Q) is tiny, and they both outperform PPO and REPPO (score-based, GAE) by a large margin, makes me wonder where the main performance gain of REPPO comes from. I am thinking that the performance benefit might mainly come from the fact that REPPO (pathwise) and REPPO (score-based, Q) query on-policy actions computed from the current policy (as opposed to the other two that use fixed, potentially off-policy actions sampled from data batches), which avoids the need for importance sampling scores. If that’s the case, then using a score-based vs. a pathwise estimator only makes a rather small difference. Please correct me if I am missing anything here. \n\n- Figure 4: Why are many of the baseline results using a different number of environment steps? For example, is SAC using 5M steps, i.e., 1/10 of those used by REPPO? If that’s the case, I think it is not a fair comparison, and the result for SAC is not a meaningful reference. Therefore, the claim about statistically significant improvement over SAC is overclaiming as well.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The design principle of the algorithm is, to my knowledge, novel, sound, and interesting\n- The algorithm is more on-policy than traditional PPO, and therefore, more stable. The authors showcase the performance benefits in extensive benchmark tests\n- Compared with off-policy methods, the proposed method does not require large memory consumption for a replay buffer\n- The presentation is clear and flows smoothly", "weaknesses": "- I think a more thorough justification (theoretical and/or empirical) about what the main factors are that bring the claimed performance benefit can significantly strengthen the authors' claims; cf. my question about Figure 3 below. Also, if learning a Q function instead of V function alleviates the high-variance problem in PPO, what are the potential costs? I think a more careful justification will have longer-term benefits for the community and further development of this type of method\n- Algorithm 1 (line 1364 of the document): V_{t+1} is not equal to Q_{\\phi}(x_{t+1}, a_{t+1}) (missing an expectation over action)\n- Policy optimization objective: the actual objective function used is not given in the text. Also, the notation used KL_{tar} is a bit arbitrary \n- If some of the improtant baselines are indeed using different number of total samples (see my question below), then, I will suggest a proper comparison to those baselines with same access to the environment samples as REPPO  \n- Minor:\n    - Line 246 typo “constraint it” -> “is”\n    - Equation 12 is the gating reverse KL, whereas other places are using the forward KL divergence. Which one is true?\n    - Current way of writing equation 12 is a bit off; consider pulling out the curly brace definition separately", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)