Reparameterization Proximal Policy Optimization

## Abstract

Abstract Reparameterization policy gradient (RPG) is promising for improving sample efficiency by leveraging differentiable dynamics. However, a critical barrier is its training instability, where high-variance gradients can destabilize the learning process. To address this, we draw inspiration from Proximal Policy Optimization (PPO), which uses a surrogate objective to enable stable sample reuse in the model-free setting. We first establish a connection between this surrogate objective and RPG, which has been largely unexplored and is non-trivial. Then, we bridge this gap by demonstrating that the reparameterization gradient of a PPO-like surrogate objective can be computed efficiently using backpropagation through time. Based on this key insight, we propose Reparameterization Proximal Policy Optimization (RPO), a stable and sample-efficient RPG-based method. RPO enables stable sample reuse over multiple epochs by employing a policy gradient clipping mechanism tailored for RPG. It is further stabilized by Kullback-Leibler (KL) divergence regularization and remains fully compatible with existing variance reduction methods. We evaluate RPO on a suite of challenging locomotion and manipulation tasks, where experiments demonstrate that our method achieves superior sample efficiency and strong performance.

## 1 Introduction

Reparameterization policy gradient (RPG) [17, 1] is a model-based method that computes the policy gradient using the reparameterization trick [12, 23]. Different from model-free policy gradients such as REINFORCE [30, 29], RPG directly backpropagates through the trajectory to obtain a policy gradient estimate. This approach has become increasingly attractive with the recent rise of differentiable simulators [9, 2, 32, 31] and learned world models [7, 1, 22].

RPG has the advantage of exploiting the underlying dynamical structure of the sampling path and can therefore have lower variance and a more accurate policy gradient estimate [17]. However, it can suffer from the exploding/vanishing gradient problem, particularly in environments with non-smooth dynamics or on long-horizon trajectories [16, 27]. Previous works have sought to mitigate these issues. A representative method, Short-Horizon Actor-Critic (SHAC) [32], reduces variance by only backpropagating through a short horizon of the trajectory. Recently, SAPO [31] builds upon SHAC by adding entropy regularization to further stabilize policy training.

Yet, even with current state-of-the-art (SOTA) variance reduction methods, we empirically observe that RPG can still suffer from unstable policy training. Specifically, we trained a locomotion policy with SAPO in the Humanoid environment of the differentiable simulator DFlex [32, 4]. As shown in Figure [1](https://arxiv.org/html/2508.06214v2#S1.F1), SAPO suffers from large policy updates (indicated by spikes in Kullback-Leibler divergence) that lead to sudden performance drops, despite incorporating both short-horizon rollouts and entropy regularization. This result indicates that although SHAC and SAPO reduce gradient variance, they lack an explicit mechanism to control the policy update size, which can lead to instability. This instability hinders RPG-based methods from boosting their sample efficiency to their full potential or from effectively incorporating techniques like sample reuse. This clearly highlights the need for a more effective algorithm to stabilize the training of RPG, thereby significantly improving its sample efficiency.

Figure: Figure 1: Comparison of RPO and SAPO in the Humanoid environment for the same random seed. Left: Training curves showing return versus environment steps. Right: The corresponding KL divergence between policy updates, displaying both raw and smoothed curves. SAPO’s large KL spikes correspond to sudden performance drops, whereas RPO maintains stable, sample-efficient learning.
Refer to caption: x1.png

In this work, we take inspiration from Proximal Policy Optimization (PPO) [25], a model-free algorithm renowned for stabilizing policy training even with multiple sample reuse. This desirable attribute comes from optimizing a its surrogate objective. However, adapting this principle to RPG-based methods is non-trivial. The surrogate objective is typically optimized via REINFORCE-style gradients and its connection to RPG has been largely unexplored.

To bridge this gap, we establish that RPG is naturally connected to a PPO-like surrogate objective via backpropagation through time (BPTT) [18], allowing us to compute the reparameterization policy gradients efficiently for both on- and off-policy updates. This connection is crucial because it provides a principled way to enable stable sample reuse in RPG, thereby stabilizing training and significantly improving its sample efficiency. Based on this insight, we propose Reparameterization Proximal Policy Optimization (RPO). First, it enables sample reuse by optimizing a surrogate objective. To maintain stability, it incorporates a clipped policy gradient mechanism tailored for RPG, which constrains updates arising from large importance weights. Second, it enhances stability with an explicit Kullback-Leibler (KL) divergence regularization term, as we find clipping alone to be insufficient. Furthermore, its practical implementation requires only a single backpropagation pass per trajectory across multiple updates. Finally, RPO is fully compatible with and benefits from existing variance reduction methods for RPG. We conduct experiments on a suite of locomotion and manipulation tasks using two differentiable simulators, DFlex [32, 4] and Rewarped [31]. Experimental results show that RPO achieves superior sample efficiency and strong performance.

In summary, our main contributions are: (i) we show that BPTT can be utilized to compute the on-policy and off-policy reparameterization policy gradients of a PPO-like surrogate objective, which enables stable sample reuse for RPG; (ii) based on this novel insight, we propose RPO, which enables stable and sample-efficient RPG-based policy learning, through multiple sample reuse, an importance weight clipping mechanism, and explicit KL regularization; (iii) we conduct experiments on a suite of locomotion and manipulation tasks using the popular differentiable simulators DFlex and Rewarped. Experiment results clearly demonstrate the superior sample efficiency of RPO and its strong performance.

## 2 Related Work

Policy Gradient Estimators. One classical class of policy gradient estimators is based on the score function, such as the REINFORCE gradient estimator [30, 29]. Many policy gradient methods, such as PPO and TRPO [25, 24], rely on variants of the REINFORCE gradient estimator. One limitation of the REINFORCE gradient is its high variance, which results in low sample efficiency. On the other hand, if one has access to the underlying dynamic model, either through differentiable simulators [2, 32, 9, 31] or learned world models [7, 1], another type of policy gradient named Reparameterization Policy Gradient (RPG), which is based on the reparameterization trick [12, 23], can be obtained. Using the reparameterization trick [12, 23], RPG directly backpropagates through the trajectory and obtains an unbiased estimate of the policy gradient. By contrast, the REINFORCE gradient estimator does not need to backpropagate through the entire computational graph and only relies on local computation [20]. Since RPG utilizes the gradients of the dynamics model, RPG typically enjoys less variance than the REINFORCE gradient estimator [17].

Reparameterization Policy Gradient-based Reinforcement Learning Algorithms. It is well known that RPG obtained by vanilla backpropagation through time over a long time horizon suffers from the vanishing/exploding gradient problem [27, 16, 34, 14]. This phenomenon is amplified when dealing with stiff dynamics, such as contact [33, 27, 36, 19]. RPG can exhibit a large variance when the gradient magnitude is large, which renders the underlying reinforcement learning algorithm unstable, struggling with non-convex loss landscapes.

Several works [21, 22, 27] weight and combine RPG and REINFORCE according to their variance, while AGPO [3] further combines RPG with gradients of Q-functions. SHAC and AHAC [32, 4] reduce the variance of RPG by only backpropagating through a truncated length of the trajectory, aided by a value function to estimate future returns. MB-MIX [35] backpropagates a mixture of trajectories with different lengths to better balance the bias-variance trade-off. GI-PPO [26] first optimizes the policy using RPG, then uses the REINFORCE gradient to perform further off-policy updates in the PPO style. However, the gradients computed by REINFORCE are not only of lower quality than those from RPG, but can also conflict with each other, thereby degrading sample efficiency and performance. Entropy is also introduced to regularize RPG-based policy updates and promote exploration [31, 1].

## 3 Preliminaries

### 3.1 Reinforcement Learning Formulation

In this work, we consider problems formulated as a Markov Decision Process (MDP) [28]. An MDP is formally defined by a tuple $(\mathcal{S},\mathcal{A},p,r,p_{0},\gamma)$, where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,1]$ is the state transition probability function, $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the reward function, $s_{0}$ is the initial state, $p_{0}(s_{0})$ is the initial state distribution, and $\gamma\in[0,1)$ is the discount factor.

The goal of reinforcement learning (RL) is to find the optimal parameter $\theta^{*}$ for a parameterized stochastic policy $\pi_{\theta}$. A parameterized stochastic policy $\pi_{\theta}(a|s)$ specifies the probability distribution over actions $a\in\mathcal{A}$ given a state $s\in\mathcal{S}$. The optimal parameter $\theta^{*}$ maximizes the expected discounted cumulative reward:

$$ $\displaystyle\theta^{*}$ $\displaystyle=\arg\max_{\theta}J(\theta)=\arg\max_{\theta}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R(\tau)\right]$ (1) $$

where the expectation is over trajectories $\tau=(s_{0},a_{0},s_{1},a_{1},\dots)$ generated by following the policy $\pi_{\theta}$. Here, $r(s_{t},a_{t})$ denotes the reward received at time step $t$, and $R(\tau)=\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})$ is the discounted cumulative reward for the trajectory $\tau$.

### 3.2 Reparameterization Policy Gradient

RPG relies on the reparameterization trick [12, 23, 6] to sample an action $a_{t}$ from a Gaussian policy $\pi_{\theta}(a_{t}|s_{t})$. This is achieved by first using the policy network to predict the mean $\mu_{\theta}(s_{t})$ and standard deviation $\sigma_{\theta}(s_{t})$, and then combining them with a reparameterization noise $\epsilon$ at time step $t$:

$$ $a_{t}=\mu_{\theta}(s_{t})+\sigma_{\theta}(s_{t})\cdot\epsilon,\quad\text{where}\ \epsilon\sim\mathcal{N}(0,\mathcal{I}).$ (2) $$

We denote this entire reparameterization transformation as $a_{t}=f_{\theta}(\epsilon;s_{t})$. With the reparameterization trick, RPG can backpropagate through trajectory by computing Jacobians, $\frac{\partial s_{t+1}}{\partial a_{t}}$ and $\frac{\partial s_{t+1}}{\partial s_{t}}$, to obtain a policy gradient estimate:

$$ $\nabla_{\theta}J(\theta)=\mathbb{E}_{s_{0},\epsilon_{0},s_{1},\epsilon_{1},\dots}\left[\nabla_{\theta}R(\tau)\right].$ (3) $$

Note that the expectation is taken with respect to the initial state distribution, sampled noises, and transition dynamics.

However, this full backpropagation through long trajectories often leads to high gradient variance and unstable training. Short-Horizon Actor-Critic (SHAC) [32], addresses this by backpropagating through only a short horizon of the trajectory, using a value function to capture the long-term return. SHAC’s variant of RPG has the following form:

$$ $\nabla_{\theta}[R(\tau_{t_{0}:t_{0}+h-1})+\gamma^{h}V(s_{{t_{0}}+h})],$ (4) $$

where $t_{0}$ is the starting time step for the trajectory, $h$ is the short horizon, $R(\tau_{t_{0}:t_{0}+h-1})$ is the cumulative reward of the trajectory within the short horizon, and $V(s_{{t_{0}}+h})$ is the value function’s estimate of the future return.

### 3.3 Surrogate Objective

PPO [25] and TRPO [24] optimize variants of the following surrogate objective function [11]:

$$ $\displaystyle L_{\pi_{\theta_{\text{old}}}}(\theta)=\int_{s}\sum_{t=0}^{\infty}\gamma^{t}p(s_{t}=s|\pi_{\theta_{\text{old}}})\int_{a}A^{\pi_{\theta_{\text{old}}}}(s,a)\pi_{\theta}(a|s)\mathrm{d}a\mathrm{d}s$ (5) $$

where $\pi_{\theta_{\text{old}}}$ is the behavior policy used to collect samples; $\sum_{t=0}^{\infty}\gamma^{t}p(s_{t}=s|\pi_{\theta_{\text{old}}})$ is the unnormalized state probability density induced by $\pi_{\theta_{\text{old}}}$; and $A^{\pi_{\theta_{\text{old}}}}(s,a)$ is the advantage function corresponding to the behavior policy. This objective measures the performance of the new policy $\pi_{\theta}$, using the state distribution and advantages for the behavior policy. PPO optimizes a clipped variant of this objective, while TRPO optimizes an explicit KL-constrained variant [25, 24]. By the law of the unconscious statistician [5], we can rewrite the surrogate objective in the reparameterization form:

$$ $\displaystyle L_{\pi_{\theta_{\text{old}}}}(\theta)$ $\displaystyle=\int_{s}d^{\pi_{\theta_{\text{old}}}}(s)\int_{\epsilon}A^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\mathrm{d}\epsilon\mathrm{d}s$ (6) $$

where $d^{\pi_{\theta_{\text{old}}}}(s)$ denotes the unnormalized state density induced by $\pi_{\theta_{\text{old}}}$ and $p_{\text{standard}}(\epsilon)$ is the probability density function for standard Gaussian distribution. This surrogate objective differs fundamentally from the objective of the stochastic value gradient (SVG) [8, 1], as equation [6](https://arxiv.org/html/2508.06214v2#S3.E6) measures performance using the value function of the behavior policy, while SVG uses the on-policy value function.

## 4 From Theoretical Connection to Algorithmic Design: Reparameterization Proximal Policy Optimization

### 4.1 A Novel Connection Between RPG and the Surrogate Objective

A critical limitation of SOTA RPG methods such as SHAC [32] and SAPO [31] is their reliance on a single policy update per batch of rollouts. This approach underutilizes the valuable, computationally expensive gradients computed from the backward pass through trajectories. While GI-PPO [26] attempts sample reuse, its hybrid approach of switching from an initial RPG update to subsequent REINFORCE gradients is also suboptimal; it introduces potential update conflicts and underutilizes the low-variance RPG for off-policy updates.

In this section, we provide a more principled solution by establishing a novel, direct connection between the PPO-style surrogate objective (equation [6](https://arxiv.org/html/2508.06214v2#S3.E6)) and the reparameterization gradient. This is a powerful connection, as the surrogate objective framework not only enables sample reuse but also allows for the adoption of a policy gradient clipping mechanism designed for RPG to enhance training stability. Our key technical contribution is a method that leverages this link, using a single BPTT pass per trajectory to efficiently compute the objective’s reparameterization policy gradient for multiple on- and off-policy updates. This approach creates a robust foundation for multi-epoch, stable sample reuse in RPG, maximizing the utility of each valuable gradient computation.

Now, we demonstrate how to compute the reparameterization gradient for the surrogate objective via BPTT. The goal is to estimate the reparameterization gradient of the reparameterization surrogate objective with respect to $\theta$, which has the following form:

$$ $\displaystyle\nabla_{\theta}L_{\pi_{\theta_{\text{old}}}}(\theta)$ $\displaystyle=\int_{s}d^{\pi_{\theta_{\text{old}}}}(s)\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}A^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\right]\mathrm{d}\epsilon\mathrm{d}s$ $\displaystyle=\int_{s}d^{\pi_{\theta_{\text{old}}}}(s)\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\right]\mathrm{d}\epsilon\mathrm{d}s.$ (7) $$

Here $A^{\pi_{\theta_{\text{old}}}}(s,a)=Q^{\pi_{\theta_{\text{old}}}}(s,a)-V^{\pi_{\theta_{\text{old}}}}(s)$ and $V^{\pi_{\theta_{\text{old}}}}(s)$ is constant with respect to the parameter $\theta$ being optimized.

Figure: Figure 2: Computing the reparameterization policy gradient of the surrogate objective involves three steps: (a) Action-gradients are computed from rollouts via a single backward pass and cached. (b) These gradients are used directly for the initial, on-policy update. (c) For subsequent off-policy updates, the cached gradients are importance-weighted by $\rho(\theta^{\prime})$ and reused, enabling stable sample reuse.
Refer to caption: x2.png

#### 4.1.1 Collect rollouts and compute action-gradients

As the first step, we collect a batch of rollouts with the behavior policy, setting $\pi_{\theta_{\text{old}}}=\pi_{\theta}$. For each trajectory, the cumulative reward from the time step $k$ onwards, $\sum_{t=k}^{\infty}\gamma^{t}r(s_{t},a_{t})=\gamma^{k}\sum_{t=k}^{\infty}\gamma^{(t-k)}r(s_{t},a_{t})$, is a single-sample Monte Carlo estimate for $\gamma^{k}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})$. We take the gradient of the discounted cumulative rewards with respect to each sampled action via BPTT. As shown in Figure [2](https://arxiv.org/html/2508.06214v2#S4.F2) (a), we cache these action-gradients, $\gamma^{k}\nabla_{a_{k}}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})$, for subsequent calculations. Note that $s_{k}$ and $a_{k}$ are sampled according to $p(s_{k}=s|\pi_{\theta_{\text{old}}})$ and $\pi_{\theta_{\text{old}}}(a_{k}|s_{k})$.

#### 4.1.2 On-policy and Off-policy reparameterization policy gradients

On-policy reparameterization policy gradient for the surrogate objective: The behavior policy $\pi_{\theta_{\text{old}}}$ is exactly the current policy $\pi_{\theta}$ being updated during the first policy update epoch. We compute the on-policy gradient by backpropagating the cached action-gradients through the policy network $\theta$, as depicted in Figure [2](https://arxiv.org/html/2508.06214v2#S4.F2) (b). This gives an unbiased on-policy reparameterization policy gradient estimate for the surrogate objective, as $\pi_{\theta_{\text{old}}}$ is identical to $\pi_{\theta}$ in this case (more details are shown in Appendix [D](https://arxiv.org/html/2508.06214v2#A4)).

Off-policy reparameterization policy gradient for the surrogate objective: The on-policy reparameterization policy gradient estimate can be used to update the policy parameters from $\theta$ to $\theta^{\prime}$. After the update, the behavior policy $\pi_{\theta_{\text{old}}}$ (which was used to generate the rollouts) is now different from the current policy $\pi_{\theta^{\prime}}$. To perform another update on the same data (i.e., sample reuse), we can reuse the exact same cached action-gradients. First, we regenerate the action with $\pi_{\theta^{\prime}}$ (details in Section [4.2.1](https://arxiv.org/html/2508.06214v2#S4.SS2.SSS1)). As shown in Figure [2](https://arxiv.org/html/2508.06214v2#S4.F2) (c), the cached action-gradients are then backpropagated through the updated policy network, which are further weighted by an importance sampling ratio $\rho(\theta^{\prime})=\frac{\pi_{\theta^{\prime}}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$ to obtain an unbiased off-policy reparameterization policy gradient estimate for the surrogate objective (a proof is given in Appendix [D](https://arxiv.org/html/2508.06214v2#A4)).

### 4.2 The Reparameterization Proximal Policy Optimization (RPO) Algorithm

Based on the novel reparameterization policy gradient for the surrogate objective above, we now introduce our proposed method: Reparameterization Proximal Policy Optimization (RPO). To achieve stable sample reuse, RPO incorporates three key mechanisms: (i) optimizing the PPO-like surrogate objective via RPG, (ii) a policy gradient clipping mechanism designed for RPG, and (iii) an explicit KL regularization term, which we found is necessary to ensure stability, as clipping alone is insufficient.

#### 4.2.1 Surrogate Objective for Policy Improvement

The foundation of our method is the surrogate objective for policy improvement, for which we could calculate its reparameterization policy gradient via the connection we derived above:

$$ $L_{surr}(\theta)=\mathbb{E}\left[A^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}\right].$ (8) $$

To optimize this objective, we first collect rollouts, then compute and cache the corresponding action-gradients.

Collecting rollouts and computing action-gradients. Following SHAC [32], we collect a batch of $N$ short-horizon trajectories using the current policy $\pi_{\theta}$, and then utilize BPTT to compute and cache the corresponding action-gradients, $\nabla_{a}R(\tau)$. Note that each trajectory in the batch is only backpropagated through once.

On-policy and Off-policy Updates. We perform $M$ optimization epochs on a batch of cached action-gradients. The first update is on-policy, while all subsequent updates ($1<m\leq M$) are off-policy. Our method for computing the reparameterization policy gradient of the surrogate objective is unified across both cases. For each update step, we perform the following procedure.

First, to compute the gradient for the current policy $\pi_{\theta}$ using off-policy data, we must re-establish a computational path from policy network parameters $\theta$ to the actions. We achieve this by computing the noise $\epsilon_{\text{reg}}$ that is required for the current policy to regenerate the actions stored in the rollout buffer:

$$ $\epsilon_{\text{reg}}=f^{-1}_{\theta}(a;s),$ (9) $$

where $f^{-1}_{\theta}(a;s)$ is the inverse of the reparameterization transform. With this recovered noise, we can express the action under the current policy as $a=f_{\theta}(\epsilon_{\text{reg}};s)$, which creates a new computational graph connecting the current policy parameters $\theta$ to the action stored in the buffer.

Next, we introduce a novel policy gradient clipping mechanism, designed specifically for RPG. This mechanism facilitates stable sample reuse by constraining policy updates from large importance weight ratios and preventing the probability ratio for an action from becoming too low. Unlike PPO’s clipping mechanism, our formulation clips the importance weight ratio asymmetrically and does not depend on the sign of the advantage function. This design is crucial because RPG, unlike REINFORCE, does not explicitly increase or decrease the log-likelihood of a specific action. Let the importance weight ratio be $\rho(\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$. The gradient contribution from this action is non-zero only if $\rho(\theta)$ is within the clipping range, and is weighted by $\rho(\theta)$:

$$ $\begin{cases}\rho(\theta)\nabla_{\theta}a\nabla_{a}R(\tau),&\text{if }1-c_{low}\leq\rho(\theta)\leq 1+c_{high},\\ 0,&\text{otherwise},\end{cases}$ (10) $$

where $\nabla_{a}R(\tau)$ is the cached action-gradient. By performing this step for all actions in the buffer, we obtain the clipped policy gradient for the surrogate objective.

#### 4.2.2 Regularization Terms

While our importance ratio-based policy gradient clipping mechanism is designed for RPG (as detailed above), we find that clipping alone is insufficient to ensure stability. This is partly because the clipping mechanism has no effect during the initial on-policy update, as all importance weight ratios are exactly $1$. Hence, we also incorporate a KL regularization [13] term, which penalizes large deviations from the previous policy:

$$ $L_{KL}(\theta)=\mathbb{E}\left[D_{KL}(\pi_{\theta_{\text{old}}}(\cdot|s)\ ||\ \pi_{\theta}(\cdot|s))\right].$ (11) $$

Note that this regularization only takes effect with sample reuse, as the KL divergence and its gradient are zero for the first on-policy update.

We also include an entropy bonus to encourage exploration [6, 31]:

$$ $L_{ent}(\theta)=\mathbb{E}\left[H(\pi_{\theta}(\cdot|s))\right],$ (12) $$

where $H(\pi_{\theta}(\cdot|s))$ denotes the entropy of $\pi_{\theta}$ at a given state.

#### 4.2.3 Overall Policy Training Objective and Policy Update

The overall policy training objective is to maximize a weighted combination of all three components:

$$ $L_{policy}(\theta)=\lambda_{surr}L_{surr}(\theta)-\lambda_{KL}L_{KL}(\theta)+\lambda_{ent}L_{ent}(\theta),$ (13) $$

where $\lambda_{surrogate},\lambda_{kl}$ and $\lambda_{ent}$ are the coefficients for the three terms. The three gradient components (from the surrogate objective and the two regularization terms) are combined according to their coefficients, and the final resulting gradient is used to update the policy parameters.

#### 4.2.4 Value Function Training

The value function network is trained by minimizing the following regression loss [32]:

$$ $L_{\phi}=\mathbb{E}\left[||V_{\phi}(s)-\hat{V}(s)||^{2}\right],$ (14) $$

where $V_{\phi}(s)$ is the estimate of the value function, and $\hat{V}(s)$ is the value target computed by TD-$\lambda$ [28]. We follow SAPO [31] using the double-critic network and including the mean of the two value functions for computing the value target.

Figure: Algorithm 1 Reparameterization Proximal Policy Optimization (RPO)

## 5 Experiments

We conduct experiments to answer the following three questions: (i) Does RPO achieve superior sample efficiency compared to that of previous RPG-based methods? (ii) Does RPO achieve strong performances? (iii) What are the impacts of RPO’s different main components on its overall performance?

### 5.1 Experimental setup

Environments and tasks. We conduct experiments on a suite of five challenging continuous control tasks from two differentiable simulators, DFlex [32, 4] and Rewarped [31]. This suite is composed of four locomotion tasks and one dexterous manipulation task. The four locomotion tasks are from DFlex [4], where the goal is to maximize the forward velocity: (i) Hopper; (ii) Ant; (iii) Anymal; and (iv) Humanoid. The manipulation task is the Hand Reorient environment from Rewarped [31], which involves an Allegro Hand learning to reorient a cube. Further details regarding the environments are provided in Appendix [A](https://arxiv.org/html/2508.06214v2#A1).

Baselines. We compare the sample efficiency and performance of RPO with leading RPG-based and model-free methods: (a) SAPO [31], an RPG-based method with short-horizon trajectories and entropy regularization; (b) SHAC [32], a variance reduction method for RPG, for which we use the implementation from [31] that includes several architectural changes that enhance its performance; (c) PPO [25], a model-free policy gradient method; (d) GI-PPO [26], which performs a single RPG update epoch followed by PPO-style updates for all subsequent epochs via REINFORCE. We made several architecture improvements to GI-PPO for a fair comparison. Detailed hyper-parameters and implementation specifics for all methods are provided in Appendix [C](https://arxiv.org/html/2508.06214v2#A3).

Metrics. We evaluate each algorithm using $12$ random seeds. To account for simulator stochasticity, we run each seed twice (for $24$ effective runs per experiment), except in the deterministic Hopper environment. Sample efficiency is evaluated via training curves in Figure [3](https://arxiv.org/html/2508.06214v2#S5.F3). We report the final performance over $128$ episodes using both deterministic and stochastic protocols in Table [1](https://arxiv.org/html/2508.06214v2#S5.T1) and Appendix [B](https://arxiv.org/html/2508.06214v2#A2), respectively.

Figure: Figure 3: Training performance comparison of RPO, SAPO, SHAC, and PPO. Each plot shows the mean episode return as a function of environment steps, with the shaded region representing the standard deviation. All curves are smoothed with a 100-episode moving average.
Refer to caption: x3.png

**Table 1: Deterministic Evaluation for the final performance after training. Each evaluation consists of 128 episodes. Mean and standard deviation.**
|  | Hand Reorient | Hopper | Ant | Humanoid | Anymal |
| --- | --- | --- | --- | --- | --- |
| PPO | 20.88 $\pm$ 15.90 | 3977.85 $\pm$ 159.95 | 4339.51 $\pm$ 745.48 | 2140.61 $\pm$ 529.07 | 10257.12 $\pm$ 2247.05 |
| GI-PPO | 36.78 $\pm$ 20.90 | 5514.00 $\pm$ 285.64 | 7812.05 $\pm$ 1165.10 | 7576.97 $\pm$ 621.58 | 12247.36 $\pm$ 3552.17 |
| SHAC | 175.13 $\pm$ 55.10 | 5068.42 $\pm$ 299.73 | 8205.95 $\pm$ 940.18 | 7722.20 $\pm$ 742.96 | 14560.59 $\pm$ 655.30 |
| SAPO | 225.17 $\pm$ 27.66 | 5478.12 $\pm$ 4.45 | 9101.36 $\pm$ 996.14 | 8676.25 $\pm$ 420.62 | 14783.27 $\pm$ 53.01 |
| RPO (ours) | 230.15 $\pm$ 33.12 | 5534.32 $\pm$ 223.68 | 9249.50 $\pm$ 831.56 | 8958.41 $\pm$ 373.64 | 16006.42 $\pm$ 342.49 |

### 5.2 Experimental Results

RPO significantly improves sample efficiency over baselines. RPO’s superior sample efficiency stems from its ability to (i) stabilize RPG policy training via a clipping mechanism and KL regularization, and (ii) reuse samples across multiple policy updates. As shown in Figure [3](https://arxiv.org/html/2508.06214v2#S5.F3), this translates to faster learning across tasks. For example, in Hand Reorient (top left), SAPO requires an additional $3$ million environment steps to achieve a performance comparable to RPO. In Hopper (top middle), RPO reaches a score of $5000$ approximately $5$ million steps sooner than all other methods. Similarly, in Ant (top right), RPO is the fastest to reach a score of $6000$. RPO surpasses the final performance of both SAPO and SHAC in Anymal (bottom right) after only $4$ million steps, and it reaches a score of $8000$ in Humanoid (bottom left) around 3 million steps faster than SAPO. Furthermore, RPO is also significantly more sample-efficient than PPO in all tasks. Notably, it demonstrates better training stability than SAPO, SHAC and GI-PPO, particularly in Humanoid, Hopper and Hand Reorient environments.

RPO achieves strong final performance. As summarized in Table [1](https://arxiv.org/html/2508.06214v2#S5.T1), RPO consistently achieves SOTA performance across all tasks. In the Anymal environment, RPO outperforms all other baselines by a significant margin. In the Humanoid and Ant environments, RPO outperforms SAPO while being significantly better than the other methods. For the Hopper and Hand Reorient tasks, RPO achieves the best performance, on par with SAPO and GI-PPO. In summary, RPO not only improves sample efficiency but also demonstrates strong final performance. Stochastic evaluation results, which follows the same trend, are provided in the Appendix [B](https://arxiv.org/html/2508.06214v2#A2).

Figure: Figure 4: Ablation study of RPO’s components in the Humanoid environment. The plot shows training curves for three variants: RPO without KL regularization, RPO with only two policy update epochs, and RPO without the importance weight clipping mechanism.
Refer to caption: x8.png

**Table 2: Ablation study of final deterministic performance in the Humanoid environment.**
| no KL loss | Policy Update Epochs (M=2) | no clipping | RPO |
| --- | --- | --- | --- |
| 8869.46 $\pm$ 430.64 | 8460.06 $\pm$ 429.92 | 8817.91 $\pm$ 436.16 | 8958.41 $\pm$ 373.64 |

### 5.3 Ablation Study

In our ablation study, we investigate the effect of RPO’s three main components. First, to test the necessity of KL regularization, we evaluate a variant of RPO without the KL loss. Second, to measure the benefit of sample reuse, we compare against a variant with only two policy update epochs, which is the minimal setting where the KL term becomes active. Finally, to isolate the effect of gradient clipping, we evaluate RPO with the clipping mechanism removed. The results are shown in Figure [4](https://arxiv.org/html/2508.06214v2#S5.F4) and Table [2](https://arxiv.org/html/2508.06214v2#S5.T2), from which we draw the following conclusions.

(i) KL regularization stabilizes policy training. As shown in Figure [4](https://arxiv.org/html/2508.06214v2#S5.F4), without KL regularization, policy training is more unstable and becomes significantly less sample-efficient. Furthermore, the final performance is also degraded. (ii) Sample reuse improves sample efficiency. The sample efficiency is significantly degraded when using only two policy update epochs (note that the default RPO uses five), and the final performance is also reduced. (iii) Clipping mechanism helps to stabilize policy training. As shown in the training curve, the clipping mechanism improves sample efficiency during the early training phase (in the first 2 million environment steps). This is because the importance weight ratio can become very large when the policy is improving rapidly, whereas the ratios tend to stay much closer to $1$ as the policy converges.

## 6 Conclusion

In this work, we addressed the training instability of Reparameterization Policy Gradient by establishing a key connection between RPG and a surrogate objective. This insight provides a principled path to stable sample reuse. Based on this, we propose Reparameterization Proximal Policy Optimization (RPO), an algorithm that stabilizes policy learning by applying a tailored gradient clipping mechanism to the surrogate objective’s policy gradient, further complemented by KL regularization. Our experiments on challenging locomotion and manipulation tasks confirm that RPO significantly outperforms prior methods in sample efficiency with strong performance. A promising direction for future work is investigating the sim-to-real transfer of RPO-trained policies.

## References

- [1]
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson.
On the model-based stochastic value gradient for continuous reinforcement learning.
In Ali Jadbabaie, John Lygeros, George J. Pappas, Pablo A.&nbsp;Parrilo, Benjamin Recht, Claire J. Tomlin, and Melanie N. Zeilinger, editors, Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 6–20. PMLR, 07 – 08 June 2021.
- [2]
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem.
Brax - a differentiable physics engine for large scale rigid body simulation.
In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
- [3]
Feng Gao, Liangzhi Shi, Shenao Zhang, Zhaoran Wang, and Yi Wu.
Adaptive-gradient policy optimization: Enhancing policy learning in non-smooth differentiable simulations.
In Forty-first International Conference on Machine Learning, 2024.
- [4]
Ignat Georgiev, Krishnan Srinivasan, Jie Xu, Eric Heiden, and Animesh Garg.
Adaptive horizon actor-critic for policy learning in contact-rich differentiable simulation.
In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
- [5]
Geoffrey R Grimmett and David R Stirzaker.
Probability and Random Processes.
Oxford University Press, 05 2001.
- [6]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 10–15 Jul 2018.
- [7]
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
Dream to control: Learning behaviors by latent imagination.
In International Conference on Learning Representations, 2020.
- [8]
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa.
Learning continuous control policies by stochastic value gradients.
In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
- [9]
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand.
Difftaichi: Differentiable programming for physical simulation.
ICLR, 2020.
- [10]
M. Hutter, C. Gehring, A. Lauber, F. Gunther, C. D. Bellicoso, V. Tsounis, P. Fankhauser, R. Diethelm, S. Bachmann, M. Bloesch, H. Kolvenbach, M. Bjelonic, L. Isler, and K. Meyer and.
Anymal - toward legged robots for harsh environments.
Advanced Robotics, 31(17):918–931, 2017.
- [11]
Sham Kakade.
A natural policy gradient.
In Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS’01, page 1531–1538, Cambridge, MA, USA, 2001. MIT Press.
- [12]
Diederik P Kingma and Max Welling.
Auto-encoding variational Bayes.
In International Conference on Learning Representations (ICLR), 2014.
- [13]
S. Kullback and R. A. Leibler.
On information and sufficiency.
The Annals of Mathematical Statistics, 22(1):79–86, 1951.
- [14]
Michel Ma, Tianwei Ni, Clement Gehring, Pierluca D’Oro, and Pierre-Luc Bacon.
Do transformer world models give better policy gradients?
In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
- [15]
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State.
Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
- [16]
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman.
Gradients are not all you need.
arXiv preprint arXiv:2111.05803, 2021.
- [17]
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih.
Monte carlo gradient estimation in machine learning.
J. Mach. Learn. Res., 21(1), January 2020.
- [18]
Michael C. Mozer.
A focused backpropagation algorithm for temporal pattern recognition, page 137–169.
L. Erlbaum Associates Inc., USA, 1995.
- [19]
Tao Pang, H. J. Terry Suh, Lujie Yang, and Russ Tedrake.
Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models.
IEEE Transactions on Robotics, 39(6):4691–4711, 2023.
- [20]
Paavo Parmas.
Total stochastic gradient algorithms and applications in reinforcement learning.
In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 10225–10235, Red Hook, NY, USA, 2018. Curran Associates Inc.
- [21]
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya.
PIPPS: Flexible model-based policy search robust to the curse of chaos.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4065–4074. PMLR, 10–15 Jul 2018.
- [22]
Paavo Parmas, Takuma Seno, and Yuma Aoki.
Model-based reinforcement learning with scalable composite policy gradient estimators.
In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27346–27377. PMLR, 23–29 Jul 2023.
- [23]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.
Stochastic backpropagation and approximate inference in deep generative models.
In Proceedings of The 30th International Conference on Machine Learning (ICML), 2014.
- [24]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.
Trust region policy optimization.
In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889–1897, Lille, France, 07–09 Jul 2015. PMLR.
- [25]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.
- [26]
Sanghyun Son, Laura Yu Zheng, Ryan Sullivan, Yi-Ling Qiao, and Ming Lin.
Gradient informed proximal policy optimization.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- [27]
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake.
Do differentiable simulators give better policy gradients?
In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20668–20696. PMLR, 17–23 Jul 2022.
- [28]
Richard S. Sutton and Andrew G. Barto.
Reinforcement Learning: An Introduction.
A Bradford Book, Cambridge, MA, USA, 2018.
- [29]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function approximation.
In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999.
- [30]
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine learning, 8:229–256, 1992.
- [31]
Eliot Xing, Vernon Luk, and Jean Oh.
Stabilizing reinforcement learning in differentiable multiphysics simulation.
In The Thirteenth International Conference on Learning Representations, 2025.
- [32]
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin.
Accelerated policy learning with parallel differentiable simulation.
In International Conference on Learning Representations, 2021.
- [33]
Shenao Zhang, Wanxin Jin, and Zhaoran Wang.
Adaptive barrier smoothing for first-order policy gradient with contact dynamics.
In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
- [34]
Shenao Zhang, Boyi Liu, Zhaoran Wang, and Tuo Zhao.
Model-based reparameterization policy gradient methods: Theory and practical algorithms.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 68391–68419. Curran Associates, Inc., 2023.
- [35]
Xiaoyuan Zhang, Xinyan Cai, Bo Liu, Weidong Huang, Song-Chun Zhu, Siyuan Qi, and Yaodong Yang.
Differentiable information enhanced model-based reinforcement learning.
Proceedings of the AAAI Conference on Artificial Intelligence, 39(21):22605–22613, Apr. 2025.
- [36]
Yaofeng Desmond Zhong, Jiequn Han, Biswadip Dey, and Georgia Olympia Brikis.
Improving gradient computation for differentiable physics simulation with contacts.
In Nikolai Matni, Manfred Morari, and George J. Pappas, editors, Proceedings of The 5th Annual Learning for Dynamics and Control Conference, volume 211 of Proceedings of Machine Learning Research, pages 128–141. PMLR, 15–16 Jun 2023.

## Appendix A Environment and Task details

In this section, we discuss the details of the environments and tasks used in this work. The four locomotion tasks (i.e., Anymal, Hopper, Ant, and Humanoid) are from the DFlex simulator [32, 4]. Specifically, we use the versions from AHAC’s official implementation ([https://github.com/imgeorgiev/DiffRL](https://github.com/imgeorgiev/DiffRL)). All locomotion tasks aim to learn a policy that maximizes the agent’s forward velocity. The Hand Reorient task is from the official implementation of Rewarped ([https://github.com/rewarped/rewarped](https://github.com/rewarped/rewarped)), version 1.3.0.

### A.1 Ant

Ant ($S\in\mathbb{R}^{37},A\in\mathbb{R}^{8}$) is a four-legged robot. The reward function is defined as [4]:

$$ $v_{x}+R_{height}+0.1R_{angle}+R_{heading}-0.01\|a\|^{2},$ $$

where $v_{x}$ is the forward velocity, and the other components are: $R_{height}$, which encourages the robot to stand up; $R_{angle}$, which rewards an upward-pointing normal vector; $R_{heading}$, which promotes forward movement; and a penalty on the action norm, $-0.01\|a\|^{2}$, to encourage energy-efficient policies.

### A.2 Anymal

Anymal ($S\in\mathbb{R}^{49},A\in\mathbb{R}^{12}$) is a real quadrupedal robot [10]. The reward function is defined as [4]:

$$ $v_{x}+R_{height}+0.1R_{angle}+R_{heading}-0.01\|a\|^{2}.$ $$

### A.3 Hopper

Hopper ($S\in\mathbb{R}^{11},A\in\mathbb{R}^{3}$) is a three-jointed planar robot. The reward function is defined as [4]:

$$ $v_{x}+R_{height}+R_{angle}-0.1\|a\|^{2}.$ $$

### A.4 Humanoid

Humanoid ($S\in\mathbb{R}^{76},A\in\mathbb{R}^{21}$) is a high-dimensional bipedal robot. The reward function is defined as [4]:

$$ $v_{x}+R_{height}+0.1R_{angle}+R_{heading}-0.02\|a\|^{2}.$ $$

### A.5 Hand Reorient

This task involves an Allegro Hand ($S\in\mathbb{R}^{72},A\in\mathbb{R}^{16}$) learning to reorient a cube to a target pose. This task was adapted for Rewarped [31] from Isaac Gym [15]. The detailed reward function can be found in the original Isaac Gym paper [15].

**Table 3: Common hyperparameters for all algorithms.**
|  | shared | PPO | SHAC | SAPO | RPO | GI-PPO |
| --- | --- | --- | --- | --- | --- | --- |
| Horizon $H$ | 32 |  |  |  |  |  |
| Epochs for critics $L$ |  | 5 | 16 | 16 | 32 | 16 |
| Epochs for actors $M$ |  | 5 | 1 | 1 | 5 | 6 |
| Discount $\gamma$ | $0.99$ |  |  |  |  |  |
| TD/GAE $\lambda$ | 0.95 |  |  |  |  |  |
| Actor MLP | $(400,200,100)$ | shared actor-critic MLP |  |  |  |  |
| Critic MLP | $(400,200,100)$ | shared actor-critic MLP |  |  |  |  |
| Actor $\eta$ |  | $5e-4$ | $2e-3$ | $2e-3$ | $5e-4$ | $5e-4$ |
| Critic $\eta$ | $5e-4$ |  |  |  |  |  |
| Entropy $\eta$ | - |  |  | $5e-3$ |  |  |
| $\eta$ schedule | - | KL($0.008$) | linear | linear | exponential | N.A. |
| Optim type | AdamW |  |  |  |  |  |
| Optim $(\beta_{1},\beta_{2})$ | $(0.7,0.95)$ | $(0.9,0.999)$ |  |  |  |  |
| Grad clip | $0.5$ |  |  |  |  | $1.0$ |
| Norm type | LayerNorm |  |  |  |  |  |
| Activation type | SiLU |  |  |  |  |  |
| Num critics $C$ | - |  | 2 | 2 | 2 | 2 |
| Target entropy $\bar{\mathcal{H}}$ | - |  |  | $-\mathrm{dim}(\mathcal{A})/2$ | $-\mathrm{dim}(\mathcal{A})/2$ |  |
| Init temperature | - |  |  | $1.0$ ($0.005$ for Hand Reorient) |  |  |

**Table 4: The number of parallel environments used for each environment. These values are kept the same as in the official implementations: we follow the AHAC repository ([https://github.com/imgeorgiev/DiffRL](https://github.com/imgeorgiev/DiffRL)) for the DFlex tasks and the Rewarped repository ([https://github.com/rewarped/rewarped](https://github.com/rewarped/rewarped)) for the Hand Reorient task.**
|  | Hopper | Ant | Humanoid | Anymal | Hand Reorient |
| --- | --- | --- | --- | --- | --- |
| Num Envs | 1024 | 128 | 64 | 128 | 64 |

**Table 5: RPO’s unique hyperparameters.**
|  | Hopper | Ant | Humanoid | Anymal | Hand Reorient |
| --- | --- | --- | --- | --- | --- |
| Entropy coefficient | $0.25$ | $0.35$ | $0.5$ | $0.25$ | $0.001$ |
| KL coefficient | $0.2$ | $0.5$ | $0.5$ | $0.2$ | $0.003$ |
| $c_{low}$ | $0.8$ | $0.8$ | $0.8$ | $0.8$ | $0.8$ |
| $c_{high}$ | $1.0$ | $1.0$ | $1.0$ | $1.0$ | $1.0$ |

**Table 6: GI-PPO’s unique hyperparameters.**
|  | Hopper | Ant | Humanoid | Anymal | Hand Reorient |
| --- | --- | --- | --- | --- | --- |
| alpha | $5e-1$ | $5e-1$ | $5e-3$ | $1e-3$ | $5e-4$ |
| max oorr | $0.7$ | $0.8$ | $0.1$ | $0.5$ | $0.1$ |
| e clip | $0.2$ | $0.2$ | $0.2$ | $0.2$ | $0.05$ |
| alpha interval | $0.4$ | $0.4$ | $0.4$ | $0.4$ | $0.4$ |
| alpha update factor | $1.02$ | $1.02$ | $1.02$ | $1.02$ | $1.02$ |

**Table 7: Stochastic Evaluation (i.e. sampling actions from the policy distribution) for the final performance after training. Each evaluation consists of 128 episodes. Mean and standard deviation.**
|  | Hand Reorient | Hopper | Ant | Humanoid | Anymal |
| --- | --- | --- | --- | --- | --- |
| PPO | 17.03 $\pm$ 14.83 | 3940.60 $\pm$ 129.72 | 4146.00 $\pm$ 164.14 | 1665.66 $\pm$ 410.93 | 8244.86$\pm$ 680.87 |
| GI-PPO | 27.19 $\pm$ 20.58 | 5512.30 $\pm$ 285.90 | 7805.47 $\pm$ 1159.23 | 7507.85 $\pm$ 639.52 | 12260.45$\pm$ 3650.35 |
| SHAC | 174.63 $\pm$ 57.54 | 5067.18 $\pm$ 299.37 | 8206.15 $\pm$ 940.46 | 7744.44 $\pm$ 858.97 | 14568.97 $\pm$ 652.72 |
| SAPO | 213.44 $\pm$ 33.95 | 5407.79 $\pm$ 4.28 | 8718.60 $\pm$ 946.94 | 8603.09 $\pm$ 402.82 | 14095.90 $\pm$ 82.02 |
| RPO (ours) | 214.36 $\pm$ 32.12 | 5469.33 $\pm$ 226.29 | 8966.00 $\pm$ 817.82 | 8767.16 $\pm$ 371.01 | 15776.12 $\pm$ 343.17 |

## Appendix B Stochastic Evaluation results

In this section, we provide the stochastic evaluation results for the five tasks used in the paper. For stochastic evaluations, we sample actions from the policy distribution. Results are shown in Table [7](https://arxiv.org/html/2508.06214v2#A1.T7). As shown in Table [7](https://arxiv.org/html/2508.06214v2#A1.T7), RPO achieves best results across tasks.

## Appendix C Hyperparameters and Implementation Details

### C.1 Hyperparameters and Architectures

We detail the hyperparameters and architectures used for all algorithms in Table [3](https://arxiv.org/html/2508.06214v2#A1.T3). For our GI-PPO baseline, we followed the official implementation ([https://github.com/SonSang/gippo](https://github.com/SonSang/gippo)) but made several improvements. These changes include using a double critic architecture, switching from the Adam to the AdamW optimizer, aligning the network size with our method, and changing the activation function from ELU to SiLU. Since GI-PPO is sensitive to its hyperparameters, we performed an extensive search within our computational budget. The resulting hyperparameters used are listed in Table [6](https://arxiv.org/html/2508.06214v2#A1.T6). Our implementations of SAPO, PPO, and SHAC are based on the official SAPO repository ([https://github.com/etaoxing/mineral](https://github.com/etaoxing/mineral)). Most hyperparameters are kept consistent with that repository, with a few key exceptions for fair comparison: the number of parallel environments and the MLP size are aligned with the official AHAC repository [4]. To ensure a fair comparison, most hyperparameters and the core architecture are shared across all tested algorithms. We tuned the initial temperature for SAPO in the Hand Reorient task, as the original setting of 1.0 from the SAPO paper was found to be too high. Specifically for SHAC, we use the improved version from the SAPO repository, which aligns its architecture with that of RPO and SAPO. Our RPO implementation is also built upon the SAPO repository.

### C.2 Implementation Details for RPO

For the critic, we use double critic and mean average as target for TD training, following [31]. For the entropy regularization, we follow SAPO to add an entropy bonus to the reward, which is scaled by a target entropy [31]. But note that the gradients of entropy are not backpropagated to actions at the same time step, so we calculate the gradients of entropy explicitly with respect to the policy parameter during policy update. We do not discount the explicitly calculated entropy gradients (but the gradients of entropy backpropagated to previous time steps are discounted), and this works empirically well. For the KL divergence computation, as we transform gaussian distribution to squashed normal distribution with tanh, and KL is invariant under such transformation. Hence, we could utilize the closed form expression for KL between gaussian to calculate the gradient of KL.

## Appendix D Details and proofs for Connecting RPG and Surrogate Objective

In this section, we give details to explain the connection between RPG and surrogate objective. First, we rewrite equation [4.1](https://arxiv.org/html/2508.06214v2#S4.Ex1) by expanding $d^{\pi_{\theta_{\text{old}}}}(s)$ and interchanging the order of integration and summation:

$$ $\begin{split}\nabla_{\theta}L_{\pi_{\theta_{\text{old}}}}(\theta)&=\int_{s}d^{\pi_{\theta_{\text{old}}}}(s)\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\mathrm{d}\epsilon\right]\mathrm{d}s,\\ &=\int_{s}\sum_{t=0}^{\infty}\gamma^{t}p(s_{t}=s|\pi_{\theta_{\text{old}}})\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\mathrm{d}\epsilon\right]\mathrm{d}s,\\ &=\sum_{t=0}^{\infty}\int_{s}\gamma^{t}p(s_{t}=s|\pi_{\theta_{\text{old}}})\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\mathrm{d}\epsilon\right]\mathrm{d}s,\end{split}$ (15) $$

which is the policy gradient to be estimated and $p_{\text{standard}}(\epsilon)$ is the probability density function for standard Gaussian distribution.

### D.1 Collect rollouts and compute action-gradients

First, as shown in Section [4.1.1](https://arxiv.org/html/2508.06214v2#S4.SS1.SSS1), we collect a batch of rollouts and compute the gradients of discounted cumulative return with respect to the action at each time step with BPTT. From here, we clearly see that the action-gradient for time step $k$,
$\gamma^{k}\nabla_{a_{k}}\sum_{t=k}^{\infty}\gamma^{(t-k)}r(s_{t},a_{t})$, is exactly an unbiased Monte Carlo estimate of $\gamma^{k}\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})$, where $s_{k}$ and $a_{k}$ are sampled according to $p(s_{k}=s|\pi_{\theta_{\text{old}}})$ and $\pi_{\theta_{\text{old}}}(a|s)$. We cache these action-gradients for further calculations.

### D.2 On-Policy Gradient

Now, we are ready to compute the reparameterization gradients. For the first policy update (on-policy update), the behavior policy $\pi_{\theta_{\text{old}}}$ is the same as the policy $\pi_{\theta}$ being updated. We can backpropagate the gradients from the action to the policy network parameters $\theta$. Then, we get an unbiased Monte Carlo estimate of $\int_{s}\gamma^{k}p(s_{k}=s|\pi_{\theta_{\text{old}}})\int_{\epsilon}\left[\nabla_{\theta}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta}(\epsilon;s)}p_{\text{standard}}(\epsilon)\right]\mathrm{d}\epsilon\mathrm{d}s$. This holds true, as $s_{k}$ is sampled according to $p(s_{k}=s|\pi_{\theta_{\text{old}}})$, $a_{k}$ is generated by sampling $\epsilon\sim p_{\text{standard}}(\epsilon)$ and then applying the transformation $a_{k}=f_{\theta_{\text{old}}}(\epsilon;s_{k})$, and $\pi_{\theta_{\text{old}}}$ coincides with $\pi_{\theta}$ for this policy update epoch.

By summing the gradients over different time steps and averaging across different trajectories, we obtain an unbiased on-policy reparameterization gradient estimation for equation [15](https://arxiv.org/html/2508.06214v2#A4.E15).

### D.3 Off-policy Gradient

After the first policy update, $\pi_{\theta}$ is updated to $\pi_{\theta^{\prime}}$, and we need to compute off-policy gradients. To do so, we need to account for the fact that to regenerate the same action collected in the rollout, a different sampled noise $\epsilon_{\text{reg}}$ is required for $\theta^{\prime}$.

To generate the sampled action $a_{k}$, $\epsilon_{\text{reg}}$ and $\epsilon$ are linked by the following relation:

$$ $\displaystyle a_{k}=f_{\theta_{\text{old}}}(\epsilon;s_{k})=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k}).$ (16) $$

Since we consider reparameterization Gaussian transformations in this work, $f_{\theta_{\text{old}}}$ and $f_{\theta^{\prime}}$ are invertible. We can therefore compute $\epsilon_{\text{reg}}=f^{-1}_{\theta^{\prime}}(a_{k};s_{k})$ and then regenerate $a_{k}$ with the updated policy $\pi_{\theta^{\prime}}$. Now, we backpropagate cached action-gradients to policy network parameters $\theta^{\prime}$ and obtain an unbiased estimation of $\gamma^{k}\nabla_{\theta^{\prime}}a_{k}\nabla_{a_{k}}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})|_{a_{k}=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}$.

However, instead of sampling $\epsilon_{\text{reg}}$ directly from $\mathcal{N}(0,\mathcal{I})$, we sample it as $\epsilon_{\text{reg}}=f^{-1}_{\theta^{\prime}}(f_{\theta_{\text{old}}}(\epsilon;s_{k});s_{k})$, where $\epsilon$ is sampled from $\mathcal{N}(0,\mathcal{I})$. Hence, the probability density $p_{\text{reg}}(\epsilon_{\text{reg}})$ is different from standard Gaussian. We therefore multiply the computed gradient term $\gamma^{k}\nabla_{\theta^{\prime}}a_{k}\nabla_{a_{k}}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})|_{a_{k}=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}$ by the importance ratio $\rho(\theta^{\prime})=\frac{\pi_{\theta^{\prime}}(a_{k}|s_{k})}{\pi_{\theta_{\text{old}}}(a_{k}|s_{k})}$ to obtain an unbiased off-policy gradient estimate. We now show that weighting by this importance weight ratio, we indeed obtain an unbiased estimate of the target gradient: $\int_{s}\gamma^{k}p(s_{k}=s|\pi_{\theta_{\text{old}}})\int_{\epsilon}\left[\nabla_{\theta^{\prime}}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta^{\prime}}(\epsilon;s)}p_{\text{standard}}(\epsilon)\right]\mathrm{d}\epsilon\mathrm{d}s$.

###### Proposition 1 .

Let the state $s_{k}$ be sampled from the state distribution of the behavior policy, $s_{k}\sim p(s_{k}=s|\pi_{\theta_{\text{old}}})$. Let the reparameterization noise $\epsilon$ be sampled from the standard Normal distribution, $\epsilon\sim p_{\text{standard}}(\epsilon)$, and define the regenerated noise as $\epsilon_{\text{reg}}=f^{-1}_{\theta^{\prime}}(f_{\theta_{\text{old}}}(\epsilon;s_{k});s_{k})$, where $f_{\theta^{\prime}}$ and $f_{\theta_{\text{old}}}$ are reparameterization Gaussian transformations.

Define the off-policy gradient estimator $G(\theta^{\prime})$ as the random variable:

$$ $G(\theta^{\prime})=\gamma^{k}\frac{\pi_{\theta^{\prime}}(a_{k}|s_{k})}{\pi_{\theta_{\text{old}}}(a_{k}|s_{k})}\nabla_{\theta^{\prime}}a_{k}\nabla_{a_{k}}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a_{k})|_{a_{k}=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}$ $$

where the action $a_{k}=f_{\theta_{\text{old}}}(\epsilon;s_{k})=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})$.

Then, $G(\theta^{\prime})$ is an unbiased estimate of the true policy gradient. That is, its expectation over the distributions of $s_{k}$ and $\epsilon$ is given by:

$$ $\mathbb{E}_{s_{k},\epsilon}\left[G(\theta^{\prime})\right]=\int_{s}\gamma^{k}p(s_{k}=s|\pi_{\theta_{\text{old}}})\int_{\epsilon}\left[\nabla_{\theta^{\prime}}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s_{k},a)|_{a=f_{\theta^{\prime}}(\epsilon;s_{k})}p_{\text{standard}}(\epsilon)\right]\mathrm{d}\epsilon\mathrm{d}s$ $$

###### Proof.

We first consider the case where the action $a\in\mathbb{R}$ is a scalar, and then generalize the result to the multi-dimensional case.

Since $\epsilon_{\text{reg}}$ is sampled as $\epsilon_{\text{reg}}=f^{-1}_{\theta^{\prime}}(f_{\theta_{\text{old}}}(\epsilon;s_{k});s_{k})$, $f_{\theta^{\prime}}$ and $f_{\theta_{\text{old}}}$ are differentiable and monotonically increasing with respect to $\epsilon_{\text{reg}}$ and $\epsilon$, we can calculate the probability density function $p_{\text{reg}}(\epsilon_{\text{reg}})$ by the change of variable formula [5]:

$$ $\displaystyle p_{\text{reg}}(\epsilon_{\text{reg}})=p_{\text{standard}}(\epsilon)\frac{\mathrm{d}f^{-1}_{\theta_{\text{old}}}(a_{k};s_{k})}{\mathrm{d}a_{k}}\frac{\mathrm{d}f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}{\mathrm{d}\epsilon_{\text{reg}}}.$ $$

We also know that the importance weight ratio has the following specific form:

$$ $\displaystyle\frac{\pi_{\theta^{\prime}}(a_{k}|s_{k})}{\pi_{\theta_{\text{old}}}(a_{k}|s_{k})}=\frac{p_{\text{standard}}(\epsilon_{\text{reg}})}{p_{\text{standard}}(\epsilon)}\frac{\frac{\mathrm{d}f^{-1}_{\theta^{\prime}}(a_{k};s_{k})}{\mathrm{d}a_{k}}}{\frac{\mathrm{d}f^{-1}_{\theta_{\text{old}}}(a_{k};s_{k})}{\mathrm{d}a_{k}}},$ $$

since $a_{k}=f_{\theta_{\text{old}}}(\epsilon;s_{k})=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})$, where $\epsilon$ and $\epsilon_{\text{reg}}$ are both sampled from standard Gaussian distributions, and applying the change of variable formula.

Now, we take the expectation of $G(\theta^{\prime})$ with respect to $\epsilon_{\text{reg}}$ with probability density function $p_{\text{reg}}$:

$$ $\displaystyle\gamma^{k}$ $\displaystyle\int_{\epsilon_{\text{reg}}}\left[\frac{\pi_{\theta^{\prime}}(a_{k}|s_{k})}{\pi_{\theta_{\text{old}}}(a_{k}|s_{k})}\nabla_{\theta^{\prime}}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s)}p_{{}_{\text{reg}}}(\epsilon_{\text{reg}})\mathrm{d}\epsilon_{\text{reg}}\right]$ $\displaystyle=\gamma^{k}\int_{\epsilon_{\text{reg}}}\left[\frac{p_{\text{standard}}(\epsilon_{\text{reg}})}{p_{\text{standard}}(\epsilon)}\frac{\frac{\mathrm{d}f^{-1}_{\theta^{\prime}}(a_{k};s_{k})}{\mathrm{d}a_{k}}}{\frac{\mathrm{d}f^{-1}_{\theta_{\text{old}}}(a_{k};s_{k})}{\mathrm{d}a_{k}}}\nabla_{\theta^{\prime}}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s)}\right.$ $\displaystyle\quad\left.\cdot p_{\text{standard}}(\epsilon)\frac{\mathrm{d}f^{-1}_{\theta_{\text{old}}}(a_{k};s_{k})}{\mathrm{d}a_{k}}\frac{\mathrm{d}f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}{\mathrm{d}\epsilon_{\text{reg}}}\mathrm{d}\epsilon_{\text{reg}}\right]$ $\displaystyle=\gamma^{k}\int_{\epsilon_{\text{reg}}}\left[\nabla_{\theta^{\prime}}a\nabla_{a}Q^{\pi_{\theta_{\text{old}}}}(s,a)|_{a=f_{\theta^{\prime}}(\epsilon_{\text{reg}};s)}p_{\text{standard}}(\epsilon_{\text{reg}})\mathrm{d}\epsilon_{\text{reg}}\right],$ $$

where the product of the derivatives $\frac{\mathrm{d}f^{-1}_{\theta^{\prime}}(a_{k};s_{k})}{\mathrm{d}a_{k}}$ and $\frac{\mathrm{d}f_{\theta^{\prime}}(\epsilon_{\text{reg}};s_{k})}{\mathrm{d}\epsilon_{\text{reg}}}$ is 1, as the functions are inverses. We can clearly see that this expectation matches our target, providing an unbiased estimate.

The extension of this result to the multi-dimensional case is straightforward. Since the source of randomness is a standard Gaussian distribution, each dimension is sampled independently. Consequently, both the overall importance ratio and $p_{\text{reg}}(\epsilon_{\text{reg}})$ factorize into a product of their respective one-dimensional components. Recalling that states are sampled according to $p(s_{k}=s|\pi_{\theta_{\text{old}}})$, and by moving $\gamma^{k}$ to the outer integral, we can conclude that this method provides an unbiased estimate.

∎

Thanks to proposition 1, by summing over different time step and averaging across different trajectories, we obtain an unbiased off-policy reparameterization gradient estimation.

Figure: Figure 5: Ablation study for comparison of RPO, SAPO, and SAPO with exponential learning rate schedule and $32$ critic update iterations. RPO outperforms SAPO in both settings.
Refer to caption: x11.png

###### Proposition 1 .

###### Proof.

## Appendix E Extended Ablation Experiments

RPO uses an exponential learning rate decay and $32$ critic update iterations. This is effective because RPO’s high sample efficiency allows it to learn quickly in the early stages of training, while the decaying learning rate helps to accelerate final convergence once a good performance level is reached. We investigate the effect of applying the exact same exponential learning rate schedule and $32$ critic update iterations to SAPO.

As shown in Figure [5](https://arxiv.org/html/2508.06214v2#A4.F5), RPO consistently outperforms SAPO with both settings. Indeed, SAPO’s final performance is degraded with an exponential learning rate decay and $32$ critic update iterations in most environments. This result indicates that RPO’s superior performance stems from its core algorithmic design—namely, the stable sample reuse mechanism—rather than being solely attributable to these specific hyperparameter choices.

## Appendix F The Use of Large Language Models

In this work, we use large language models for checking and fixing grammar and typos for both main text and appendix.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: see above section.
soundness: 2
strengths: 1. replacing the policy gradient-based objective of RPG with an PPO objective is interesting in terms of sample reuse.
2. using the old backpropagation through time (BPTT) trick to train the RPG model has not been done before.
summary: This paper enhances the Reparameterization Policy Gradient (RPG) method through two key modifications. The method improves sample efficiency by substituting the policy gradient objective with that of PPO, and it ensures training stability by employing the reparameterization trick, which facilitates gradient computation via standard Backpropagation Through Time (BPTT). Experiments validate that the proposed method outperforms existing RPG baselines.
weaknesses: 1.The primary contribution of this work lies in the reuse of trajectory samples. Given this, a comparison with off-policy evaluation and multi-step Q-learning methods is warranted, as they similarly grapple with the risk of high variance or inaccurate estimation caused by products of importance ratios. Although the authors employ a clipping trick to mitigate numerical instability, this approach comes at the cost of sample effectiveness, potentially preventing a significant portion of trajectories from being reused.

2. The paper introduces a design choice in Eq. (9) to explicitly align the new policy with the old trajectory. We argue that the method should also be feasible without this alignment. Specifically, one could directly assume the actions from the old trajectory are executed by the new policy—given the identical action space—and then calculate the importance sampling ratio conventionally, similar to PPO. The authors should arguably include this approach as a necessary baseline and present experimental comparisons to justify the superiority of their proposed design.

3. The experimental validation presented is incomplete. The authors have not adequately discussed or analyzed the method's key hyperparameters, such as by providing a sensitivity analysis on how different parameter values affect final performance. This omission makes it difficult for readers to assess the method's stability and its generalization capability under different settings.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 1
questions: 1. the clipping hyperparameters $c_{low}, c_{high}$. I check Table 4, they use very large clipping range (0.8), while in PPO they usually use 0.2. Can you tell the reasons? How to choose them in practice?

2. Figure 1 only shows the result of one seed. Is that also common in other seeds?

3. Also see my concerns in Weakness.
soundness: 4
strengths: Generally, I like the idea of the paper. Using the PPO's idea of surrogate function to update the computation graph and reuse the previous samples is smart.

In addition, the paper explores an important problem — stabilizing reparameterization-based RL in differentiable simulators — and provides an algorithm that is simple, general, and compatible with existing frameworks.
The empirical results on DFlex and Rewarped demonstrate that the method can improve stability and efficiency across multiple continuous-control tasks.
summary: This paper proposes Reparameterization Proximal Policy Optimization (RPO), which integrates the PPO surrogate objective into the reparameterization policy gradient (RPG) framework.
By caching action gradients from a single BPTT pass and reusing them through importance-weighted clipping and KL regularization, RPO enables stable and sample-efficient updates in differentiable simulators.
Experiments on DFlex and Rewarped show improved stability and efficiency compared with SHAC and SAPO.
weaknesses: Though the idea of the paper looks good, its writing is not very good to me.

### The clearness of the presentation:
I found the paper hard to follow and understand. The derivation of the formulas is not very clear and solid, which need largely polish.
1. Eq. (8) and the following text are confusing to me. They use $A(s,a)$ in Eq. (8), and then in line 283 they use $\nabla R(\tau)$. It would greatly increase readability if Eq. (8) could be written in the form of $R(\tau)$.
2. In Eq. (3), the expectation is over the whole trajectory ($s_t,\epsilon_t$), while in Eq. (8) it would be clearer to express it in trajectory form as well. Otherwise it is unclear to the readers. 
3. Eq. (10) should give a reference to Proposition 1. Otherwise readers cannot understand why it takes this form.

### The experiments are not very sufficient:
1. Regarding fairness: it seems that the method updates the policy for additional epochs (while using the same samples). It is unclear what happens if the compared methods are also trained with more epochs. I do not necessarily expect the proposed method to achieve higher scores, but it may save computation time. The paper should include such results to demonstrate the practical benefit.
2. The ablation study on the clipping hyperparameters $c_{low}, c_{high}$ is missing. 
3. From the experiments, KL and clipping appear to have less effect, while the number of update epochs has more effect. This may reduce the necessity of the proposed method.

### Some minor problems:
1. The method relies on differentiable transitions; this should be stated clearly in the Preliminaries. I am also uncertain whether the method requires the transition to be deterministic.
2. The y label in Figure 1 should be cleared shown.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1) A theoretical analysis can be provided to prove that the improved training algorithm can lead to a better final performance and more stable training.

2) It is hoped to see an additional section, to discuss (from a theoretical/experimental perspective) that whether the redundancy still exists between KL and clipping in model-based methods.

3) Among the experimental results of all tasks, PPO showed a very obvious, even counterintuitive performance disadvantage. Can the authors analyze the reasons behind this phenomenon? Is it caused by the task selection?
soundness: 3
strengths: 1) The paper is well-written and the ideas are presented clearly, making it accessible and easy to follow.

2) The experimental part provides sufficient verification on multiple tasks, and well-chosen baseline methods, clearly demonstrating the advantages of RPO in terms of the sample efficiency and training stability.

3) Ablation experiment is well-organized, which separately describes the contributions of three algorithm improvements: KL regularization, sample reuse and clipping mechanism.
summary: This paper identifies that Reparameterization Policy Gradient (RPG) methods, though with a low-variance, suffer from the instability when long-horizon back-propagation or sample reuse is attempted. Inspired from Proximal Policy Optimization (PPO), this paper shows that the reparameterization gradient of a PPO-like surrogate objective can be computed efficiently using Back-Propagation Through Time (BPTT). Leveraging this connection, the authors propose Reparameterization Proximal Policy Optimization (RPO), which re-uses cached action-gradients for multiple epochs while keeping updates conservative via (i) asymmetric clipping of importance weights tailored to RPG and (ii) an explicit KL-penalty that activates only during off-policy epochs.
weaknesses: 1) Introducing the clip and KL divergence penalty operations from PPO into model-based methods is a relatively intuitive idea, and it is not novel in terms of the algorithm improvement. If a theoretical analysis can be provided to prove that the improved training algorithm can lead to a better final performance and more stable training, the contributions of this paper will be enriched.

2) During the design of the PPO algorithm, it was observed that there is a certain degree of redundancy between the KL regularization and the clipping mechanism, and only one of them was used. In the ablation study of this paper, the results of “no KL loss” and “no clipping” were very similar. This raises the suspicion that in model-based methods, the redundancy seems to still hold, and there is no need to jointly use them both.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: - Line 242: How do the authors compute the infinite sum given collected finite-length rollouts?
- Algorithm 1: If all the collected rollouts are short-horizon, how does the learned value function know about longer-term returns?
soundness: 3
strengths: - The problem being solved is meaningful and practically useful, i.e., sample reusing for RPG methods. 
- Empirically, the proposed approach is among the leading methods across all the evaluated tasks.
summary: The authors propose a PPO-resembling mechanism enabling reusing collected rollouts for multiple policy update steps in reparameterization policy gradient methods (policy gradients estimated in a model-based fashion, e.g., using a differentiable simulator).
weaknesses: - The proposed approach seems to be a bit incremental—mostly a direct map from PPO to the RPG world. Although caching action gradients for restoring policy gradients for the updated policy does look like a smart engineering workaround. 
- Line 243 “We take the gradient of the discounted cumulative rewards with respect to each sampled action via BPTT.” - This assumes a known reward function (and smoothness as well as differentiability w.r.t. states and actions)
- Presentation can be organized in a better way. Right now, section 4.1.2 is describing the main idea while referencing a bunch of other sections and jumping back and forth; then, section 4.2.1 is repeating things already described above. I feel like readers can benefit from a more focused presentation. 
- “RPO significantly improves sample efficiency over baselines.” — This sounds a bit like overclaiming, as the proposed approach is on par with strong baselines on a good number of tasks
- The ablation seems insufficient; also, there does not seem to be a big difference in all three subplots in Figure 4. I would suggest a more detailed ablation study to better understand the necessity and benefit of each component.