On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling

## Abstract

Abstract On-policy reinforcement learning ( rl ) algorithms perform policy updates using i.i.d. trajectories collected by the current policy.
However, after observing only a finite number of trajectories, on-policy sampling may produce data that fails to match the expected on-policy data distribution.
This sampling error leads to noisy updates and data inefficient on-policy learning.
Recent work in the policy evaluation setting has shown that non-i.i.d., off-policy sampling can produce data with lower sampling error than on-policy sampling can produce [Zhong et al., 2022 ] .
Motivated by this observation, we introduce an adaptive, off-policy sampling method to improve the data efficiency of on-policy policy gradient algorithms.
Our method, P roximal R obust O n- P olicy S ampling ( props ), reduces sampling error by collecting data with a behavior policy that increases the probability of sampling actions that are under-sampled with respect to the current policy.
We empirically evaluate props on both continuous-action MuJoCo benchmark tasks as well discrete-action tasks
and demonstrate that (1) props decreases sampling error throughout training and (2) improves the data efficiency of on-policy policy gradient algorithms.

### 1 Introduction

One of the most widely used classes of reinforcement learning (rl) algorithms is the class of on-policy policy gradient algorithms.
These algorithms use gradient ascent on the parameters of a parameterized policy so as to increase the probability of observed actions with high expected returns under the current policy.
The gradient is commonly estimated using the Monte Carlo estimator, an average computed over i.i.d. samples of trajectories from the current policy.
The Monte Carlo estimator is consistent and unbiased; as the number of sampled trajectories increases, the empirical distribution of trajectories converges to the true distribution under the current policy, and thus the empirical gradient converges to the true gradient.
However, the expense of environment interactions forces us to work with finite samples.
Thus, the empirical distribution of the trajectories often differs from the desired on-policy data distribution.
This sampling error produces inaccurate gradient estimates, resulting in noisy policy updates, slower learning, and potentially convergence to suboptimal policies.
With i.i.d. on-policy sampling, the only way to reduce sampling error is to collect more data.

Since on-policy sampling is so widely used to produce on-policy data, on-policy sampling is often taken to be an essential feature of data collection for on-policy learning [Silver, , Achiam, ].
However, on-policy sampling is not explicitly required for consistent policy updates;
on-policy learning requires on-policy data – data whose state-conditioned empirical distribution of actions matches that of the current policy.
On-policy sampling is a straightforward way to acquire on-policy data, though we can obtain such data more efficiently without on-policy sampling.
To better illustrate this concept, consider an MDP with two discrete actions A and B, and suppose the current policy $\pi$ places equal probability on both actions in some state $s$.
When following $\pi$, after 10 visits to $s$, we may observe A 2 times and B 8 times rather than the expected 5 times.
Alternatively, if we adaptively select the most under-sampled action upon every visit to $s$, we will observe each action an equal number of times.
The first scenario illustrates on-policy sampling but not on-policy data; the second scenario uses off-policy sampling yet produces on-policy data.

These observations raise the following question:
can on-policy policy gradient algorithms learn more efficiently using on-policy data acquired without on-policy sampling?
Recently, Zhong et al. [] showed that adaptive, off-policy sampling can yield data that more closely matches the on-policy distribution than data produced by i.i.d. on-policy sampling.
However, this work was limited to the policy evaluation setting in which the on-policy distribution remains fixed.
Turning from evaluation to control poses the challenge of a continually changing current policy.

In this work, we address this challenge and show for the first time that on-policy policy gradient algorithms are more data-efficient learners when they use on-policy data acquired with adaptive, off-policy sampling.
We introduce a method, Proximal Robust On-Policy Sampling (props)
, that adaptively corrects sampling error in previously collected data by increasing the probability of sampling actions that are under-sampled with respect to the current policy.
Figure 1 provides an overview of props.
We empirically evaluate props on continuous-action MuJoCo benchmark tasks as well as discrete action tasks and show that (1) props reduces sampling error throughout training and (2) that doing so improves the data efficiency of on-policy policy gradient algorithms.
In summary, our contributions are

- 1.
We introduce a scalable adaptive sampling algorithm that reduces sampling error in on-policy data collection.
- 2.
We demonstrate empirically that our method improves the data efficiency of on-policy policy gradient algorithms and increases the fraction of training runs that converge to high-return polices.
- 3.
Building off of the theoretical foundation laid by Zhong et al. [], this work improves the rl community’s understanding of a nuance in the on-policy vs off-policy dichotomy: on-policy learning requires on-policy data, not on-policy sampling.

Figure: Figure 1: An overview of props for on-policy policy gradient learning. Rather than collecting data ${\mathcal{D}}$ via on-policy sampling from the agent’s current policy $\pi_{\bm{\theta}}$, we collect data with a separate data collection policy $\pi_{\bm{\phi}}$ that we continually adapt to reduce sampling error in ${\mathcal{D}}$ with respect to the agent’s current policy.
Refer to caption: extracted/5905669/figures/overview_final.png

### 2 Related Work

Our work focuses on data collection in rl.
In rl, data collection is often framed as an exploration problem, focusing on how an agent should explore its environment to efficiently learn an optimal policy.
Prior rl works have proposed several exploration-promoting methods such as intrinsic motivation [Pathak et al., , Sukhbaatar et al., ], count-based exploration [Tang et al., , Ostrovski et al., ], and Thompson sampling [Osband et al., , Sutton and Barto, ].
In contrast, our objective is to learn from the on-policy data distribution; we use adaptive data collection to more efficiently obtain this data distribution.

Prior works have used adaptive off-policy sampling to reduce sampling error in the policy evaluation subfield of rl.
Most closely related is the work of Zhong et al. [] who first proposed that adaptive off-policy sampling could produce data that more closely matches the on-policy distribution than on-policy sampling could produce.
Mukherjee et al. [] use a deterministic sampling rule to take actions in a particular proportion.
Other bandit works use a non-adapative exploration policy to collect additional data conditioned on previously collected data [Tucker and Joachims, , Wan et al., , Konyushova et al., ].
Since these works only focus on policy evaluation, they do not have to contend with a changing on-policy distribution as our work does for the control setting.

Several prior works propose importance sampling methods [Precup, ] to reduce sampling error without further data collection.
In the rl setting, Hanna et al. [] showed that reweighting off-policy data according to an estimated behavior policy can correct sampling error and improve policy evaluation.
Similar methods have been studied for temporal difference learning [Pavse et al., ] and policy evaluation in the bandit setting [Li et al., , Narita et al., ].
Conservative Data Sharing [Yu et al., ] reduces sampling error by selectively integrating offline data from multiple tasks.
Our work instead focuses on using additional data collection to reduce sampling error.

As we will discuss in Section 5, the method we introduce permits data collected in one iteration of policy optimization to be re-used in future iterations rather than discarded as typically done by on-policy algorithms.
Prior work has attempted to avoid discarding data by combining off-policy and on-policy updates with separate loss functions or by using alternative gradient estimates [Wang et al., , Gu et al., , , Fakoor et al., , O’Donoghue et al., , Queeney et al., ].
In contrast, our method modifies the sampling distribution at each iteration so that the entire data set of past and newly collected data matches the expected distribution under the current policy.

### 3 Preliminaries

#### 3.1 Reinforcement Learning

We formalize the rl environment as a finite horizon
Markov decision process (MDP) [Puterman, ] $({\mathcal{S}},{\mathcal{A}},p,r,d_{0},\gamma)$ with state space ${\mathcal{S}}$, action space ${\mathcal{A}}$, transition dynamics $p:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{S}}\to[0,1]$, reward function $r:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}$, initial state distribution $d_{0}$, and reward discount factor $\gamma\in[0,1)$.
The state and action spaces may be discrete or continuous.
We write $p(\cdot\mid{\bm{s}},{\bm{a}})$ to denote the distribution of next states after taking action ${\bm{a}}$ in state ${\bm{s}}$.
We consider stochastic policies $\pi_{\bm{\theta}}:{\mathcal{S}}\times{\mathcal{A}}\to[0,1]$ parameterized by ${\bm{\theta}}$, and we write $\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})$ to denote the probability of sampling action ${\bm{a}}$ in state ${\bm{s}}$ and $\pi_{\bm{\theta}}(\cdot|{\bm{s}})$ to denote the probability distribution over actions in state ${\bm{s}}$.
We additionally let $d_{\pi_{\bm{\theta}}}:{\mathcal{S}}\times{\mathcal{A}}\to[0,1]$ denote the state-action visitation distribution, the distribution over state-action pairs induced by following $\pi_{\bm{\theta}}$.
The rl objective is to find a policy that maximizes the expected sum of discounted rewards, defined as:

$$ $\begin{split}J({\bm{\theta}})=\mathbb{E}_{({\bm{s}},{\bm{a}})\sim d_{\pi_{\bm{ \theta}}}}\left[\sum\nolimits_{t=0}^{H}\gamma^{t}r({\bm{s}}_{t},{\bm{a}}_{t}) \right],\end{split}$ (1) $$

where the horizon $H$ is the random variable representing the time-step when an episode ends.
Throughout this paper, we refer to the policy used for data collection as the behavior policy and the policy trained to maximize its expected return as the target policy.

#### 3.2 On-Policy Policy Gradient Algorithms

Policy gradient algorithms are one of the most widely used methods in rl.
These methods perform gradient ascent over policy parameters to maximize an agent’s expected return $J({\bm{\theta}})$ (Eq. 1).
The gradient of the $J({\bm{\theta}})$ with respect to ${\bm{\theta}}$, or policy gradient, is often given as:

$$ $\nabla_{\bm{\theta}}J({\bm{\theta}})=\mathbb{E}_{({\bm{s}},{\bm{a}})\sim d_{ \pi_{\bm{\theta}}}}\left[A^{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})\nabla_{\bm{ \theta}}\log\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})\right],$ (2) $$

where $A^{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})$ is the advantage of choosing action ${\bm{a}}$ in state ${\bm{s}}$ and following $\pi_{\bm{\theta}}$ thereafter.
In practice, the expectation in Eq. 2 is approximated with Monte Carlo samples collected from $\pi_{\bm{\theta}}$ and an estimate of $A^{\pi_{\bm{\theta}}}$ used in place of the true advantages [Schulman et al., ].
After updating the policy parameters with this estimated gradient, the previously collected trajectories ${\mathcal{D}}$ become off-policy with respect to the updated policy.
To ensure gradient estimation remains unbiased, on-policy algorithms discard historic data after each update and collect new data with the updated policy.

This foundational idea of policy learning via stochastic gradient ascent was first proposed by Williams [Williams, ] under the name reinforce.
Since then, a large body of research has focused on developing more scalable policy gradient methods [Kakade, , Schulman et al., , Mnih et al., , Espeholt et al., , Lillicrap et al., , Haarnoja et al., ].
Arguably, the most successful variant of policy gradient learning is proximal policy optimization (ppo) [Schulman et al., ], the algorithm of choice in several high-profile success stories [Berner et al., , Akkaya et al., , Vinyals et al., ].
Rather than maximizing the standard rl objective (Eq. 1), ppo maximizes a surrogate objective:

$$ $\begin{split}{\mathcal{L}}_{\text{{ppo}{}}}&({\bm{s}},{\bm{a}},{\bm{\theta}},{ \bm{\theta}}_{\text{old}})=\min(g({\bm{s}},{\bm{a}},{\bm{\theta}},{\bm{\theta} }_{\text{old}})A^{\pi_{{\bm{\theta}}_{\text{old}}}}({\bm{s}},{\bm{a}}),\\ &\mathtt{clip}(g({\bm{s}},{\bm{a}},{\bm{\theta}},{\bm{\theta}}_{\text{old}}),1 -\epsilon,1+\epsilon)A^{\pi_{{\bm{\theta}}_{\text{old}}}}({\bm{s}},{\bm{a}})), \end{split}$ (3) $$

where ${\bm{\theta}}_{\text{old}}$ denotes the policy parameters prior to the update, $g({\bm{s}},{\bm{a}},{\bm{\theta}},{\bm{\theta}}_{\text{old}})$ is the policy ratio $g({\bm{s}},{\bm{a}},{\bm{\theta}},{\bm{\theta}}_{\text{old}})=\frac{\pi_{\bm{
\theta}}({\bm{a}}|{\bm{s}})}{\pi_{{\bm{\theta}}_{\text{old}}}({\bm{a}}|{\bm{s}
})}$, and the $\mathtt{clip}$ function with hyperparameter $\epsilon$ clips $g({\bm{s}},{\bm{a}},{\bm{\theta}},{\bm{\theta}}_{\text{old}})$ to the interval $[1-\epsilon,1+\epsilon]$.
The first term inside the minimum of ${\mathcal{L}}_{\text{{ppo}{}}}$ is the conservative policy iteration (CPI) objective [Kakade and Langford, ].
The second term clips the CPI objective to disincentivizes large changes to $\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})$.
In contrast to other policy gradient algorithms which perform a single gradient update per data sample to avoid destructively large weight updates, ppo’s clipping mechanism allows the agent to perform multiple epochs of minibatch policy updates.

### 4 Correcting Sampling Error in Reinforcement Learning

In this section, we illustrate how sampling error can produce inaccurate policy gradient estimates and then describe how adaptive, off-policy sampling can reduce sampling error.
For exposition, we assume finite state and action spaces.
The policy gradient can then be written as:

$$ $\nabla_{\bm{\theta}}J({\bm{\theta}})=\sum_{({\bm{s}},{\bm{a}})\in{\mathcal{S}} \times{\mathcal{A}}}d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})\left[A^{\pi_{\bm{ \theta}}}({\bm{s}},{\bm{a}})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}({\bm{a}} |{\bm{s}})\right].$ (4) $$

The policy gradient is thus a linear combination of the gradient for each $({\bm{s}},{\bm{a}})$ pair $\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})$ weighted by $d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})A^{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})$.
Let ${\mathcal{D}}$ be a dataset of trajectories.
It is straightforward to show that the Monte Carlo estimate of the policy gradient can be written in a similar form as Equation 4 except with the true state-action visitation distribution replaced with the empirical visitation distribution, $d_{\mathcal{D}}({\bm{s}},{\bm{a}})$ [Hanna et al., ].
Consequently, when $({\bm{s}},{\bm{a}})$ is over-sampled (i.e., $d_{\mathcal{D}}({\bm{s}},{\bm{a}})>d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})$), the $\nabla_{\bm{\theta}}\log{\pi_{\bm{\theta}}}({\bm{a}}|{\bm{s}})$ contributes more to the overall gradient than it should.
Similarly, when $({\bm{s}},{\bm{a}})$ is under-sampled, $\nabla_{\bm{\theta}}\log{\pi_{\bm{\theta}}}({\bm{a}}|{\bm{s}})$ contributes less than it should.

We now provide a concrete example illustrating how small amounts of sampling error can cause the wrong actions to be reinforced, resulting in sub-optimal convergence.
Suppose that in a particular state ${\bm{s}}_{0}$, an agent places equal probability on two actions ${\bm{a}}_{0}$ and ${\bm{a}}_{1}$ with advantages $A^{\pi_{\bm{\theta}}}({\bm{s}}_{0},{\bm{a}}_{0})=20$ and $A^{\pi_{\bm{\theta}}}({\bm{s}}_{0},{\bm{a}}_{1})=15$, respectively.
Since $\nabla_{\bm{\theta}}\log{\pi_{\bm{\theta}}}({\bm{a}}_{0}|{\bm{s}}_{0})=\nabla_
{\bm{\theta}}\log{\pi_{\bm{\theta}}}({\bm{a}}_{1}|{\bm{s}}_{0})$ and $d_{{\pi_{\bm{\theta}}}}({\bm{s}}_{0},{\bm{a}}_{0})=d_{{\pi_{\bm{\theta}}}}({
\bm{s}}_{0},{\bm{a}}_{1})$, the expected gradient will increase the probability of sampling the action with the larger advantage (${\bm{a}}_{0}$).
With on-policy sampling, after 10 visits to ${\bm{s}}_{0}$, the agent will sample both actions 5 times in expectation.
However, the agent may actually observe ${\bm{a}}_{0}$ 4 times and ${\bm{a}}_{1}$ 6 times.
A Monte Carlo estimate of the policy gradient would then place $0.4\cdot A^{\pi_{\bm{\theta}}}({\bm{s}}_{0},{\bm{a}})=8$ weight on the gradient of ${\bm{a}}_{0}$ and $0.6\cdot A^{\pi_{\bm{\theta}}}({\bm{s}}_{0},{\bm{a}}_{1})=9$ weight on ${\bm{a}}_{1}$, thus decreasing the probability of sampling the optimal ${\bm{a}}_{0}$ action.

Sampling error in on-policy sampling vanishes as the size of the batch of data used to estimate the gradient tends toward infinity.
However, the preceding example suggests a simple strategy that would eliminate sampling error with finite data: have the agent adapt its probability on the next action it takes based on what actions it has already sampled.
Continuing with our example, suppose the agent has visited ${\bm{s}}_{0}$ 9 times and sampled ${\bm{a}}_{0}$ 4 times and ${\bm{a}}_{1}$ 5 times.
With on-policy sampling, the agent may observe ${\bm{a}}_{1}$ again upon the next visit to ${\bm{s}}_{0}$.
Alternatively, the agent could sample its next action from a distribution that puts probability 1 on ${\bm{a}}_{0}$ and consequently produce an aggregate batch of data that contains both actions in their expected frequency.
While this adaptive method is an off-policy sampling method, it produces data that exactly matches the on-policy distribution and will thus produce a more accurate gradient.

This example suggests that we can heuristically reduce sampling error by taking the most under-sampled action at a given state.
Under a strong assumption that the MDP had a DAG structure,
Zhong et al. [] proved that this heuristic results in the empirical distribution of states and actions in a fixed-horizon MDP converging to $d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})$ and moreover converging at a faster rate than on-policy sampling.
We remove the limiting DAG assumption with the following result:

###### Proposition 1 .

Assume that data is collected with an adaptive behavior policy that always takes the most under-sampled action in each state, $s$, with respect to policy $\pi$, i.e, $a\leftarrow\arg\max_{a^{\prime}}(\pi(a^{\prime}|s)-{\pi_{\mathcal{D}}}(a^{
\prime}|s))$, where ${\pi_{\mathcal{D}}}$ is the empirical policy after $m$ state-action pairs have been collected.
Assume that ${\mathcal{S}}$ and ${\mathcal{A}}$ are finite and that the Markov chain induced by $\pi$ is irreducible.
Then we have that the empirical state visitation distribution, $d_{m}$, converges to the state distribution of $\pi$, $d_{\pi}$, with probability $1$:

$$ $\forall s,\lim_{m\rightarrow\infty}d_{m}(s)=d_{\pi}(s).$ $$

###### Proof.

See Appendix A.
∎

While adaptively sampling the most under-sampled action can reduce sampling error, this heuristic is difficult to implement in practice.
In tasks with continuous states and actions, the $\operatorname*{arg\,max}$ often has no closed-form solution, and the empirical policy can be expensive to compute at every timestep.
Building upon the concepts discussed in this section, the following section presents a scalable adaptive sampling algorithm that reduces sampling error in on-policy policy gradient learning.

###### Proposition 1 .

###### Proof.

### 5 Proximal Robust On-Policy Sampling for Policy Gradient Algorithms

Our goal is to develop an adaptive, off-policy sampling algorithm that reduces sampling error in on-policy data collection for on-policy policy gradient algorithms.
We outline a general framework for on-policy learning with an adaptive behavior policy in Algorithm 1.
In this framework, the behavior policy $\pi_{\bm{\phi}}$ and target policy $\pi_{\bm{\theta}}$ are initially the same.
The behavior policy collects a batch of $m$ transitions, adds the batch to a data buffer ${\mathcal{D}}$, and then updates its weights such that the next batch it collects reduces sampling error in ${\mathcal{D}}$ with respect to the target policy $\pi_{\bm{\theta}}$ (Lines 7-10).
Every $n$ steps (with $n>m$), the agent updates its target policy with data from ${\mathcal{D}}$ (Line  11).
We refer to $m$ and $n$ as the behavior batch size and the target batch size, respectively.

Figure: Algorithm 1 On-policy policy gradient algorithm with adaptive sampling

A subtle implication of adaptive sampling is that it can correct sampling error in any empirical data distribution – even one generated by a different policy.
Rather than discarding off-policy data from old policies – as is commonly done in on-policy learning – we let the data buffer hold up to $b$ target batches ($bn$ transitions) and call $b$ the buffer size.
If $b>1$, then ${\mathcal{D}}$ will contain historic off-policy data used in previous target policy updates.
Regardless of how $b$ is set, the role of the behavior policy is to continually adjust action probabilities for new samples so that the aggregate data distribution of ${\mathcal{D}}$ matches the expected on-policy distribution of the current target policy (Line 10).
Implementing Line 10 is the core challenge we address in the remainder of this section.

To ensure that the empirical distribution of ${\mathcal{D}}$ matches the expected on-policy distribution, updates to $\pi_{\bm{\phi}}$ should attempt to increase the probability of actions which are currently under-sampled with respect to $\pi_{\bm{\theta}}$.
Zhong et al. [] recently developed a method called Robust On-policy Sampling (ros) for making such updates.
In particular, the gradient $\nabla_{\bm{\phi}}{\mathcal{L}}:=-\nabla_{\bm{\phi}}\sum_{({\bm{s}},{\bm{a}})
\in{\mathcal{D}}}\log\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})$ when evaluated at ${\bm{\phi}}={\bm{\theta}}$ provides a direction to change ${\bm{\phi}}$ such that under-sampled actions have their probabilities increased. Thus a single step of gradient ascent will increase the probability of under-sampled actions.(^1^11To add further intuition for this update, note that it is the opposite of a gradient ascent step on the log likelihood of ${\mathcal{D}}$. When starting at ${\bm{\theta}}$, gradient ascent on the data log likelihood will increase the probability of actions that are over-sampled relative to $\pi_{\bm{\theta}}$. Hence, the ros update changes ${\bm{\phi}}$ in the opposite direction.)
In theory and in simple rl policy evaluation tasks, this update was shown to improve the rate at which the empirical data distribution converges to the on-policy distribution – even when the empirical data distribution contains off-policy data.
Unfortunately, there are two main challenges that render ros unsuitable for Line 10 in Algorithm 1.

Figure: Algorithm 2 props Update

Challenge 1: Destructively large policy updates. Since the buffer ${\mathcal{D}}$ may contain data collected from older target policies, some samples in ${\mathcal{D}}$ may be very off-policy with respect to the current target policy such that $\log\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})$ is large and negative.
Since $\nabla_{\bm{\phi}}\log\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})$ increases in magnitude as $\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})$ tends towards zero, ros incentivizes the agent to continually decrease the probability of these actions despite being extremely unlikely under the current target policy.
Thus, off-policy samples can produce destructively large policy updates.

Challenge 2: Improper handling of continuous actions. In a continuous-action task, ros may produce behavior policies that increase sampling error.
A continuous-action task policy $\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})$ is typically parameterized as a Gaussian ${\mathcal{N}}({\bm{\mu}}({\bm{s}}),\Sigma({\bm{s}}))$ with mean ${\bm{\mu}}({\bm{s}})$ and diagonal covariance matrix $\Sigma({\bm{s}})$.
Since actions in the tail of the Gaussian far from the mean will usually be under-sampled, the ros update will continually push the components of ${\bm{\mu}}({\bm{s}})$ towards $\pm\infty$ and the diagonal components of $\Sigma({\bm{s}})$ towards 0 0 to increase the probability of sampling these actions.
The result is a degenerate behavior policy that is so far from the target policy that sampling from it increases sampling error.
We illustrate this scenario with 1-dimensional continuous actions in Figure 6 of Appendix B.

To address these challenges, we propose a new behavior policy update.
Observing that the gradient of the ros loss $\nabla_{\bm{\phi}}{\mathcal{L}}=\nabla_{\bm{\phi}}\log\pi_{\bm{\phi}}({\bm{a}}
|{\bm{s}})\rvert_{{\bm{\phi}}={\bm{\theta}}}$ resembles the policy gradient (Eq. 2) with $A^{\pi_{{\bm{\theta}}}}({\bm{s}},{\bm{a}})=-1,\forall({\bm{s}},{\bm{a}})$, we draw inspiration from ppo [Schulman et al., ] to address these challenges.
Rather than using the ros objective to update ${\bm{\phi}}$, we use a clipped surrogate objective:

$$ $\begin{split}{\mathcal{L}}_{\textsc{clip}}{}({\bm{s}},{\bm{a}},{\bm{\phi}},{ \bm{\theta}},\epsilon_{\textsc{props}}{}{})=\min\Bigg{[}-\frac{\pi_{\bm{\phi}} ({\bm{a}}|{\bm{s}})}{\pi_{{\bm{\theta}}}({\bm{a}}|{\bm{s}})},-\mathtt{clip} \left(\frac{\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})}{\pi_{{\bm{\theta}}}({\bm{a}}|{ \bm{s}})},1-\epsilon_{\textsc{props}}{}{},1+\epsilon_{\textsc{props}}{}{} \right)\Bigg{]}.\end{split}$ (5) $$

Table 1 in Appendix B summarizes the behavior of ${\mathcal{L}}_{\text{{clip}{}}}$.
Intuitively, this objective incentivizes the agent to decrease the probability of observed actions by at most a factor of $1-\epsilon_{\textsc{props}}{}{}$.
Let $g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})=\frac{\pi_{\bm{\phi}}({\bm{a}}|
{\bm{s}})}{\pi_{{\bm{\theta}}}({\bm{a}}|{\bm{s}})}$.
When $g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})<1-\epsilon_{\textsc{props}}{}{}$, this objective is clipped at $-(1-\epsilon_{\textsc{props}}{}{})$.
The loss gradient $\nabla_{\bm{\phi}}{\mathcal{L}}_{\text{{clip}{}}}$ becomes zero, and the $({\bm{s}},{\bm{a}})$ pair has no effect on the policy update.
When $g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})>1-\epsilon_{\textsc{props}}{}{}$, clipping does not apply, and the (non-zero) gradient $\nabla_{\bm{\phi}}{\mathcal{L}}_{\text{{clip}{}}}$ points in a direction that decreases the probability of $\pi_{\bm{\phi}}({\bm{a}}|{\bm{s}})$.
As in the ppo update, this clipping mechanism avoids destructively large policy updates and permits us to perform many epochs of minibatch updates with the same batch of data.

To address the second challenge and prevent degenerate behavior policies, we introduce an auxiliary loss that incentivizes the agent to minimize the kl divergence between the behavior policy and target policy at states in the observed data.
The full props objective is then:

$$ $\begin{split}{\mathcal{L}}_{\textsc{props}}{}({\bm{s}},{\bm{a}},{\bm{\phi}},{ \bm{\theta}},\epsilon_{\textsc{props}}{}{},\lambda)={\mathcal{L}}_{\textsc{ clip}}{}({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}},\epsilon_{\textsc{props}} {})-\lambda D_{\textsc{kl}}(\pi_{{\bm{\theta}}}(\cdot|{\bm{s}})||\pi_{\bm{\phi }}(\cdot|{\bm{s}}))\end{split}$ (6) $$

where $\lambda$ is a regularization coefficient quantifying a trade-off between maximizing ${\mathcal{L}}_{\text{{props}{}}}$ and minimizing $D_{\textsc{kl}}$.
We provide full pseudocode for the props update in Algorithm 2.
Like ros, we set the behavior policy parameters ${\bm{\phi}}$ equal to the target policy parameters at the start of each behavior update, and then make a local adjustment to $\phi$ to increase the probabilities of under-sampled actions.
We stop the props update when $D_{\textsc{kl}}{}(\pi_{\bm{\theta}}||\pi_{\bm{\phi}})$ reaches a chosen threshold $\delta_{\textsc{props}}{}$.
This technique further safeguards against large policy updates and is used in widely adopted implementations of ppo [Raffin et al., , Liang et al., ].
The props update allows us to efficiently learn a behavior policy that keeps the distribution of data in the buffer close to the expected distribution of the target policy.

### 6 Experiments

The central goal of our work is to understand whether on-policy policy gradient algorithms are more data efficient learners when they use on-policy data acquired without on-policy sampling.
Towards this goal, we design experiments to answer the two questions:

- Q1:
Does props achieve lower sampling error than on-policy sampling during policy gradient learning (i.e., does props produce data that is more on-policy than on-policy sampling produces)?
- Q2:
Does props increase the fraction of training runs that converge to high-return policies and improve the data efficiency of on-policy policy gradient algorithms?

Our empirical analysis focuses on continuous-state continuous-action MuJoCo benchmark tasks and a tabular 5x5 GridWorld task (Figure 2(a)).
We additionally consider three continuous-state discrete-action tasks: CartPole-v1, LunarLander-v2, and Discrete2D100-v0 – a 2D navigation task with 100 discrete actions.
Due to space constraints, we include these tasks in Appendix D.4.

Figure: (a) GridWorld
Refer to caption: extracted/5905669/figures/gridworld.png

#### 6.1 Correcting Sampling Error for a Fixed Target Policy

We first study how quickly props decreases sampling error when the target policy is fixed.
This setting is similar to the policy evaluation setting considered by  Zhong et al. [].
As such, we provide two baselines for comparison: on-policy sampling and ros.

Sampling error metrics. In GridWorld, we compute sampling error as the total variation (TV) distance between the empirical state-action visitation $d_{\mathcal{D}}({\bm{s}},{\bm{a}})$ distribution – denoting the proportion of times $({\bm{s}},{\bm{a}})$ appears in buffer ${\mathcal{D}}$ – and the true state-action visitation distribution under the agent’s policy: $\sum_{({\bm{s}},{\bm{a}})\in{\mathcal{D}}}|d_{\mathcal{D}}({\bm{s}},{\bm{a}})-
d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})|$.
In continuous MuJoCo tasks where it is difficult to compute $d_{\mathcal{D}}({\bm{s}},{\bm{a}})$, we follow Zhong et al. [] and measure sampling error using the kl-divergence $D_{\textsc{kl}{}}(\pi_{\mathcal{D}}||\pi_{\bm{\theta}})$ between the empirical policy $\pi_{\mathcal{D}}$ and the target policy $\pi_{\bm{\theta}}$.
We estimate $\pi_{\mathcal{D}}$ as the maximum likelihood estimate under data in the buffer via stochastic gradient ascent.
Further details on how we compute $\pi_{\mathcal{D}}$ are in Appendix C.

Since it is straightforward to compute the true policy gradient in the GridWorld task, we additionally investigate how sampling error reduction affects gradient estimation by measuring the cosine similarity between the empirical policy gradient
$\nabla_{\bm{\theta}}\widehat{J}({\bm{\theta}})$
and the true policy gradient.
As the empirical gradient aligns more closely with the true gradient, the cosine similarity approaches 1.

Figure: (a) Fixed target policy.
Refer to caption: extracted/5905669/figures/sampling_error/se_expert_one.png

Experimental setup. In all tasks, we use a buffer with capacity of $\lfloor T/2\rfloor$ samples, where $T$ is the total number of samples collected by the agent.
Thus, we expect sampling error to decrease over the first $\lfloor T/2\rfloor$ samples and then remain roughly constant afterwards once the buffer is full.
We use randomly initialized target policies.
Further experimental details such as hyperparameter tuning are described in Appendix D.2.

Results. As shown in Figure 2(b) and 2(c), in GridWorld, props decreases sampling error faster than on-policy sampling, resulting in more accurate policy gradient estimates.
props and ros perform similarly, though this behavior is expected; in a tabular setting with a fixed target policy (i.e. there is no off-policy data in the buffer), we do not encounter Challenge 1 and 2 described in the previous section.
In continuous MuJoCo tasks where Challenge 2 arises, props decreases sampling error faster than on-policy sampling and ros (Figure 3(a)).
In fact, ros shows little to no improvement over on-policy sampling in every MuJoCo task.
This limitation of ros matches the findings of Zhong et al. [] where ros struggled to reduce sampling error even in low-dimensional continuous-action tasks.
Moreover, props decreases sampling error without clipping and regularization, emphasizing how adaptive off-policy sampling alone decreases sampling error.
Due to space constraints, we include results for the remaining environments in Appendix D.2.
We additionally include experiments using a fixed, randomly initialized target policy as well as ablation studies isolating the effects of props’s objective clipping and regularization in Appendix D.2.
Results with a random target policy are qualitatively similar to those in Figure 3(a), and we observe that clipping and regularization both individually help reduce sampling error.
In Appendix D.1, we show empirically that the empirical state-action visitation distributions produced by props (and ros) are unbiased and lower variance than the distribution produced by on-policy sampling.

#### 6.2 Correcting Sampling Error During RL Training

We are ultimately interested in understanding how replacing on-policy sampling with props affects the data efficiency of on-policy learning, where the target policy is continually updated.
In the following experiments, we train rl agents with props and on-policy sampling to evaluate (1) the data efficiency of training, (2) the distribution of returns achieved at the end of training, and (3) the sampling error throughout training.
We use the same sampling error metrics described in the previous section and measure data efficiency as the return achieved within a fixed training budget.
Since ros [Zhong et al., ] is computationally expensive and fails to reduce sampling error in MuJoCo tasks even with a fixed policy, we omit it from MuJoCo experiments.

Experimental setup. We use ppo [Schulman et al., ] to update the target policy.
We consider two baseline methods for providing data to compute ppo updates: (1) vanilla ppo with on-policy sampling, and (2) ppo with on-policy sampling and a buffer of size $b$ (ppo-buffer).
ppo-buffer is a naive method for improving data efficiency of on-policy algorithms by reusing off-policy data collected by old target policies as if it were on-policy data.
Although ppo-buffer computes biased gradients, it has been successfully applied in difficult learning tasks [Berner et al., ].
Since props and ppo-buffer have access to the same amount of data for each policy update, any performance difference between these two methods can be attributed to differences in how they sample actions during data collection.

In MuJoCo experiments, we set $b=2$ such that agents retain a batch of data for one extra iteration before discarding it.
In GridWorld, we use $b=1$ and discard all historic data.
Since props and ppo-buffer compute target policy updates with $b$ times as much learning data as ppo, we integrate this extra data by increasing the minibatch size for target and behavior policy updates by a factor of $b$. Further experimental details including hyperparameter tuning are described in Appendix D.6.
For MuJoCo tasks, we plot the interquartile mean (IQM) return throughout training as well as the distribution of returns achieved at the end of training (i.e., the performance profile) [Agarwal et al., ].
For GridWorld, we plot the agent’s success rate, the fraction of times it finds the optimal goal.

Figure: (a) Training curves.
Refer to caption: extracted/5905669/figures/return_iqm.png

Results. As shown in Figure 5(a), on-policy sampling has approximately a 77% success rate on GridWorld, whereas props and ros achieve 100% success rate.
In Figure 4(a), props achieves higher return than both ppo and ppo-buffer throughout training in all MuJoCo tasks except Ant-v4 where props dips slightly below ppo’s return near the end of training.
Moreover, in Figure 4(b), the performance profile of props almost always lies above the performance profiles of ppo and ppo-buffer, indicating that any given run of props is more like to obtain a higher return than ppo-buffer.
Thus, we affirmatively answer Q2 posed at the start of this section: props increases the fraction of training runs with high return and increases data efficiency.

Figure: (a) Success rate.
Refer to caption: extracted/5905669/figures/gridworld/returns.png

In Appendix D.4, we provide additional experiments demonstrating that props improves data efficiency in discrete-action tasks.
We additionally ablate the buffer size $b$ in Appendix D.3.
We find that data efficiency may decrease with a larger buffer size.
Intuitively, the more historic data kept around, the more data that must be collected to impact the aggregate data distribution.

Having established that props improves data efficiency, we now investigate if props is appropriately adjusting the data distribution of the buffer by comparing the sampling error achieved throughout training with props and ppo-buffer.
Training with props produces a different sequence of target policies than training with ppo-buffer produces.
To provide a fair comparison, we compute sampling error for ppo-buffer using the target policy sequence produced by props.
More concretely, we fill a second buffer with on-policy samples collected by the target policies produced while training with props and then compute the sampling error using data in this buffer.

As shown in Figure 5(b), props achieves lower sampling error than on-policy sampling with a buffer in Humanoid-v4.
Due to space constraints, we provide sampling error curves for the remaining MuJoCo environments in Appendix D.3.
In GridWorld, props and ros reduce sampling error in the first 300 steps and closely matches on-policy sampling afterwards.
We use a batch size of 80 in these experiments, and as the target policy becomes more deterministic, larger batch sizes are needed to observe differences between props and on-policy sampling.(^2^22When the target policy is deterministic, we always have zero sampling error, and props will exactly match on-policy sampling.)

We additionally ablate the effects of the clipping coefficient $\epsilon_{\textsc{props}}{}{}$ and regularization coefficient $\lambda$ in Appendix D.3.
Without clipping or without regularization, props often achieves greater sampling error than on-policy sampling, indicating that both help to keep sampling error low.
Moreover, data efficiency generally decreases when we remove clipping or regularization, showing both are essential to props.
Thus, we affirmatively answer Q1 posed at the start of this section: props achieves lower sampling error than on-policy sampling when the target policy is fixed and during rl training.

### 7 Discussion

This work has shown that adaptive, off-policy sampling can be used to reduce sampling error in data collected throughout rl training and improve the data efficiency of on-policy policy gradient algorithms.
We have introduced an algorithm that scales adaptive off-policy sampling to continuous control rl benchmarks and enables tracking of the changing on-policy distribution.
By integrating this data collection procedure into the popular ppo algorithm, the main conclusion of our analysis is that on-policy learning algorithms learn most efficiently with on-policy data, not on-policy sampling.
In this section, we discuss limitations of our work and present opportunities for future research.

props builds upon the ros algorithm of Zhong et al. [].
While Zhong et al. [] focused on theoretical analysis and policy evaluation in small scale domains, we chose to focus on empirical analysis with policy learning in standard rl benchmarks.
An important direction for future work would be theoretical analysis of props, in particular whether props also enjoys the same faster convergence rate that was shown for ros relative to on-policy sampling.

A limitation of props is that the update indiscriminately increases the probability of under-sampled actions without considering their importance in gradient computation.
For instance, if an under-sampled action has zero advantage, it has no impact on the gradient and need not be sampled.
An interesting direction for future work could be to prioritize correcting sampling error for $({\bm{s}},{\bm{a}})$ that have the largest influence on the gradient estimate, i.e., large advantage (positive or negative).

Beyond these more immediate directions, our work opens up other opportunities for future research.
A less obvious feature of the props behavior policy update is that it can be used track the empirical data distribution of any desired policy, not only that of the current policy.
This feature means props has the potential to be integrated into off-policy rl algorithms and used so that the empirical distribution more closely matches a desired exploration distribution.
Thus, props could be used to perform focused exploration without explicitly tracking state and action counts.

### 8 Conclusion

In this paper, we ask whether on-policy policy gradient methods are more data efficient using on-policy sampling or on-policy data acquired without on-policy sampling.
To answer this question, we introduce an adaptive, off-policy sampling method for on-policy policy gradient learning that collects data such that the empirical distribution of sampled actions closely matches the expected on-policy data distribution at observed states.
Our method, Proximal Robust On-policy Sampling (props), periodically updates the data collecting behavior policy so as to increase the probability of sampling actions that are currently under-sampled with respect to the on-policy distribution.
Furthermore, rather than discarding collected data after every policy update, props permits more data efficient on-policy learning by using data collection to adjust the distribution of previously collected data to be approximately on-policy.
We replace on-policy sampling with props to generate data for the widely-used ppo algorithm and empirically demonstrate that props produces data that more closely matches the expected on-policy distribution and yields more data efficient learning compared to on-policy sampling.

## Appendix

### Appendix A Theoretical Results

In this section, we present the proof Proposition 2.
We use $d_{m}$, $\pi_{m}$, and $p_{m}$ as the empirical state visitation distribution, empirical policy, and empirical transition probabilities after $m$ state-action pairs have been taken, respectively.
That is, $d_{m}(s)$ is the proportion of the $m$ states that are $s$, $\pi_{m}(a|s)$ is the proportion of the time that action $a$ was observed in state $s$, and $p_{m}(s^{\prime}|s,a)$ is the proportion of the time that the state changed to $s^{\prime}$ after action $a$ was taken in state $s$.

###### Proposition 2 .

Assume that data is collected with an adaptive behavior policy that always takes the most under-sampled action in each state, $s$, with respect to policy $\pi$, i.e, $a\leftarrow\arg\max_{a^{\prime}}(\pi(a^{\prime}|s)-\pi_{m}(a^{\prime}|s))$.
We further assume that ${\mathcal{S}}$ and ${\mathcal{A}}$ are finite.
Then we have that the empirical state visitation distribution, $d_{m}$, converges to the state distribution of $\pi$, $d_{\pi}$, with probability $1$:

$$ $\forall s,\lim_{m\rightarrow\infty}d_{m}(s)=d_{\pi}(s).$ $$

###### Proof.

The proof of this theorem builds upon Lemma 1 and 2 by Zhong et al. [].
Note that these lemmas superficially concern the ROS method whereas we are interested in data collection by taking the most under-sampled action at each step.
However, as stated in the proof by Zhong et al. [], these methods are equivalent under an assumption they make about the step-size parameter of the ROS method.
Thus, we can immediately adopt these lemmas for this proof.

Under Lemma 1 of Zhong et al. [], we have that $\lim_{m\rightarrow\infty}\pi_{m}(a|s)=\pi(a|s)$ for any state $s$ under this adaptive data collection procedure.
We then have the following $\forall s$:

$$ $\displaystyle\lim_{m\rightarrow\infty}d_{m}(s)$ $\displaystyle\overset{(a)}{=}\lim_{m\rightarrow\infty}\sum_{\tilde{s}}\sum_{ \tilde{a}}p_{m}(s|\tilde{s},\tilde{a})\pi_{m}(\tilde{a}|\tilde{s})d_{m}(\tilde {s})$ $\displaystyle=\sum_{\tilde{s}}\sum_{\tilde{a}}\lim_{m\rightarrow\infty}p_{m}(s |\tilde{s},\tilde{a})\pi_{m}(\tilde{a}|\tilde{s})d_{m}(\tilde{s})$ $\displaystyle=\sum_{\tilde{s}}\sum_{\tilde{a}}\lim_{m\rightarrow\infty}p_{m}(s |\tilde{s},\tilde{a})\lim_{m\rightarrow\infty}\pi_{m}(\tilde{a}|\tilde{s})\lim _{m\rightarrow\infty}d_{m}(\tilde{s})$ $\displaystyle\overset{(b)}{=}\sum_{\tilde{s}}\sum_{\tilde{a}}p(s|\tilde{s}, \tilde{a})\pi(\tilde{a}|\tilde{s})\lim_{m\rightarrow\infty}d_{m}(\tilde{s}).$ $$

Here, (a) follows from the fact that the empirical frequency of state $s$ can be obtained by considering all possible transitions that lead to $s$.
The last line, (b), holds with probability 1 by the strong law of large numbers and Lemma 2 of Zhong et al. [].

We now have a system of $|{\mathcal{S}}|$ variables and $|{\mathcal{S}}|$ linear equations.
Define variables $x(s)\coloneqq\lim_{m\rightarrow\infty}d_{m}(s)$ and let ${\bm{x}}\in\mathbf{R}^{|{\mathcal{S}}|}$ be the vector of these variables.
We then have $x=P^{\pi}x$ where $P^{\pi}\in\mathbf{R}^{|{\mathcal{S}}|\times|{\mathcal{S}}|}$ is the transition matrix of the Markov chain induced by running policy $\pi$.
Assuming that this Markov chain is irreducible, $d_{\pi}$ is the unique solution to this system of equations and hence $\lim_{m\rightarrow\infty}d_{m}(s)=d_{\pi}(s),\forall s$.

∎

###### Proposition 2 .

###### Proof.

### Appendix B PROPS Implementation Details

In this appendix, we summarize the behavior of props’s clipping mechanism in Table 1 and additionally describe two relevant implementation details for the props update (Algorithm 2).

- 1.
props update:
The props update adapts the behavior policy to reduce sampling error in the buffer ${\mathcal{D}}$.
When performing this update with a full buffer, we exclude the oldest batch of data collected by the behavior policy (i.e., the $m$ oldest transitions in ${\mathcal{D}}$); this data will be evicted from the buffer before the next behavior policy update and thus does not contribute to sampling error in ${\mathcal{D}}$.
- 2.
Behavior policy class:
We compute behavior policies from the same policy class used for target policies.
In particular, we consider Gaussian policies which output a mean $\mu({\bm{s}})$ and a variance $\sigma^{2}({\bm{s}})$ and then sample actions ${\bm{a}}\sim\pi(\cdot|{\bm{s}})\equiv{\mathcal{N}}(\mu({\bm{s}}),\sigma^{2}({
\bm{s}}))$.
In principle, the target and behavior policy classes can be different.
However, using the same class for both policies allows us to easily initialize the behavior policy equal to the target policy at the start of each update.
This initialization is necessary to ensure the props update increases the probability of sampling actions that are currently under-sampled with respect to the target policy.

Figure: Figure 6: In this example, $\pi(\cdot|{\bm{s}})={\mathcal{N}}(0,1)$. After several visits to ${\bm{s}}$, all sampled actions (blue) satisfy $a>0$ so that actions $a<0$ are under-sampled. Without regularization, props will attempt to increase the probabilities of under-sampled action in the tail of target policy distribution (green). The regularization term in the props objective ensures the behavior policy remains close to target policy.
Refer to caption: extracted/5905669/figures/regularization.png

**Table 1: Behavior of props’s clipped surrogate objective (Eq. 5).**
| $g({\bm{s}}\,{\bm{a}},{\bm{\phi}},{\bm{\theta}})>0$ | Is the objective clipped? | Return value of $\min$ | Gradient |
| --- | --- | --- | --- |
| $g({\bm{s}}\,{\bm{a}},{\bm{\phi}},{\bm{\theta}})\in[1-\epsilon_{\textsc{props}}<br>{}{},1+\epsilon_{\textsc{props}}{}{}]$ | No | $-g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})$ | $\nabla_{\bm{\phi}}{\mathcal{L}}_{\text{{clip}{}}}$ |
| $g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})>1+\epsilon_{\textsc{props}}{}{}$ | No | $-g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})$ | $\nabla_{\bm{\phi}}{\mathcal{L}}_{\text{{clip}{}}}$ |
| $g({\bm{s}},{\bm{a}},{\bm{\phi}},{\bm{\theta}})<1-\epsilon_{\textsc{props}}{}{}$ | Yes | $-(1-\epsilon_{\textsc{props}}{}{})$ | $\mathbf{0}$ |

### Appendix C Computing Sampling Error

We claim that props improves the data efficiency of on-policy learning by reducing sampling error in the agent’s buffer ${\mathcal{D}}$ with respect to the agent’s current (target) policy.
To measure sampling error, we use the kl-divergence $D_{\textsc{kl}{}}(\pi_{\mathcal{D}}||\pi_{\bm{\theta}})$ between the empirical policy $\pi_{\mathcal{D}}$ and the target policy $\pi_{\bm{\theta}}$ which is the primary metric Zhong et al. [] used to show ros reduces sampling error:

$$ $D_{\textsc{kl}{}}(\pi_{\mathcal{D}}||\pi_{\bm{\theta}})=\mathbb{E}_{{\bm{s}} \sim{\mathcal{D}},{\bm{a}}\sim\pi_{\mathcal{D}}(\cdot|{\bm{s}})}\left[\log \left(\frac{\pi_{\mathcal{D}}({\bm{a}}|{\bm{s}})}{\pi_{\bm{\theta}}({\bm{a}}|{ \bm{s}})}\right)\right].$ (7) $$

We compute a parametric estimate of $\pi_{\mathcal{D}}$ by maximizing the log-likelihood of ${\mathcal{D}}$ over the same policy class used for $\pi_{\bm{\theta}}$.
More concretely, we let ${\bm{\theta}}^{\prime}$ be the parameters of neural network with the same architecture as $\pi_{\bm{\theta}}$ train and then compute:

$$ ${\bm{\theta}}_{\textsc{MLE}}=\operatorname*{arg\,max}_{{\bm{\theta}}^{\prime}} \sum_{({\bm{s}},{\bm{a}})\in{\mathcal{D}}}\log\pi_{{\bm{\theta}}^{\prime}}({ \bm{a}}|{\bm{s}})$ (8) $$

using stochastic gradient ascent.
After computing ${\bm{\theta}}_{\textsc{MLE}}$, we then estimate sampling error using the Monte Carlo estimator:

$$ $D_{\textsc{kl}{}}(\pi_{\mathcal{D}}||\pi_{\bm{\theta}})\approx\sum_{({\bm{s}}, {\bm{a}})\in{\mathcal{D}}}\left(\log\pi_{{\bm{\theta}}_{\textsc{MLE}}}({\bm{a} }|{\bm{s}})-\log\pi_{\bm{\theta}}({\bm{a}}|{\bm{s}})\right).$ (9) $$

### Appendix D Additional Experiments

In this appendix, we include additional experiments and ablations.

#### D.1 Bias and Variance of props

Figure: (a) Sampling error bias
Refer to caption: extracted/5905669/figures/gridworld/sa_occupancy_bias.png

In Figure 7, we investigate the bias and variance of the empirical state-action visitation distribution $d^{\mathcal{D}}({\bm{s}},{\bm{a}})$ under props, ros, and on-policy sampling.
Our experimental setup is similar to the setup in Section 6.1.
We collect $T$ samples with a fixed policy that samples actions uniformly at random in each state.
We compute the bias and variance averaged over all $({\bm{s}},{\bm{a}})\in{\mathcal{S}}\times{\mathcal{A}}$ computed as follows:

$$ $\text{bias}=\frac{1}{|{\mathcal{S}}\times{\mathcal{A}}|}\sum_{({\bm{s}},{\bm{a }})\in{\mathcal{S}}\times{\mathcal{A}}}\left(\mathbb{E}\left[d^{{\mathcal{D}}} ({\bm{s}},{\bm{a}})\right]-d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})\right)$ (10) $$

$$ $\text{variance}=\frac{1}{|{\mathcal{S}}\times{\mathcal{A}}|}\sum_{({\bm{s}},{ \bm{a}})\in{\mathcal{S}}\times{\mathcal{A}}}\mathbb{E}\left[\left(d^{{\mathcal {D}}}({\bm{s}},{\bm{a}})-d_{\pi_{\bm{\theta}}}({\bm{s}},{\bm{a}})\right)^{2}\right]$ (11) $$

As shown in Figure 7, the visitation distribution under props and ros empirical have near zero bias (note that the vertical axis has scale $10^{-18}$) and have lower variance than on-policy sampling.

#### D.2 Correcting Sampling Error for a Fixed Target Policy

Figure: Figure 8: Sampling error with a fixed, expert target policy. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Refer to caption: extracted/5905669/figures/sampling_error/se_fixed_expert.png

Figure: Figure 9: Sampling error with a fixed, randomly initialized target policy. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Refer to caption: extracted/5905669/figures/sampling_error/se_fixed_random.png

Figure: Figure 10: Sampling error ablations with a fixed, expert target policy. Here, “no clipping” refers to setting $\epsilon_{\textsc{props}}{}{}=\infty$, and “no regularization” refers to setting $\lambda=0$. Solid curves denote the mean over 5 seeds, and shaded regions denote 95% bootstrap confidence intervals.
Refer to caption: extracted/5905669/figures/sampling_error/expert_ablation.png

In this appendix, we expand upon results presented in Section 6.1 of the main paper and
provide additional experiments investigating the degree to which props reduces sampling error with respect to a fixed target policy.
We include empirical results for all six MuJoCo benchmark tasks as well as ablation studies investigating the effects of clipping and regularization.

We tune props and ros using a hyperparameter sweep.
For props, we consider learning rates in $\{10^{-3},10^{-4}\}$, regularization coefficients $\lambda\in\{0.01,0.1,0.3\}$, and props target KLs in $\delta_{\textsc{props}}{}\in\{0.05,0.1\}$.
We fix $\epsilon_{\textsc{props}}{}=0.3$ across all experiments.
For ros, we consider learning rates in $\{10^{-3},10^{-4},10^{-5}\}$.
We report results for the hyperparameters yielding the lowest sampling error.

Figure 8 and 9 show sampling error computed with a fixed expert and randomly initialized target policy, respectively.
We see that props achieves lower sampling error than both ros and on-policy sampling across all tasks.
ros shows little to no improvement over on-policy sampling, highlighting the difficulty of applying ros to higher dimensional tasks with continuous actions.

Figure 10 ablates the effects of props’s clipping mechanism and regularization on sampling error reduction.
We ablate clipping by setting $\epsilon_{\textsc{props}}{}{}=\infty$, and we ablate regularization by setting $\lambda=0$.
We use a fixed expert target policy and use the same tuning procedure described earlier in this appendix.
In all tasks, props achieves higher sampling error without clipping nor regularization than it does with clipping and regularization.
However, it nevertheless outperforms on-policy sampling in all tasks except Hopper where it matches the performance of on-policy sampling.
Only including regularization slightly decreases sampling error, whereas clipping alone produces sampling error only slightly higher than that achieved by props with both regularization and clipping.
These observations indicate that while regularization in is helpful, clipping has a stronger effect on sampling error reduction than regularization when the target policy is fixed.

Figure: Figure 11: Sampling error throughout rl training. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Refer to caption: extracted/5905669/figures/sampling_error/se_rl.png

Figure: Figure 12: Sampling error throughout rl training without clipping the props objective. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Refer to caption: extracted/5905669/figures/sampling_error/se_updating_target_no_clip.png

Figure: Figure 13: Sampling error throughout rl training without regularizing the props objective. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Refer to caption: extracted/5905669/figures/sampling_error/se_updating_target_no_regularization.png

#### D.3 Correcting Sampling Error During RL Training

In this appendix, we include additional experiments investigating the degree to which props reduces sampling error during rl training, expanding upon results presented in Section 6.2 of the main paper.
We include sampling error curves for all six MuJoCo benchmark tasks and additionally provide ablation studies investigating the effects of clipping and regularization on sampling error reduction and data efficiency in the rl setting.

As shown in Fig 11, props achieves lower sampling error than on-policy sampling throughout training in 5 out of 6 tasks.
We observe that props increases sampling error but nevertheless improves data efficiency in HalfCheetah as shown in Figure 4(a).
This result likely arises from our tuning procedure in which we selected hyperparameters yielding the largest return.
Although lower sampling error intuitively correlates with increased data efficiency, it is nevertheless possible to achieve high return without reducing sampling error.

In our next set of experiments, we ablate the effects of props’s clipping mechanism and regularization on sampling error reduction and data efficiency.
We ablate clipping by tuningrl agents with $\epsilon_{\textsc{props}}{}{}=\infty$, and we ablate regularization by tuning rl agents with $\lambda=0$.
Figure 12 and Figure 13 show sampling error curves without clipping and without regularization, respectively.
Without clipping, props achieves larger sampling than on-policy sampling in all tasks except Humanoid.
Without regularization, props achieves larger sampling error in 3 out of 6 tasks.
These observations indicate that while clipping and regularization both help reduce sampling during rl training, clipping has a stronger effect on sampling error reduction.
As shown in Figure 14 props data efficiency generally decreases when we remove clipping or regularization.

Lastly, we consider training with larger buffer sizes $b$ in Figure 15.
We find that data efficiency may decrease with a larger buffer size.
Intuitively, the more historic data kept around, the more data that must be collected to impact the aggregate data distribution.

Figure: Figure 14: IQM return over 50 seeds of props with and without clipping or regularizing the props objective. Shaded regions denote 95% bootstrap confidence intervals.
Refer to caption: extracted/5905669/figures/ablations/return_iqm_ablation.png

Figure: Figure 15: IQM return over 50 seeds for props with different buffer sizes. We exclude $b=8$ for Humanoid-v4 due to the expense of training and tuning. Shaded regions denote 95% bootstrap confidence intervals.
Refer to caption: extracted/5905669/figures/ablations/props_b_ablation_iqm.png

#### D.4 Discrete-Action Tasks

We include 3 additional discrete-action domains of varying complexity. The first two are the widely used OpenAI gym domains CartPole-v1 and LunarLander-v2 [Brockman et al., ].
The third is a 2D navigation task, Discrete2D100-v0, in which the agent must reach a randomly sampled goal. There are 100 actions, each action corresponding to different directions in which the agent can move.
From Figure 16 and 17 we observe that props with $b=2$ achieves larger returns than ppo and ppo-buffer all throughout training in all three tasks.
props with $b=1$ (no historic data) achieves larger returns than ppo all throughout training in all three tasks and even outperforms ppo-buffer in CartPole-v1 and Discrete2D100-v0 even though ppo-buffer learns from twice as much data.
Thus, props can improve data efficiency without historic data.

Figure: Figure 16: IQM return for discrete action tasks over 50 seeds. Shaded regions denote 95% bootstrap confidence intervals.
Refer to caption: extracted/5905669/figures/discrete_return_iqm.png

Figure: Figure 17: Performance profiles for discrete-action tasks over 50 seeds. Higher values correspond to more reliable convergence to high-return policies. Shaded regions denote 95% bootstrap confidence intervals.
Refer to caption: extracted/5905669/figures/discrete_performance_profile.png

Figure: Figure 18: Runtimes for props, ppo-buffer, and ppo. We report means and standard errors over 3 independent runs.
Refer to caption: extracted/5905669/figures/runtimes.png

#### D.5 Runtime Comparisons

Figure 18 shows runtimes for props, ppo-buffer, and ppo averaged over 3 runs.
We trained all agents on a MacBook Air with an M1 CPU and use the same tuned hyperparameters used throughout the paper. props takes at most twice as long as ppo-buffer; intuitively, both props and ppo-buffer learn from the same amount of data but props learns two policies.

We note that ppo-buffer is faster than ppo is HalfCheetah-v4 because, with our tuned hyperparameters, ppo-buffer performs fewer target policy updates than ppo. In particular, ppo-buffer is updating its target policy every 4096 steps, whereas ppo is updating the target policy every 1024 steps.

#### D.6 Hyperparameter Tuning for RL Training

For all rl experiments in Section 6.2 and Appendix D.3, we tune props, ppo-buffer, and ppo separately using a hyperparameter sweep over parameters listed in Table 2 and fix the hyperparameters in Table 3 across all experiments.
Since we consider a wide range of hyperparameter values, we ran 10 independent training runs for each hyperparameter setting.
We then performed 50 independent training runs for the hyperparameters settings yielding the largest returns at the end of rl training.
We report results for these hyperparameters in the main paper.
Figure 19 shows training curves obtained from a subset of our hyperparameter sweep.

**Table 2: Hyperparameters used in our hyperparameter sweep for rl training.**
| ppo learning rate | $10^{-3},10^{-4}$, linearly annealed to 0 0 over training |
| --- | --- |
| ppo batch size $n$ | $1024,2048,4096,8192$ |
| props learning rate | $10^{-3},10^{-4}$ (and $10^{-5}$ for Swimmer) |
| props behavior batch size $m$ | $256,512,1024,2048,4096$ satisfying $m\leq n$ |
| props KL cutoff $\delta_{\textsc{props}}{}$ | $0.03,0.05,0.1$ |
| props regularizer coefficient $\lambda$ | $0.01,0.1,0.3$ |

**Table 3: Hyperparameters fixed across all experiments. We use the ppo implementation provided by CleanRL [Huang et al., ].**
| ppo number of update epochs | 10 |
| --- | --- |
| props number of update epochs | 16 |
| Buffer size $b$ | 2 target batches (also 3, 4, and 8 in Figure 15) |
| ppo minibatch size for ppo update | $bn/16$ |
| props minibatch size for props update | $bn/16$ |
| ppo and props networks | Multi-layer perceptron |
|  | with hidden layers (64,64) |
| ppo and props optimizers | Adam [Kingma and Ba, ] |
| ppo discount factor $\gamma$ | 0.99 |
| ppo generalized advantage estimation (GAE) | 0.95 |
| ppo advantage normalization | Yes |
| ppo loss clip coefficient | 0.2 |
| ppo entropy coefficient | 0.01 |
| ppo value function coefficient | 0.5 |
| ppo and props gradient clipping (max gradient norm) | 0.5 |
| ppo KL cut-off | 0.03 |
| Evaluation frequency | Every 10 target policy updates |
| Number of evaluation episodes | 20 |

Figure: (a) props kl cutoff values $\delta_{\textsc{props}}{}$.
Refer to caption: extracted/5905669/figures/sensitivity/kl.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: What is the final choice of the batch size for PPO? Only candidates 1024, 2048, 4096, and 8192 are reported.
soundness: 2
strengths: Strengths:
1. Estimating the on-policy gradient requires on-policy samples. Because we do not have access to the true gradient, using the target policy to draw on-policy samples to estimate the true gradient has an estimation error. This error is caused by under-sampled or over-sampled data. Encouraging under-sampled on-policy samples reduces the sampling error and improves gradient estimation. This paper is the first to apply this idea to the PPO algorithm.
2. This paper proposes an algorithm to learn the behavior policy to encourage data in PPO algorithms.
summary: This paper studies the problem that on-policy sampling may have sample error throughout the sampling process. Such sampling errors could be remedied by sampling using a different policy. This paper applied this idea to PPO by remediating the sampling error inside each minibatch. Specifically, this paper divides the minibatch of the PPO into more fine-grained nano batches and adjusts the sampling policy based on sampled nano batches to encourage under-sampled samples. This paper claims their results are better than PPO.
weaknesses: Weaknesses:
1. This paper adjusts the batch size of PPO to "1024,2048,4096,8192" which is much larger than the original batch size (64 samples) of PPO. This paper makes this change because they need to learn a behavior policy to encourage under-sampled data at each update step. In other words, they run a mini-PPO to encourage under-sampled data by finding a behavior policy. This drastic increase in batch size is not well-justified. In fact, it is confusing that their PPO algorithm still achieves the comparable performance of the original PPO with at least 16 times less update due to the batch size increase. I am not sure if the original PPO paper has such large room to change the batch size to 16 times larger while achieving the same performance under the same hyperparameters.
2. This paper is not well-written. The length of this paper can be greatly reduced. Currently, this paper has too many unnecessary examples and sentences. The theoretical inside of this paper is also weak. I think a good improvement direction is to discuss the relationship between the sampling error and the number of samples and how this paper can remediate the sampling error from the theoretical perspective. Some new interesting findings might also be discovered during this process.
3. This paper also introduces many fragile hyperparameters. For example, the learning rate for its proposed behavior policy ranges from 10^{-3} to 10^{-5} for an ADAM optimizer. Such a large difference in learning rate is not common for the ADAM optimizer. Moreover, they also require KL cutoff and regularizer coefficient inside the mini-PPO to encourage under-sampled data, which are two more dimensions of hyperparameters.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. As a follow-up insight, there may be cases where we can improve the sampling error in one state, but this will cause worse sampling errors down the road because we will be forced to go into specific states. Wouldn't it make more sense to push this sampling error into the reward instead, and form a suitable policy? if you think this is not an issue, it would be helpful if you can explain why.

2. Something doesn't sit right with me regarding equation 6 and its motivation. If we are only aiming to improve the sampling, in the discrete case we wouldn't care about the distance from the target policy. If we are aiming to also improve control, then most algorithms prevent policy changing too much to improve stability in which case we don't need this additional term (for example in PPO). So is this specifically for the continuous case? It does not make a lot of sense to me, at least according to the examples for the discrete case. I would be happy if you could elaborate on this point. 

3. Why did you chose the form given in Equation 5? why not its log or difference? Some explanation, ideally guided by theory would be helpful (it is the importance sampling term, widely used in off-policy learning, so it should make some sense using it in some variation).
soundness: 2
strengths: 1. The sampling errors problem the authors tackle is interesting, there has not been sufficient discussion on this topic in the RL community. 

2. The presentation and writing is generally clear. 

3. The sampling errors problem can arise in many real-world setups. 

4. The proposes solution is simple in a good way.
summary: The authors tackle the problem of sampling errors from insufficient data such that the empirical data has some discrepancy with desired policy. The authors propose a method to adaptively adjust the policy to resolve this discrepancy. They show their method works in several test cases including control and continuous action space.

** Post rebuttal change **
Now that I understood the paper better and following the discussions I have decided to increase my score.
weaknesses: 1. The experiments and motivation the authors provided are not accommodating for the problem. When is this really a problem in the real-world? In most standard examples there is ample time to correct the sampling errors with more data and simulation (this happens with a not a lot of samples in practice). Perhaps in cases of non-stationarity of the system itself this "quicker" adaptation become crucial? or maybe for very large state\action-spaces this is more of a problem. 

I propose the authors will try to distill where they're problem is really important and design experiments correspondingly, or at least motivate more strongly towards these cases.

2. Even in a quick look, Equation (5) can be rewritten much simpler to be -max(g, 1-eps). I'm not sure forcing it to look like PPO helps with insight, but it is confusing and unnecessary. That is, unless you formulate it similarly to the advantage.

3. Some questions were raised for me regarding the soundness of the proposed solution, see in the questions section.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I have only one question:

In Zhong et al., the policy is updated at every step in an online fashion: to me, this makes a lot of sense, since we want constantly to "correct" the sampling distribution. Ideally, one wants to employ the method you describe in Proposition 1. 

However, if I understand well, you propose first to use the target policy to collect the samples and, afterward, to generate a policy that "corrects" the previous behavior. However, such a policy is not well defined: it depends on how many samples will be drawn from such a policy. For example, suppose that the target policy is the distribution $\pi(a=0|s) = 0.5$ and $p(a=1|s)=0.5$. Suppose then that the samples drawn from this distribution are:

0, 0, 0, 0, 0.

The policy that you want to learn should then put much more probability density on 1 to compensate, but if we sample hundreds of samples with this new policy, then the dataset will be again "unbalanced". In my understanding, the idea of Zhong et al. to continuously update the policy is to avoid this issue.

Why did you propose this "batch update" rather than an "online update"?
soundness: 3
strengths: Originality
-------------

The idea proposed is novel. Although the method proposed until page 5 is from Zong's paper, and the application to policy gradient estimation seems somewhat trivial, I think it is still valuable and necessary.

Quality
----------

The quality of the paper is really good. The authors explain the problem and the main idea very well. The method is sounds and directly addresses the problem presented. The experiments are well-designed, explained, and commented on. The results are equipped with good statistical significance, and the appendix reports the hyperparameters necessary for reproducibility. 

Clarity
--------

The paper is well-written, and the overall idea is exposed very clearly.

Significance
-----------------

It is hard to assess the significance of this paper: 1) it heavily relies on a previous publication, and 2) it does not seem that the method improves so much over the baselines. However, I think the core idea is significant as it can serve as a source of inspiration for other RL problems, as MC is a core technique in RL to estimate many different quantities.
summary: This paper tackles the usual problem of the high-sample complexity of on-policy policy gradient algorithms by refining a very interesting idea proposed by Zhong et al. in 2022. In that paper, the authors observe that when the number of samples is scarce, the samples might resemble the on-policy distribution poorly, which would be equivalent, in a sense, to dealing with an off-policy distribution. In more simple terms, a low number of samples usually cause a high variance policy gradient estimation. 

Zhong et al. observe that on-policy samples do not necessarily need to come from the on-policy distribution; for example, the samples

x_i = 0, 1, 1, 0 

can come from p(x=1) = 0.5 and p(x=0)=0.5, but also from q(x=1) = 0.4 and q(x=0) = 0.6. 
Similarly, a batch of samples that look very off-policy might actually have been generated from the on-policy distribution, for example

x_i = 1, 1, 1, 1, 

might have been generated from the policy $p(x=0) = 0.5$, $p(x=1) = 0.5$. 
Monte-Carlo estimation (i.e., $E[X] \approx 1/n \sum_i X_i$) has higher variance exactly due to the issue described above, when the number of samples $n$ is low, while the variance disappears for $n\to\infty$, making it a consistent estimator.

The variance is the main problem affecting MC estimators and, thus, policy gradient algorithms as well (which are all MC estimators, as they attempt to estimate a respected value using $1/n \sum_i X_i$).

The variance of MC estimators can be ideally reduced if small batches of samples seem to better adhere to the target distribution. This can be achieved by breaking the i.i.d sampling assumption. For example, when the target distribution is $p(x=0) = 0.5$, $p(x=1) = 0.5$, One can choose a deterministic policy that alternates 0s and 1s. Such policy makes the MC estimator strictly more efficient. In this sense, **an off-policy distribution (that violates the i.i.d assumption) can generate better samples "on-policy" samples than the target distribution.**

Zhong et al. in 2022 propose to learn a policy that aims to lower the variance of the policy gradient estimators by generating samples that adhere better to the target policy distribution.

The contribution of this ICLR 2025 paper is to use the idea of Zhong et al. for on-policy policy gradient estimation (and policy improvement), while in the original publication, the idea was only proposed for policy evaluation.
weaknesses: Contribution
-----------------

The paper's contribution starts on page 5: the author focuses on presenting Zhong et al.'s paper there. The core contribution is the application of Zhon's idea for PG estimation and the use of PPO clipping and KL divergence to prevent too aggressive updates. 

While I think it is necessary that the authors devote that space to expose Zhong et al.'s idea, I think that it might not be clear to the reader that **that is not** the main core idea of the paper. 

Clarity
--------

While the paper is well written overall, I do not always like the choice of words and how the problem is described. 
I find talking about on-policy "data" vs. "sampling" confusing. A sample is not per se on or off-policy. When we speak about "on-policy" and "off-policy" in estimation, we refer to the generating process (i.e., the distribution) that generates the samples. A sample X_i can be generated by an off-policy distribution $\beta$ and simultaneously very unlikely w.r.t. $\beta$ and very likely w.r.t. the target distribution $\pi$, but that does not make the sample on-policy (or off-policy). Policy gradient estimation is simply off-policy when the distribution that is used to generate the samples is off-policy. 

Pairwise, I find speaking of "empirical policy" as in Proposition 1 confusing. One thing is samples, and one thing is the distribution. What can be told, in general, is that if $n > m$, then $Var[1/n \sum_i^n X_i] < Var[1/m \sum_i^m X_i]$, where X_i are samples coming from the same distribution. Now, it would be possible to introduce a new distribution (as done in Proposition 1), and show that the variance of the batch of samples decreases even faster in $n$, when compared to the original distribution. 

Overall, I am conscious that much of the "nomenclature" is borrowed from Zhong et al., thus the authors might have preferred to keep consistency with that. 

Minor Comments
-----------------------

Equation 1 is unclear as it is written: s_0 comes from a different distribution than s_1, which comes from a different distribution than s_2, and so on. Writing $s \sim d_\pi$ gives the impression that all states are drawn from the same distribution. Usually, there are two ways to express the policy's return,

1. 

$$
J(\theta) = \mathbb{E}_{s \sim d^\gamma_\pi, a \sim \pi}\left[r(s, a)\right]
$$

 where $d^\gamma_\pi$ is the discounted state-action visitation (see Nota and Thomas "Is policy gradient a gradient?" and Sutton et al. "Policy gradient with function approximation")

2. 

$$
J(\theta) = \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r(s_t, a_t)\right]
$$

where $s_0 \sim d_0, a_t \sim \pi(\cdot | s_t), s_{t+1} \sim p(\cdot | s_t, a_t)$.

Equation 2 and 4, should be with the discounted state visitation (see Nota and Thomas "Is policy gradient a gradient?" and Sutton et al. "Policy gradient with function approximation" :D )

References
---------------

Nota, C., & Thomas, P. S. (2019). Is the policy gradient a gradient?. arXiv preprint arXiv:1906.07073.

Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12. (**see page 3, definition of $d^\pi$**)

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: **[Q1]** What do the authors believe are the main benefits of PROPS relative to existing methods that address data or computational efficiency, including: 
- Methods that reuse data by making off-policy corrections (references in lines 122-124): These algorithms appear to be the most similar to PROPS, and some of them introduce similar levels of added complexity / runtime when compared to on-policy algorithms. However, they are not compared against in the experiments.
- Off-policy RL algorithms: These methods have been shown to be extremely data efficient and can even be used to train complex tasks directly from real-world data (e.g., [1]). They have also been shown to significantly outperform on-policy methods on some of the MuJoCo tasks used in this paper.
- Massively parallel on-policy implementations: When a simulator is available, it has been shown that massively parallel implementations of on-policy algorithms such as PPO can result in very efficient RL training in terms of wall-clock time (e.g., [2]). 

**[Q2] Experiments**
- The use of different hyperparameter values across each algorithm makes the comparison of PROPS and PPO difficult to interpret. I appreciate the value in the best vs. best comparisons presented in the paper, but how does PROPS perform when using the same hyperparameter values as PPO (and only tunes the PROPS-specific hyperparameters)?
- It is not obvious to me that reducing sampling error will necessarily lead to improved performance. Given the non-convex objective of policy gradient methods, noisy updates could even be helpful to avoid / escape local optima. Did you perform any experimental analysis on this connection (e.g., comparing PPO for the same number of total updates but varying batch sizes)? A comparison of PPO and PROPS for $b=1$ (with the same PPO hyperparameters) would also provide some insight into this question.
- It seems that “on-policy sampling” refers to PPO-Buffer in the sampling error figures, which does not seem like a fair comparison because PPO-Buffer does not attempt to correct for the use of off-policy data at all. What is the sampling error of standard on-policy PPO (and how does it compare to PROPS for $b=1$ with the same PPO hyperparameters)? Are similar trends observed?

**Minor:**
- In (1), the combination of an expectation over the visitation distribution and a summation over time inside the expectation is a bit strange. When summing over time inside the expectation, the expectation is typically written w.r.t. trajectories rather than the visitation distribution.
- Equation (5) can be written more directly as $\min(-\pi_{\phi} / \pi_{\theta}, -(1-\epsilon_{\textnormal{PROPS}}))$. The more complicated structure currently being used is needed in PPO’s objective because advantages can be positive or negative, but this is not relevant in (5).
- Why is IQM used as the performance metric? Typically the mean return is reported.
- Typos: line 341: Samping -> Sampling, line 485: Figure 5b -> Figure 3b
- Some of the legends appear inconsistent with other figures (Figure 12) / with the figure caption (Figure 15)

---

**References:**

[1] Smith et al. Demonstrating a Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning. In RSS 2023.

[2] Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. In CoRL 2022.
soundness: 3
strengths: - **[S1] Clear and well-written:** The paper is easy to read. The authors clearly present the interesting idea of on-policy data vs. on-policy sampling, as well as the proposed PROPS algorithm. 
- **[S2] Detailed, transparent presentation of experimental implementation:** The authors do a nice job of clearly describing the implementation details used in the experiments. This includes supporting analysis in the Appendix showing results across hyperparameter values.
summary: This paper proposes Proximal Robust On-Policy Sampling (PROPS), an adaptive off-policy sampling method designed to reduce sampling error and improve data efficiency in on-policy policy gradient algorithms. The paper generalizes the adaptive off-policy sampling method ROS introduced in Zhong et al. (2022) by proposing regularization techniques to handle continuous action spaces, and incorporates PROPS into RL training with PPO. Experiments on GridWorld, other discrete-action tasks, and MuJoCo benchmarks compare PPO with PROPS sampling to standard on-policy PPO and a version of PPO that naively reuses recent data without off-policy corrections.
weaknesses: **[W1] Lack of novel theoretical analysis**
- While Proposition 1 is valuable because it is not obvious that this sampling procedure converges to the correct state visitation distribution, it is a slight generalization of a result that already appears in Zhong et al. (2022). It is also restricted to finite states and actions, and is based on a different sampling procedure than what is proposed in Section 5. 
- Proposition 1 only analyzes what happens in the limit of infinite data, but the transient behavior is important in the context of policy gradient algorithms where a finite (and often limited) amount of data is collected between each policy update.
- It seems that there would likely be an important interplay between the size of policy updates, the amount of data to reuse, and the size of behavior policy updates determined by the PROPS objective. Unfortunately, the paper does not perform any analysis to connect these components, and instead sets all hyperparameters using a hyperparameter sweep. 

**[W2] Experiments do not demonstrate convincing performance benefits**
- The experimental results suggest that PROPS can achieve slightly better sampling error during RL training, but these improvements only lead to marginal performance benefits in continuous control MuJoCo tasks. 
- Experiments only compare against PPO and a version of PPO that naively reuses recent data (PPO-Buffer). There have been several works that improve data efficiency in a more principled way than PPO-Buffer that are similar to the goals of this work (see references in lines 122-124), but none of these are considered as baselines. Off-policy RL algorithms are also not included for comparison, which can significantly outperform on-policy or near on-policy methods in some of the tasks considered in this work.
- Due to the regularization needed for stable performance, PROPS requires the behavior policy to remain close to the current policy (small $\delta_{\textnormal{PROPS}}$). The experimental results suggest that this prevents PROPS from reusing a significant amount of past data (as mentioned in lines 470-471 and 1181-1183), limiting the data efficiency that can be achieved.

**[W3] Contribution is not clear relative to existing methods:** The paper does not clearly position its contribution relative to existing RL methods that also address data or computational efficiency. See [Q1] below.

Meta-review: This paper introduces an adaptive sampling method aimed at reducing the distribution mismatch between the empirical data distribution in the buffer and the target policy. The problem addressed is both interesting and underexplored in existing literature. However, the primary concerns raised by reviewers focus on the experimental evaluation. Specifically, given that the work centers on improving the sample efficiency of policy gradient methods through off-policy sampling, it should include comparisons with established off-policy methods. Furthermore, the improvement over PPO appears quite marginal, especially considering the additional algorithmic complexity introduced.
On a minor note, the theoretical contribution, particularly Proposition 1, seems trivial and appears very loosely connected to the subsequent algorithmic design.

Decision: Reject