On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling
Authors: Nicholas E. Corrado
Josiah P. Hanna
Venue: Accepted by TMLR ()

Abstract: On-policy reinforcement learning (RL) algorithms are typically characterized as algorithms that perform policy updates using i.i.d. trajectories collected by the agent's current policy. However, after observing only a finite number of trajectories, such on-policy sampling may produce data that fails to match the expected on-policy data distribution.  This \textit{sampling error} leads to high-variance gradient estimates that yield data inefficient on-policy learning. Recent work in the policy evaluation setting has shown that non-i.i.d.\@, off-policy sampling can produce data with lower sampling error w.r.t. the expected on-policy distribution than on-policy sampling can produce~\citep{zhong2022robust}. Motivated by this observation, we introduce an adaptive, off-policy sampling method to reduce sampling error during on-policy policy gradient RL training. Our method, Proximal Robust On-Policy Sampling (PROPS), reduces sampling error by collecting data with a \textit{behavior policy} that increases the probability of sampling actions that are under-sampled w.r.t. the current policy.  We empirically evaluate PROPS on both continuous-action MuJoCo benchmark tasks as well as discrete-action tasks and demonstrate that (1) PROPS decreases sampling error throughout training and (2) increases the data efficiency of on-policy policy gradient algorithms.

Published in Transactions on Machine Learning Research (2/2026)
On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling
Nicholas E. Corrado
ncorrad
Deportment of Computer Sciences
University of Wisconsin- MassisinnMMadioon
Josiah P. Hanna
jphanma &ccs wisc.. edu
Department of Computer Sciences
University of Wisconsin-Madison
Reviewed on OpenReview: https: //
// opwnreview. net/ forum? id= #CoopppBBB0
Abstract
On-policy reinforcement learning (RL) algorithms are typically characterized as algorithms that perform policy updates using i..d. trajectories collected by the agent's current policy. However, after observing only finite number of trajectories, such on-policy sampling may produce data that fails to match the expected on-policy data distribution. This sampling error leads to high-variance gradient estimates that yield data-inefficient ou-policy learning. Recent work in the policy evaluation setting has shown that non-i.i.d., off-policy sampling can produce data with lower sampling error w.r.t. the expected on-policy distribution than on-policy sampling can produce (Zhong et al. 2022). Motivated by this observation, we introduce an adaptive, off off-policy sampling method reduce sampling error during on-policy policy gradient RL training. Our method, Proximal Robust On-Policy Sampling (PROPS), reduces sampling error by collecting data with behavior policy that increases the probability of sampling actions that are under-sampled w.r.t. the current policy. We empirically evaluate PROPS on contimuous-acteo action MuJoCo benchmark tasks as well as discrete-action tasks and demonstrate that (1) PROPS decreases sampling error throughout training and (2) increases the data efficiency on-policy policy gradient algorithms.
1 Introduction
One of the most widely used classes of reinforcement learning (RL) algorithms the class of on-policy policy gradient algorithms. These algorithms gradient ascent the paramcters of a parameterized policy to increase the probability of observed actions with high expected returns under the current policy. The gradient is commonly estimated using the Monte Carlo estimator, an average computed over i.i.d. trajectories sampled from the current policy. The Monte Carlo estimator is consistent and unbiased; as the number of sampled trajectories increases, the empirical distribution data converges to the expected distribution under the current policy, and thus the estimated gradient. converges the true gradient. However, the expense of environment interaction forces us to work with finite sample sizes. Thus, the empirical distribution of data often differs from the desired on-policy data distribution, mismatch we call sampling error. Sampling error causes inaccurate gradient estimates, resulting in high-variance policy updates, slower learning, and potentially convergence to suboptimal policies.
With i.i.d. on-policy sampling, the only way to reduce sampling error is to collect more data. Alternatively, we can reduce sampling error more efficiently using adaptive, ofo-colic sampling. To illustrate this approach, consider an MDP with two discrete actions and B, and suppose the current policy places equal probability on both actions in some state s. When following म, after 10 visits s, we will observe both actions times in expectation. Now suppose that after the first 9 visits to 8, we actually observe A 4 times and B 5 times.

---

Published in Transactions on Machine Learning Research (2/2026)
Adaptive, Off-Policy Data Collection
Policy Optimization
T T''
Φ
π
Φ
Collect data with data collection policy Thg and store data in buller D.
Update To increase probubility of actions that are under-saappeed in D w.r.t. the curreat policy TT.
Update the agent's currens policy Rig using an on-poliey policy gradient algorithm.
Figure An overview of PROPS. Rather than collecting data D via on-policy sampling from the agent's current policy ne; we collect data with a separate data collection policy a that we continually adapt to reduce sampling error in D with respect to the agent's current policy.
If we sample an action from upon next visit to 8, we may sample B, and our data will not match the expected on-policy distribution. Alternatively, if we select the under-sampled action A with probability 1, we will observe each action 5 times, making the aggregate data match the on-policy distribution even though this final action was sampled off-policy. The first scenario illustrates on-policy sampling with sampling error; the second scenario uses adaptive, off-policy sampling to reduce sampling error.
Recently, Zhong et al. (2022) introduced an adaptive, off-policy sampling method (ROS) that can pro- duce data that more closely matches the on-policy distribution than data acquired through i.i.d. on-policy sampling. However, this work was limited to low-dimensional policy evaluation tasks. Moreover, ROS required large batches of data to reduce sampling error error- approximately 5000 samples on tasks like CartPole- struggled to reduce sampling error on tasks with continuous actions. To
v1 BBrockman et al. make adaplive sampling practical for data-efficient RL, we need methods that can reduce sampling error in high-dimensione contimuous-acte action tasks while using the smaller batch sizes typically used in RL. These obsorvations raise the following question: Can reducing sampling error through adaptive sampling increase the dafa-effc of on-policy policy gradient methods?
In this work, we address these challenges and show for the first time that on-policy policy gradient algorithms are more data-cfficient learners when they use adaptive, off-policy sampling to reduce sampling error w.r.t. the on-policy distribution. Our method, Proximal Robust On- Policy Sampling (PROPS), adaptively corrects sampling error in previously collected data by increasing the probability of sampling actions that are under- sampled with respect to the current policy. Fig. [ provides an overview of PROPS. We empirically evaluate PROPS on matinuous-action MuJoCo benchmark tasks and show that (1) PROPS reduces sampling error throughout training and (2) increases the data elficiency of on-policy policy gradient algorithms. In summary, our contributions are:
We introduce PROPS, a scalable adaptive sampling algorithm for on-policy policy gradient learning that reduces sampling error w.r.t. the agent's current policy.
We demonstrate empirically that PROPS reduces sampling error more efficiently than on-policy sampling and ROS.
3. We demonstrate empirically that reducing sampling error via PROPS increases data efficiency in ou-policy policy gradient RL.
2
Related Work
Data Collection and Exploration. Our work focuses on data collection in RL. In RL, data collec- tion is often framed as exploration problem, focusing how agent should explore its environment
2

---

Published in Transactions on Machine Learning Research (2/2026)
to efficiently learn an optimal policy. Prior works have proposed exploration-promoting methods such intrinsic motivation (Pathak et al.) 2017 Sukhbaatar et al. 2018), count-based exploration (Tang et al. 2017 Ostrovski et al. 2017), and Thompson sampling (Osband et al. 2013 Sutton & Barto 2018). Rather than adjusting the agent's ' sampling distribution to promote exploration, our objective to learn from the on-policy data distribution; we use adaptive data collection to more efficiently oblain this data distribution.
Adaptive Sampling for Sampling Error Reduction. Prior works have used adaptive off-policy sampling to reduce sampling error in the policy evaluation subfield of RL. Most closely related is the work of [Zhon et al. (022),, who first proposed that adaptive off-policy sampling could produce data that more closely matches the on-policy distribution than on-policy sampling could produce. MMkhherjee et al. (2022) use a deterministic sampling rule to take actions in a particular proportion. Other bandit works use a non-adaptive exploration policy to collect additional data conditioned on previously collected data (Tucker & Joachims,
2022 Wan et al. 2022 Konyushova et al. 2021). Since these works only focus on policy evaluation, they do not have to contend with a changing on-policy distribution as our work does for the control sctting.
Importance Sampling for Sampling Error Reduction. Several prior works propose importance sam- pling methods (Precup 2000) to reduce sampling error without further data collection. In the RL setting, Hanna et al. (2021) showed that reweighting off off-policy data according to an estimated behavior policy can correct sampling error and improve policy evaluation. Liu et al. (2024) compute a globally optimal behavior policy using an offline dataset. [aapini et al. (2024) focused on finding a behavior policy that minimizes the variance of imporraance-wigghtee policy gradient estimates. Similar methods have been studied for temporal difference learning (Pavse et al., 2020 and policy evaluation in the bandit setting (Li et al.) 2015 Narita ett al. 2019 Conservative Data Sharing YYu et al., 2021) reduces sampling error by selectively integrating ofiline data from multiple tasks. While these works reduce sampling error by reweighting existing data, work focuses on whether we can reduce sampling error during data collection. Moreover, works like Liu et al. (2024) require a large amount of offline data (e.g., 1000 trajectories) to accurately compute a behavior policy, but in online on-policy RL, agents typically collect only a few trajectorics between policy updates.
On-Policy Learning with Off-Policy Data. As we will discuss Section 5 the method we introduce permits data collected in one iteration of policy optimization to be re-used in future iterations rather than discarded as typically done by on-policy algorithms. Prior work has attempted to avoid discarding data by combining olf-policy and on-policy updales with separate loss functions or by using alternative gradient estimates (Wang et al.) 2016 Gu et al. 2016 2017 Fakoor et al. 2020 O D'Doooghue et al.) 2016 Queeney et al. 2021). In contrast, our method modifies the sampling distribution at each iteration so that the entire data set of past and newly collected data matches the expected distribution under the current policy.
3 Preliminaries
3.1
Reinforcement Learning
We formalize the RL environment
infinite horizon Markov decision process (MDP) Puterman 2014
p, do. 7) with state space S, action space A, transition dynamics S x AA x S + 10 reward function r: S x A + R, initial state distribution do, and reward discount factor action spaces may be discrete or continuous. We considor stochastic policies To : Sx.A + [0. 1] parameterized
1). The state and
by 0, and we write (a)s)) to denote the probability of sampling action a state 8 and π (-1s) to denote
the probability distribution over actions in state s. We additionally let drp S A A [0. 1] denote the state-action visitation distribution, the distribution over state-action pairs induced by following me. The RL objective is to find policy that maximizes the expected sum of discounted rewards, defined as:
J(0)= Er |n 2'r((8r
Throughout this paper, we refer to the policy used for data collection as the behavior policy and the policy trained to maximize its expected return the target policy-
3

---

Published in Transactions on Machine Learning Research (2/2026)
3.2 On-Policy Policy Gradient Algorithms
Policy gradient algorithms perform gradient ascent over policy parameters to maximize an agent's 's expected return J(O). The gradient of (() with respect to O, or policy gradient, is often given as:
(() = Bg amam [A (s, a) Vo log [e(a)))]
(2)
where (s, a) is the advantage of choosing action a in state s and following Te thereafter In practice, the expectation in Eq. _ is approximated with Monte Carlo samples collected from Te, and an estimate of A is used in place of the true advantages (Schulman et al., 2016). Since off-policy data will bias this gradient estimator, on-policy algorithms discard historic off-policy data after each policy update and collect new data with the updated p
This foundational idea of policy learning via stochastic gradient ascent was first proposed by Williams (9922) under the name REINFORCE. Since then, large body of research has focused on developing more scala scalable
policy gradient methods (Kakade, 2001 Schulman et al.) 2015) Muih et al. 2016 [Espehott et al.) 2018)
Lillicrpp et al. 2015; Haarnoja et al. 2018). Currently, the most successful variant of on-policy learning is
proximal policy oplimization (PPO) Schulman et al. 2017), the algorithm of choice in several high-prolile
success stories (Berner et al. 2019 Akkaya et al. 2019 Vinyals et al. 2019). PPO maximizes the clipped
surrogate objective
Eppo(s, a, 0, Oata) = min(g(s, a, 0, 0,0 (8, a), clip(g(s, a, 0, O,l)), c, 1 -) (s, a)):
(3)
where Oold denotes the policy parameters prior to the update, g(s, a, O, Oata) the policy ratio g(s, a, O.a) = na(a|s]
me [a|s]) and the clip function with hyperparameter e clips g(s, 0, doia) to the in- terval [1 -1-c, e, 1 e]. This objective disincentivizes large changes to ke(als). While other policy gradient algorithms perform a single gradient update per data sample to avoid destructively large policy updates, PPO's clipping mechanism permits multiple epochs of minibatch policy updates.
4 Correcting Sampling Error in Reinforcement Learning
In this section, we illustrate how sampling error can produce inaccurate policy gradient estimates and then describe how adaptive, off-policy sampling can reduce sampling error. For exposition, assume finite state and action spaces. The policy gradicnt can then be written as:
VoJ(O)= - doo a) [A log
[A
(4)
The policy gradient is thus linear combination of the gradient for each (s, a) pair Ve log (a)ss)) weighted by (s, (s, a). Let 'D be a dataset of trajectories. It is straightforward to show that the Monte Carlo estimate of the policy gradient can be written in a form similar to Eq. g except with the true state-action visitation distribution replaced with the empirical visitation distribution dp(s, a), denoting the fraction of times (s, a) appears DD (aanna et al. 2021): :
voā(0)= = (100)=Eraasee Eisa)ssa do a) (A a) Togassss))
(5)
By comparing Eq. and Eq. 5 we can understand how sampling error affects the Monte Carlo policy gradient estimate. When (s, a) is over-sampled (i..,, dd(ss, a) dzp (8, (8,a)), then Vo log To s) contributes more to the overall gradient than it should. Similarly, when (s, a) is under-sampled, Vo log To (als))) contributes less
We note that Eq. is biased estimator of the policy gradient; the expectation is taken over the undiscounted state than the discounted state distribution di. so Eq. 2 is not the gradient of the discounted objective Nooaa & Thomas
distribution dag rather
Eq.
2019
Nevertheless, since many popular policy gradient algorithms estimate the policy gradient using
Eq.
Schulman al 2017
Haarnoja al. 2018 Muih all 2016), we dug rather than di, in this work.
practice, policy gradient algorithms have other sources gradient. blas, such as using X-returns in place of Monte Carlo
returns
(Sutton Rarto
2018
and using de. in place drg Nota Thomas 2019
4

---

Published in Transactions on Machine Learning Research (2/2026)
than it should. Below, we provide concrete example illustrating how small amounts of sampling error cause the wrong actions to be reinforced.
Example: Sampling error can cause incorrect policy updates.
Let Te be a policy for an MDP with two discrete actions dg and 01, and suppose that in a particular
state 80, the advantage of each action w.r.t. wo is (sp, ao) = 20 and A (80 a1) = 15. For simplic- ity, suppose the policy has a direct parameterization To (o(an]s) s) = 2.. To a 8) = 1 1-0, 0, and places equal probability on both actions in So (0..- = 0.5). Then, we have Vo log To( su) and d=g (80, ao) dx (8: a1) that the expected gradient increases the probability of sampling do, the optimal action. With on-policy sampling, after 10 visits to Sø, the agent will sample both actions 5 times in expectation. However, if the agent actually observes do 4 times and a 6 times, Monte Carlo estimate of the policy gradient then yields
Vo log ( (s))
4
6 15 Vo log no (a (80) Vo log (ao)
20 Vo log (aol
10
which decreases the probability of sampling the optimal au action.
Sampling error in on-policy sampling vanishes as the size of the batch of data used to estimate the gradient lends loward infinily. However, the preceding example suggests simple strategy to eliminate sampling error with finite data: have the agent adapt its probability on the next action it takes based on the actions it has alrcady sampled. Continuing with our example, suppose the agent has visited so 9 times and sampled ag 4 times and a1 5 times. With on-policy sampling, the agent may observe a1 again upon the next visit to 80- Alternatively, the agent could sample its next action from a distribution that puts probability 1 on dg and consequently produce an aggregate batch of data that contains both actions in their expected frequency. While this adaptive method uses off-policy sampling, it produces data that exactly matches the on-policy distribution and thus produces a more accurate gradient.
This example suggests that we can heuristically reduce sampling error by selecting the most under-sampled action at a given state. Under a strong assumption that the MDP has a directed acyelic graph (DAG) structure, Zhong et al. (2022) proved that in a fixed-horizon MDP, this heuristic produces an empirical state-action distribution that converges to dra (s, a) and morcover converges at a faster rate than on-policy sampling. We remove this limiting DAG assumption with the following result:
Proposition Assume that data is collected with on adaplive behavior policy that always lakes the most mnder-samplee action in each state w.r.l. n, i.e., <- arg max (T (a']s) TD (s)])ss);, where Tn is the empirical policy after m state-action pairs have been collected. Assume that S and A are finite and that the Markov chain induced by π is irreducible. Then we have that the empirical state visitation distribution, dm. converges to the state distritution of π,, dr, with probabilily 1:
Vs, lim (m(ss) = =d))))ss)) m0000
We prove Proposition ! in Appendix A While adaptively sampling the most under-sampled action can reduce sampling error, this heuristic is difficult to implement in practice; in tasks with contimuuu states and actions, the arg max in Proposition ! often has no closed-form solution, and the empirical policy πD can be expensive to compute at every timestep. Building upon the concepts discussed in this section, the following section presents a scalable adaptive sampling algorithm that reduces sampling error in on-policy policy gradient learning.
5 Proximal Robust On-Policy Sampling for Policy Gradient Algorithms
Our goal to develop an adaptive, off-policy sampling algorithm that reduces sampling error in on-policy data collection for on-policy policy gradient algorithms. We outline a general framework for on-policy learning with an adaptive behavior policy in Algorithm The behavior policy collects a batch of transitions, adds the batch to data buffer D, and then updates its weights such that the next batch it collects reduces
5

---

Published in Transactions on Machine Learning Research (2/2026)
sampling error in D with respect to the target policy To (Lines 7 10.. Every steps (with n > m), the agent updates its target policy with data from D (Tine 11p. We refer to and n as the behanior update frequency and the target botch size, respectively.
The behavior policy must continually adjust action prob- abilities for new samples so that the aggregate data dis- tribution of 'D matches the expected on-policy distribu- tion of the current target policy (Line [1). A subtle im- plication of this adaptive sampling is that it can correct sampling error in any empirical data distribution-even one generated by a different policy. Thus, rather than discarding off- off-polic data from old policies as is com- monly done on-policy learning, we let the data buffer hold up to b target batches (bn transitions) and call the buffer size. If b > 1, then D will contain historic off policy data used in previous target policy updates 3 Implementing Line [0] is the core challenge we address in the remainder of this section.
Algorithm Ou-policy policy gradient algo- rithm with adaptive sampling
Target batch size n, behavior update frequency m, buffer size b.
Inputs:
2: Output: Target policy parameters O.
3: Initialize target policy parameters 0.
4: Initialize behavior policy paramcters Φ <- 0.
5: Initialize emply buffer D with capacity bn.
6: for target update = 1,2,... do
1...., ................................ Ln/m] do &: Collect batch of data B by running Tg. Append B to buffer D.
T: for behavior update
9:
10: Update VΦ with D using Algorithm
11: Update To with D.
5.1 Robust On-Policy Sampling
12:
return 0
To ensure that the empirical distribution of 'D matches the expected on-policy distribution, updates to T should attempt to increase the probability of actions which are currently under-sampled with respect to me. Zhong et al. (2022) recently developed a simple method called Robust On-policy Sampling (ROS) for making such updates, which PROPS builds upon. In this section, we brielly review ROS and discuss its limitations.
Let (a s) denote the empirical policy, the fraction of times action a is taken at state s in D. If (D(aas) < (a(ss)) then appears fewer times at than it would in expectation under To and is therefore under- sampled. Thus, wp provides a way to identily actions whose probabilities should be increased to reduce sampling error. In the tabular setting, TD can be computed exactly as the maximizer of the logelikel,hood,
C((') (( log ne ( (a s),
(6)
i.c., TD arg max ((0). However, in tasks with high-dimensional continuous state and action spaces, computing To explicitly is generally intractable. The key insight of ROS is that we do not need to compute wD explicitly; we only need a direction that tells us how to adjust action probabilities based on how the data in D differs from Tg. The gradient of the log- biglliiei at & = 0, vol+)) x= points in a direction that moves म toward TD, so a step in this direction decreases the probabilities of actions that are under-sampled in D w.r.t. To and increases those of over-sampled actions. Therefore, taking a step in the opposite direction increases the probabilities of under-sampled actions and decreases those of over-sampled actions, providing a way to reduce sampling error without ever computing TD. Thus, ROS performs single step of descent on the negative log- likellioohoo
-V 00 E EioOaED V log
at each timestep to increase the probability of under-sampled actions. In theory and in low-dimensional RL policy evaluation tasks such CartPole-vl (Brockman et al., 2016), this update was shown to improve the rate at which the empirical data distribution converges to the on-policy distribution-even when the empirical data distribution contains off-policy data. Unfortunately, two main challenges render ROS unsuitable for Line 10 in Algorithm
Challenge Destructively large policy updates. Since D may contain data collectod by old target policies, some samples in D may be very off-policy w.r.t. the current target policy such that log (als)) is
Thhe advantage estimates in historic data correspond to historic policies and are thus biased advantage estimates of the current policy. In Section we provide additional detail on how mitigate this bias with generalized advantage estimation (GAF).
6

---

Published in Transactions on Machine Learning Research (2/2026)
large and negative. Since V log (o(ls) increases in magnitude TT(aas) approaches zero, those off-policy samples can produce restructyvely large updates.
Challenge 2: Improper handling of contin- uous actions. In contimuous-action tasks, ROS may produce behavior policies that increase sam- pling error. Continuous policies (e(a)s) are typ- ically parameterized as Gaussians N ( [{)) with mean p(s) and diagonal covariance matrix E(s). Since actions in the tail of the Gaussian far from the mean will generally be under-sampled, the ROS update will contimually push the com- ponents of (s() towards +oo and the diagonal increase the probability of sampling these actions. The result is degenerate behavior policy that is so far from the target policy that sampling from it increases sampling error. We illustrate this scenario with 1-dimensional continuous actions in Fig. of Ap- pendix C In the next section, we discuss how to address both of these challenges.
Algorithm 2 PROPS Update
1: Inputs:
Target policy parameters 0, bulfer D, target clipping coefficient EPROPS, regularizer coefficient X, n a_epoch, n minibatch.
KL óPROPS;
Output: Behavior policy paramcters $.
3: Φ +
4: for epoch i = 1.2..... n_ epech do
5: for minibatch j j=
= 1.2. 1.2.... ....._miinn n_minibatch do 6: Sample minibatch Dj ~D
components of E(s) towards
7: Update & with a step of gradient ascent on loss
Σ (a,a)ED,
CPROPS(s, o, ePROPS,
[
8:
if DxI((Tol > SPrOPS then return Φ
9:
10:
return Φ
5.2 Proximal Robust On-Policy Sampling
To address the challenges ROS faces in continuous control, we propose a new behavior policy update. To address Challenge 1, first observe that the gradient the ROS loss VaC= = V log TT(aas))aa is equivalent to the policy gradient (Eq. 2 with A a) -1, V(s, a). Since the clipped surrogate objective of PPO (Eq. 33 prevents destructively large updates in on-policy policy gradient learning, we can use a similar clipped objective in place of the ROS objective to prevent destructive behavior policy updates:
(a(s)) (a(a)))
(a(a)s) To (a(())))
1 EPROPS, 1 EPROPS) )
Cear((8, a, Φb, 0, EPROPS) min
-lip
(8)
Table 2 in Appendix C |summarizes the behavior of Leur. Intuitively, this objective is equivalent to the PPO objective (Eq.B) 3 with A(s, a) = -1,v(s, a) and incentivizes the agent to decrease the probability of observed actions by at most factor of 1 <PROPS- Tet g(s, a, ②b, A) this objective is clipped at -(1 - EPROPS). The loss gradient V VøLcur becomes zero, and the (s, a) pair has no effect on the policy update. When g(s,, a, db, 0) > <pROPS, clipping does not apply, and the gradient VoLeu points in a direction that decreases the probability of (als)). As the PPO update, this clipping mechanism avoids destructively large policy updates and permits perform multiple epochs of minibatch updates with the same batch of data.
(aaa))
Ta(aas) When g(s, a, Φ 0) < 1 6PROPS:
To address the second challenge and prevent degenerate behavior policies, we introduce an auxiliary loss that ivcens the agent to minimize the KL divergence between the behavior policy and target policy at states in the observed data. The full PROPS objective is then:
LPROPS (8, a, & 0, epropS, X) Leaw a, 0. (1)
(9)
where À is regularization cocfficient quantifying a tradc-off between maximizing LrROrS and minimizing D Algorithm 2 provides pseudocode for the PROPS update. Like ROS, we set the behavior policy parameters & equal to the target policy parameters at the start of each behavior update, and then make a local adjustment to é to increase the probabilitics of under-sampled actions. We stop the PROPS update early when DKL (To ((T) reaches chosen threshold òPROPs. This technique further safeguards against large policy updates and is used in widely adopted implementations of PPO (Raffin et al. 2021 Liang et. al.

---

Published in Transactions on Machine Learning Research (2/2026)
we provide theoretical intuition for the relationship between different PROPS hyperparameters. PROPS enables us to efficiently learn behavior policy that keeps the distribution of data in the buffer close to the expected distribution of the target policy.
2018 In Appendix B
6 Experiments
The central goal of our work is to understand if reducing sampling error with adaptive, off-policy sampling results in more data-efficient on-policy policy gradient learning. Towards this goal, we design experiments on contimuous-state and continuons-action MuJoCo benchmark tasks [Brockman et al. 2016) and a tabular 5x5 GridWorld lask (Fig. 2a lo answer the following questions:
Q1: Does PROPS achieve lower sampling error than on-policy sampling during RL training?
Q2: Does PROPS improve the data efficiency of on-policy policy gradient algorithms? That is, does PROPS reduce the number of environment interactions required to achieve returns comparable to on-policy sampling?
6.1 Sampling Error Metrics
In GridWorld, we compute sampling error the total variation (TV) distance between the empirical state- action visitation dp(s,a) a) distribution iitribuutoo-ddeotinng denoting the proportion of times (s,a) appears in buffer D-and the true state-action visitation distribution under the agent's policy: E( dn, (s. a) -ppss a)l. Since it is straightforward to compule the true policy gradient in the GridWorld task, we additionally investigate how sampling error reduction affects gradient estimation by measuring the cosine similarity between the empirical policy gradient VoJ(O) and the true policy gradient. As the empirical gradient aligns more closely with the true gradient, the cosine similarity approaches the KT--divereence DKL (T) between the empirical policy TD and the target policy me, which is the primary metric Zhong et al. 2022 used to show ROS reduces sampling error:
continuous MuJoCo tasks where it is difficult
to compute dD(s, a), we compute sampling
Tt TD((as)) To s)
(TD(TTO)) = F (a-DDa log
(10)
We estimate un as the maximum likelihood estimate under data in the buffer via stochastic gradient ascent. More concretely, we let O' be the parameters of a neural network with the same architecture me and then compute:
i MLE = arg max
Σ E(,,)eD log Toy To (a|s)
(11)
6'
using stochastic gradient ascent. After computing OMIE, we then estimate sampling error using the Monte
Carlo estimator:
DKL (KDl TO) R E (log Te log (a(ss))
(12)
6.2 Correcting Sampling Error for Fixed Target Policy
We first study how quickly PROPS decreases sampling error when the target policy is fixed. This set- ting is similar to the policy evaluation setting considered by Zhong al. (2022) As such, we pro- vide two core baselines for comparison: on-policy sampling and ROS. In GridWorld, we addition- ally include an oracle sampling method that always selects the most under-sampled statc-action pair (s" (s",a") ) = arg max dra (s, a) - dp(s, a). This sampler is not realizable in standard RL settings because it assumes the ability to reset. the agent an arbitrary state, but provides a useful reference point for the minimum sampling error achievable for given number of samples. We provide details on hyperparameter tuming in Appendix D.2
The KL regularizer Eq. 9 and the KL cutoff rule (To(TTl óynors diffcrent purposcs. KL rogularization limits how far the behavior policy can move from the target policy after each gradient step, while the cutoff rule prevents the behavior policy from drifting too across madtiple epochs of minibatch updates, even each step individually is small
8

---

Published in Transactions on Machine Learning Research (2/2026)
GridWorld-5x5-v0
GridWorld-5x5-v
10
10a
Cosine Similarity
Sampling Error
AmA
10
PRCPS
0.6
PROPS
RDS
BOS
Samp ing
Oi-Relity
0.4
Sampling
On-hni ey
Oracle
10
10a
Samples Collected
Samples Collected
(a) GridWorld
(b) Sampling error
(c) Gradient accuracy
Figure 2: (a) A GridWorld task in which the agent receives reward +1 upon reaching the bottom right corner (the optimal goal), reward of +0.5 upon reaching the top left corner (the suboptimal goal), and a reward of -0.01. The agent always starts in the center of the grid. Under an initially uniform policy, the agent visits both goals with equal probability, and thus the true policy gradient increases the probability of reaching the optimal goal. However, sampling error can yield an empirical gradient that increases the probability of reaching the suboptimal goal and cause the agent to converge suboptimally. To converge optimally, the agent must have low sampling error. (b, c) PROPS reduces sampling and achieves accurate gradients faster than on-policy sampling, Solid curves denote means over 50 seeds. Shaded regions denote 95% bootstrap confidence belts.
PROPS
ROS
On-Policy Sampling
Sampling Enor
Swimmer-v4
Bampinng Error
Hopper-v4
sampling Error
Walker2d-v4
Sempling Error
HailCheetah-v4
Sampling Error
Antové
Sampling Error
Humnaooidvv4
10
10"
10"
1
10
1024
2048
4096
8192
1024
2048
4096
8192
102A
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
Samples Collected
Samples Collected
Samples Collected
Samples Collected
Samples Collected
Samples Collected
Figure 3: Sampling error with fixed, randomly initialized target policy. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
Results. We collect 8192 samples with a randomly initialized policy and compute sampling error throughout data collection. As shown in Fig. 2b and 2c in GridWorld, PROPS decreases sampling error faster than on- policy sampling, resulting in more accurate policy gradient estimates. PROPS and ROS perform similarly, though this behavior expected: in a tabular setting with a fixed target policy, there is no off-policy data in the buffer, so we do not encounter Challenge 1 and described in the previous section. In Appendix D.I, empirically demonstrate that PROPS is unbiased and has lower variance than on-policy sampling. The oracle sampler attains lower sampling error and higher gradient accuracy than PROPS, as expected, since it assumes access to state resets
In contimous MuJoCo tasks where Challenge 2 arises, PROPS decreases sampling error faster than on-policy sampling and ROS (Fig. 3 In fact, ROS shows no improvement over on-policy sampling in all MuJoCo tasks. This linitation of ROS is unsurprising, as Zhong et al. (2022) showed that ROS struggled reduce sampling error even in low-dimensional puutinuous-action tasks. Since batch sizes in the range of 2048 8192 are standard for RI in these tasks, ROS alone will not be able to improve data efficiency during RL. In
Sampling error does not decrease monotonically under the oracle sampler. With I' samples, the expected number of visits to each (s, a) under on-policy sampling is T', dr, (8, a), but these quantities generally not integers, the oracle sampler pains be
must therefore chouse the nearest integer allocation, which forces this rounding may align more less closely with the target distribution, leading to fluctuations in the minimum achievable state action pairs and deg uniform. When T is multiple of n, the oracle can allocate counts evenly across all pairs; for other values (e-g., n I,..., 2n 1), at least one pair must. receive an or missing count, necessarily increasing sampling error.
under-sampled. Depending on
TV distance. For concrete illustration, suppose there

---

Published in Transactions on Machine Learning Research (2/2026)
PiPps
PrO-Buffer
WD
PROPS
PPO-auffer
PPQ
Swimmerua
Hoppers4
Swimmerv4
Hopparye
HailChestahv4
3000
1.00
100
100
Mian Retuu
0.75
0.75
50
2000
2000
fraction of
with Score
0.50
0.30
1000
1000
0.2
0.25
0.23
000
0.00
0.00
4350
le5
Timesrp Humanoddv
Return (1)
Velker2ddv4
Ant
Walke
Humanoid
100
2.00
1.00
Neam Retur
3000
6000
0.75
0.79
0.75
ofction
With zare
2000
0.92
0.30
2000
1000
0.5
0.23
0.25
01
000
0.00
0.08
255
3001
Timestop
1e5
Timestop
166
Timestop
les
Raturn (7)
Retum
Retum (T)
(a) Mean return.
(b) Performance profiles.
Figure 4: (a) Mean returns over 50 seeds. (b) Performance profiles over 50 seeds. Higher values correspond to more reliable convergence to high-return policies. Shaded regions denote 95% bootstrap confidence intervals.
33%
training budget
66% of training budget
100% of training budget
Environment
PROPS
PPO-Bufier
PPO
PROPS
PPO-Buffer
PPO
PROPS
PPO-Buffer
PPO
Swimmer-vd
70.5
64.8
55.6
97.4
90.5
90.9
109.7
102.9
100.9
Hopper-v4
2675
2307
2347
3144
2801
2946
3253
2967
3148
HaliCheetah-vl
2292
1546
1810
2591
2032
2133
2751
2237
2270
Walker2d-v1
2880
2107
1989
3590
3336
2858
3908
3737
3397
Ant-v4
3519
3493
3484
4668
4419
4589
4930
4741
5014
Humanoid-v4
2270
1938
1376
4891
4449
4067
5184
4958
4955
Table 1: Return achieved at different points throughout training. We use color to indicate statistical sig- nificance at a 95% confidence level according to a paired I-test. Green indicates PROPS outperforms PPO and PPO-Buffer with significance. Grey indicates PROPS outperforms only PPO with significance. Blue indicates PROPS outperforms only PPO-Buffer with significance. Bold values indicate the largest average return. PROPS achieves a larger return than PPO and PPO-Buffer at all points except at the end of training in Ant-v4, where PPO achieves slightly larger (but similar) return.
Appendix D.2. we include similar experiments with expert target policies as well as ablations on PROPS's objective clipping and regularization. Rosults with expert target policics are qualitatively similar to Fig. 3 and we observe that clipping and regularization both individually help reduce sampling error.
6.3 Correcting Sampling Error During RL Training
We are ultimately interested in understanding how replacing on-policy sampling with PROPS affects the data efficiency of on-policy learning. In the following experiments, we train RL agents with PROPS and on-policy sampling to evaluate (1) the data efficiency of training, (2) the distribution of returns achieved at the end of training, and (3) the sampling error throughout training. We use the same sampling error metries described in the previous section and measure data efficiency the return achieved within a fixed training budget. Since ROS is computationally expensive and fails to reduce sampling error in MuJoCo tasks even with a fixed policy, we omit it from MuJoCo experiments.
Experimental setup. We use PPO (Schulman et al. 2017) to update the largel policy and consider two baseline methods for providing data to compute PPO updates: (1) vanilla PPO with on-policy sampling, and (2) PPO with on-policy sampling using buffer of size (PPO-Buffer). PPO-Buffer is a naive method for improving data efficiency of on-policy algorithms by reusing off off-policy data collected by old target policies as if it were on-pol data. Although PPO-Buffer computes biased gradients, it has been successfully applied
10

---

Published in Transactions on Machine Learning Research (2/2026)
Aggregate Performance
Aggregate Performance
0.8
1.00
PROPS
PPO-Buffer
Mean Normalizeo wallllzed Return
0.6
Fraction runs with score
0.75
PPO
0.5
0.4
0.50
0.3
0.2
0.25
PROPS
0.1
PPO-Buffer
PPO
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of Training Budget
Normalized Return (T)
(a) Mean normalized return (aggregated).
(b) Performance proflle (aggregated).
Figure 5: Mean normalized return and performance profiles aggregated over all six MuJoCo tasks. We compute normalized returns as R where Rmax is the maximum return achieved by any algorithm in particular task, and Rg is the return at timestep Solid curves denote the mcan over 50 soeds per task (300 seeds total). Shaded regions denote 95% boolstrap confidence belts.
Rese
in difficult learning tasks [Berner et. al. 2019). Since PROPS and PPO-Buffer have access to the amount of data for each policy update, any performance difference between these two methods arises from differences in how they sample actions during data collection.
In MuJoCo experiments, we set = 2 such that agents retain each batch of data for one extra iteration before discarding it. In GridWWold, we use b= and discard all historic data. Since PROPS and PPO-Buffer compute target policy updates with b times as much learning data as PPO, we integrate this extra data by increasing the minibatch size for target and behavior policy updates by factor of Further experimental details including hyperparameter tuning described in Appendix F For MuJoCo tasks, we plot the mean return throughout training well the distribution of returns achieved at the end of training (i.e., the performance profile) (Agarwal et al. 2021 For GridWorld, we plot the fraction of times agents find the optimal goal (i.e., the success rate).
Results. Fig. (Ba shows that on-policy sampling has approximately 77% success rate on GridWorld, whereas PROPS and ROS achieve 100% success rate. Fig. 4a and Table I show that PROPS can achieve the same returns as PPO and PPO-Buffer with fewer environment interactions. For instance, in Humanoid-v4 and Hopper-v4, PROPS closely matches the final return of PPO with 33% fewer interactions. To provide more compact and read- able summary of performance throughout training, also report results aggregated across all six Mu- JoCo tasks in Fig. 5a PROPS matches the normal- ized return of PPO and PPO-Buffer using only 60% of the training budget. Moreover, the performance
GridWorid-5x5-vo
GridWorld-5x5.
1.0
Success Rate
0.8
Sampling ling Error
G.8
666
0.6
0.4
0.4
PROPS
PROPS
ROS
0.2
Policy
Sampling
0.2
0.0
Timesstyp
Timestep
(a) Success rate.
(b) Sampling error.
Figure 6: GridWorld RL experiments over 50 sceds.
profiles of PROPS in Fig. [bb and Fig. 5b are almost always above the performance profiles of PPO and PPO-Buffer, indicating that any given run of PROPS is more likely obtain higher return than PPO- Buffer. Thus, we affirmativlyy answer Q2 posed the start of this section: PROPS increases the fraction of training with high return and increases data efficiency.
11

---

Published in Transactions on Machine Learning Research (2/2026)
PROPS
On-Policy Sampling
Swimmer-v4 V4
Hopper vl
HaitCheetah vi
Walker20 14
Ant
Humanoid 14
0.088
Sarrping Enor
Sarpping Enour
Enor
Samppegg Ener
9.050
Sarreino Enct
0.095
Sen Senpine Enor
0.050
Samping
2.012
0.020
0,000
0.045
0.080
0.045
0010
0,020
0,030
0.0
0.040
0.080
0.036
0.008
0.016
0.050
0.5
2.5
Timeslap
Timestep
106
Timeslap
166
Timestep
Timestep
Figure 7: Sampling error throughout RL training. Solid curves denote the mean over 5 seeds. Shaded regions
denote 95% confidence belts.
Having established that PROPS improves data efficiency, we now investigate if PROPS is appropriately adjusting the data distribution of the bulfer by comparing the sampling error achieved throughout training with PROPS and PPO-Buffer. Training with PROPS produces a different sequence of target policies than training with PPO-Buffer produces. To provide a fair comparison, we compute sampling error for PPO- Buffer using the target policy sequence produced by PROPS. More concretely, we fill a second buffer with on-policy samples collected by the larget policies produced while training with PROPS and then compute the sampling error using data in this buffer.
As shown in Fig. 7 PROPS achieves lower sampling error than on-policy sampling throughout RL training in 5 out of 6 tasks. In HalfCheetah-v4, PROPS decreases sampling error only in the first 400k timesteps, but nevertheless improves data efficiency. This result likely reflects our hyperparameter tuning procedure in which we selected hyperparameters yielding the largest return (Appendix F). Although lower sampling error intuitively correlates with increased data efficiency, it is nevertheless possible to achieve high return without roducing sampling error. Alternatively, the carly reduction in sampling error itsclf may be sufficient to explain the observed efficiency gains. In GridWorld, PROPS and ROS reduce sampling error in the first 300 steps and closely match on-policy sampling afterwards. We use batch size of 80 in these experiments, and as the target policy becomes more deterministic, larger batch sizes are needed to observe differences between PROPS and on-policy sampling
We ablate the effects of the clipping coefficient EpRors, regularization coefficient X, and buffer size b in Appendix D.3 Without clipping or without regularization, PROPS often achieves greater sampling error than on-policy sampling, indicating that both help to keep sampling error low. Moreover, data efficiency generally decreases when we remove clipping regularization, showing both are essential to PROPS. We find that data efficiency may decrease with a larger buffer size. Intuitively, the more historic data kept around, the more data that must be collected to impact the aggregate data distribution. Last, we show that PROPS is robust to hyperparameter choices using a sensitivity analysis in Appendix Thus, we affirmatively answer Q1 posed at the start of this section: PROPS achieves lower sampling error than on-policy sampling when the target policy is fixed and during RL training.
7 Discussion
This work has shown that adaptive, off-policy sampling can reduce sampling error in data collected through- out RL training and increase the data efficiency of on-policy policy gradient algorithms. In this scction, we discuss limitations of our work and present opportunities for future research.
Convergence of PROPS. PROPS builds upon the ROS algorithm of Zhong et al. (0022) While Zhong ct al. (2022) focused on thcoretical analysis and policy evaluation in low-dimensional domains, we chose focus on empirical analysis with policy learning in standard RL benchmarks. Proposition and previous convergence results by Zhong et al. (2022) focus tabular MDPs and provide conceptual grounding for PROPS, but this theory does not immediately extend to continuous MDPs. An important direction for future work would be theoretical analysis of PROPS, in particular whether PROPS also enjoys the same faster convergence rate that was shown for ROS relative to on-policy sampling in continuous MDPs
6 As policy becomes deterministic, sampling error approaches zero, so there less sampling error for PROPS correct.
12

---

Published in Transactions on Machine Learning Research (2/2026)
Advantage Estimates of Historic Data. While PROPS can reduce sampling error in off-policy historic data, the advantage estimates associated with this data still correspond to historic policies and are thus biased advantage estimates of the current policy. PPO computes advantages using GAE (Schulman et al, 2016) with parameter À E [0, controlling how much the advantage estimates weight multi-step Monte Carlo returns versus value function predictions. In practice, PPO uses a relatively large X, typically À = 0.95, which heavily weights Monte Carlo returns. Since these returns are from trajectories generated by a behavior policy correcting sampling error w.r.t. historic policy, the returns correspond to the historic policy, not the current one. In other words, PROPS corrects the off-policy bias of the (s, a) distribution but not that of the advantage estimates. Empirically, this bias appears minor, as PROPS still improves data efficiency. Nevertheless, it can be mitigated by decreasing À so GAE places greater weight on value function predictions, which are updated to reflect the current policy and will thus produce advantage est estimates that better align with the current policy.
Advantago- Advantago-Weighted Weighted Sampling Error Correction. A limitation of PROPS is that the update in- diseriminately increases the probability of under-sampled actions without considering their importance in gradient computation. For instance, if an under-sampled action has zero advantage, it has no impact on the gradient and need not be sampled. An interesting dircction for future work could be to prioritize correct- ing sampling error for (s, a) that have the largest influence on the gradient estimate, i.e., those with large advantages (positive or negative).
Sampling Error Correction for Off-Policy RL. Beyond these more immediate directions, our work opens up other opportunities for future research. A less obvious feature of the PROPS behavior policy update is that it can be used to track the empirical data distribution of any desired policy, not only that of the current policy. This feature means PROPS has the potential to be integrated into off-policy RL algorithms and used so that the empirical distribution more closely matches a desired exploration distribution. Thus, PROPS could be used to perform focused exploration without explicitly tracking state and action counts.
8 Conclusion
In this work, we ask if adaptive, off-policy action sampling can produce data that more closely matches the expected on-policy data distribution and improve the data efficiency of on-policy policy gradient methods. To answer this question, we introduce Proximal Robust On-policy Sampling (PROPS), a method that periodically updates data collection behavior policy to increase the probability of sampling actions that are currently under-sampled with respect to the on-policy distribution. Furthermore, rather than discarding collected data after every policy update, PROPS permits more data-efficient on-policy learning by using data collection to adjust the distribution of previously collected data to be approximately on-policy. We replace on-policy sampling with PROPS to collect data for the popular PPO algorithm and empirically demonstrate that PROPS produces data that more closely matches the expected on-policy distribution and yields more data-efficient learning compared to on- on-policy sampling.
Acknowledgments
We thank Adam Labiosa, Abhinav Harish, and Brahma Payse for discussion and comments that improved the final version of this work. This work took place the Prediction and Action Lab (PAL) at the University of Wisconsin Madison. PAL research is supported by the National Science Foundation (TTS-2410811) and the Wisconsin Alumni Research Foundation.
13

---

Published in Transactions on Machine Learning Research (2/2026)
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Ngge Akkaya, Marcin Andrychowiez, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arX:::1910.071: 2019.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. ar preprint arXiv: 1912. 06680, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.0155 01540, 2016.
Lasse Espcholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416, PMLR, 2018.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy optimization. In Uncertainty in Artificial Intelligence, pp. 1017-1027. PMLR, 2020.
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E' Turner, and Sergey Levine. Q-prop: Sample- efficient policy gradient with an off-policy critic. arXiv preprint arXiu:1611.0224 2016.
Shixiang Shane Gu, Timothy Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprinl 11v: 05905, 2018.
Josiah P Hanna, Scott Niekum, and Peter Stone. Importance sampling in reinforcement learning with an estimated behavior policy. Machine Learning, 11066):1267-1317, 2021.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274): 18, 2022. URL /ttppppprr.rrr/ppprrse/r// -23/21-1342.htmm html
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. UtLppttp://arxxi..oo org/ abs//412.6990
Ksenia Konyushova, Yutian Chen, Thomas Painc, Caglar Gulcchre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active ofline policy selection. Advances Neural Information Processing Systems, 34:24631 24644, 2021.
Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estimation. In Artificial Intelligence and Statistics, pp. 608 616. PMLR, 2015.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 3053 3062. PMLR, 2018.
14

---

Published in Transactions on Machine Learning Research (2/2026)
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ar preprinl arXiv:1509.028 2015.
Shuze Daniel Liu, Claire Chen, and Shangtong Zhang. Doubly optimal policy evaluation for reinforcement learning. arXiv preprint ar.iv::440 10. 00.0226, 2024.
Volodymyr Mnih, Adria Puigdomenoch Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national conference on machine learning, pp.
PMLR, 2016.
Subhojyoti Mukherjee, Josiah P Hanna, and Robert D Nowak. Revar: Strengthening policy evaluation via reduced variance sampling. In Uncertainty in Artificial Intelligence, pp. 1413 1422. PMLR, 2022.
Yusuke Narita, Shota Yasui, and Kohei Yata. Efficient counterfactual learning from bandit feedback. In Proceedings of the AAAI Conference Artificial Intelligncce volume 33, pp. 4634-4641, 2019.
Chris Nota and Philip S Thomas. Is the policy gradient a gradient? arXiv preprint 1xXi::106.0007:: 2019.
Brendan O'Donoghue, Rémi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Pgq: Combining policy gradient and q-learning, ArXiv, abs/1611.01633 2016.
lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances Neural Information Processing Systems, 26, 2013.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Matteo Papini, Giorgio Manganini, Alberto Maria Metelli, and Marcello Restelli. Policy gradient with active importance sampling. arXiv preprint Xiv:2400 05630, 2024.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pp. 2778 2787. PMLR, 2017.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Xiv preprint arXiv:2304.01315, 2023.
Brahma Pavse, Ishan Durugkar, Josiah P. Hanna, and Peter Stone. Reducing sampling error in batch temporal difference learning. In International Conference on Machine Learning. 2020.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Martin L Puterman. Markow decision processes: discrele slochastic dynamic programming. John Wiley & Sons, 2014.
James Queeney, Ioannis Ch. Paschalidis, and Christos G. Cassandras. Generalized proximal policy opti- mization with sample reuse. In Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-bassllnns::: Reliable reinforcement learning implementations.. Journal of Machine Learning Re- search, 22(268):118 8, 2021. URL htjt://jmlr... otgg/pppere//22220-136444
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbool. High-dimensional con- tinuous control using generalized advantage estimation. In International Conference on Learning Repre- sentalions (ICLR), 2016.
15

---

Published in Transactions on Machine Learning Research (2/2026)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithins. Xr preprinl arXin:1707. 707.06347, 2017.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. In- trinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.nel, 2018. URL https:://ppenreviieet http:///wwwseeeeefoooooooossss/a
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in newral information processing systems, 30, 2017.
Aaron David Tucker and Thorsten Joachims. Variance-optimall augmentation logging for counterfactual evaluation in contextual bandits. arXiv preprint arXiv:2202.011 01721, 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nalure, 575(7782):350 354, 2019.
Runzhe Wan, Branislav Kveton, and Rui Song. Safe exploration for efficient policy evaluation and comparison. In International Conference on Machine Learning, pp. 22491 -25111. PMLR, 2022.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukeuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXic:1611.0192) 2016.
Ronald J Williams. Simple statistical maddientfoolloing algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5-32, 1992.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:11501-11516, 2021.
Rujie Zhong, Duohan Zhang, Lukas Schäfer, Stefano Albrecht, and Josiah Hanna. Robust on-policy sampling for data-efficient policy evaluation reinforcement learning. Advonces in Neural Information Processing Systems, 35:37376-311 -73388, 2022.
16

---

Published in Transactions on Machine Learning Research (2/2026)
Appendix Contents
A Core Theoretical Results
17
B Additional Theoretical Results
18
PROPS Implementation Details
20
D Additional Experiments
21
D.1 Bias and Variance of PROPS
21
D.2 Correcting Sampling Error for Fixed Target Policy
22
D.3 Correcting Sampling Error During RL Training
23
Runtime Comparisons
25
Hyperparameter Tuning for RL Training
25
Hyperparameter Sensitivity Analysis
26
A Core Theoretical Results
In this section, we present the proof of Proposition We use dm: Tm, and Pm as the empirical state visitation distribution, empirical policy, and empirical transition probabilitics after statc-action pairs have been taken, respectively. That is, (s) is the proportion of the states that are (m(a))) is the proportion of the time that action was observed in state and Pm a) is the proportion of the time that the state changed to s' after action a was taken in state s.
Proposition Assume that data collected with on adaptive behawior policy that always takes the most srnder-sample action in each state, s, with respect to policy n, i.e., + arg max. (x(alss) (a)sss))) We further assume that S and A are finite and that the Markov chain induced by is irreducible. Then have that the empirical slate visilation distribution, dmn converges to the state distribution of dx, with probability 1:
Va, lim (s) = (d))) m2-20
Proof. The proof of this theorem builds upon Lemma 1 and 2 by Zhong et al. (002)). Note that these lemmas superficially concern the ROS method whereas we are interested in data collection by taking the most under-sampled action at each step. However, as stated in the proof by Zhong et al. (2022). these methods are equivalent under an assumption that they make about the step-size parameter of the ROS method. Thus, we can immediately adopt these lemmas for this proof.
Under Lemia 1 of Zhong el al. (2022). we have that limm Tm (als) nt (uls) for any state under this adaptive data collection procedure. We then have the following Vs:
lim dm (s) lim ΣΣ m-+00
Pmn
lim Pm m=+00 8
à
Σ
lim Pm (m m-+ m=
lim nm (ā)3)
lim dm (5)
m0
(b)
p( ())()) lim (5).
17

---

Published in Transactions on Machine Learning Research (2/2026)
Here, (a) follows from the fact that the empirical frequency of state can be obtained by considering all possible transitions that lead to The last line, (b), holds with probability by the strong law of large numbers and Lemma 2 of Zhong et al. (2022).
We now have a system of [S] variables and |S] linear equations. Define variables r(s) = mimm- dm (s) and let E R| be the vector of these variables. We then have x == p where pa E R|S the transition matrix of the Markov chain induced by running policy æ. Assuming that this Markov chain is irreducible, d is the unique solution to this system of equations and hence limm- dm (s),), Vs.
B Additional Theoretical Results
In this section, we provide additional theory to describe the relationship between different hyperparameters in PROPS:
The amount of sampling error in previously collected data and the size of behavior policy updates.
2. The amount of historic data retained by an agent and the amount of additional data the behavior policy must collect to reduce sampling error.
For simplicity, we first focus on a simple bandit setting and then extend to a tabular RL setting.
Suppose we have already collectod m state-action pairs and these have been observed with empirical dis- tribution (a). From what distribution should we sample an additional k state-action pairs that the empirical distribution over the k samples is equal in expectation to TT?
Proposition 3. Assume that m actions have been collected by running some policy Te(a) and (a) is the empirical distribution on this dataset. If we collect an additional h state-action pairs using the following distribution, and if k))o((a) 2 m mm Tm(a), then the aggregale empirical distribution over the m + k pairs is equal to Ta(a) in expectation:
(a) = 7 Ta(a) + m (5o(a) - Fm (m((a))
where Z = EaEA (T((a) m
& (Te(a)= =m is normalization coefficient.
Proof. Observe that (m + ()) is the expected number of times sampled under To after m++ k steps, m Fm (a) the number of times each a was sampled thus far, and k ma(a) the expected nunber of times is sampled under behavior policy after steps. We want to choose Ta(a) such that (m k))o((a) Tm (a) + k Ta(a) in expectation.
(m (a) = k TT(a)) + m nm (a)
k (T(a) m (a) ()) (m + (a)
(a) = - nm (m((a)) + (Th m + 1 (a
m
k
= (a) + (me(a) nm (a))
Note that To(a) will be valid probability distribution after normalizing only if
(a) + (To(a) Tm (a)) 2
m
ग. (a) + (e(a) 2 m k
k
(m + k) (e(a) 2 m nm T (a).
If (m + k) (a) < m (a), then prior to collecting additional data with our behavior policy, already appears in our data more times than it would in expectation after m steps under ne- In other words.
18

---

Published in Transactions on Machine Learning Research (2/2026)
would need to collect more than l additional samples achieve zero sampling error (or discond some previously collected samples).
When sampling error is large, behavior policy updates must also be large. Intuitively, the difference (e(a)- - Tm is the mismatch between the true and empirical visitation distributions, so adding this lerm to d7g adjusts dr reduce this mismatch. If TT(a) - Fm (a) < 0, then is over-sampled w.r.t. ne, and Ty will decrease the probability of sampling a. TT(a)) 7m Tm (a) > 0, then a is under-sampled w.r.t, me, and m will increase the probability of sampling a. When (T) - (m)) is small, the optimal ma(a) requires only small adjustment from To (i.e., small update to the behavior policy is sufficient to reduce sampling error). When (To(a) Tm mm(a) (a) is large, the optimal Tz(a) requires a large adjustment from To (i.e., a large update to the behavior policy is needed to reduce sampling error). We can increase (or decrease) the target KL cutoff 8PROPS to allow for larger (or smaller) behavior updates.
When we retain large amounts of historic data, the behavior policy must collect a large amount of additional data to reduce sampling error in the aggregate distribution. The m factor implies how much data we have already collected (m) and how much additional data will collect (k). If the k additional samples to collect represent small fraction of the aggregate m + k samples (i.e. & << m); then is large, and the adjustment to dx, is large. This case generally arises when we retain more and more historic data. If the additional samples to collect represent a large fraction of the aggregate m + k samples (i.e. A >> m), then m is small. This case generally arises when we retain little to no historic data.
that how much we adjust dxe depends
I is small, and the adjustment to d
The next proposition extends this analysis to the tabular RL setting.
Proposition 4. Assume that m slate-action pairs have been collected by running some policy and d (s, a) is the empirical distribution on this dataset. If we collect on additional state-action pairs using the following distribution, and (m + k), a) m dm (s, a), then the aggregate empirical distribution over the m + k pairs is equal to dao (8, a) in expeclation:
a) = Z (s. a) + m
1
k (dro (s, a) - dm a))
where Z Ela.a)Exxa (o. a) + the (dxo (s, - dm (s) is normalization coefficient.
Proof. The proof is identical to the proof of Proposition replacing Th Ta)a), Te (a), and ma(a) with dzo (s, a), dm (s, a), and do(s,
practice, we cannot sample directly from the visitation distribution d,(s, a) Proposition and in- stead approximate sampling from this distribution by sampling from its corresponding policy ma(als) do(8, a)/ Ea'es d
19

---

Published in Transactions on Machine Learning Research (2/2026)
Empirical policy up
Target policy mı(a|s)
PROPS Behavior policy mp(als) without regularization
PROPS Behavior policy nt(aas) with regularization
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
-2
0
2
Figure In this example, <(-ls) (O 1). After several visits 8, all sampled actions (blue) satisfy 0 so that actions < 0 are under-sampled. Without regularization, PROPS will attempt to increase the probabilities of under-sampled actions in the tail of target policy distribution (green). The regularization term in the PROPS objective ensures the behavior policy remains close to target policy.
g a. Φ, 0 >
Is the objective clipped?
Return value of min
Gradient
g(80, Φ 0)
ePROPs; + éprOps
No
[[,, a, Φ, 0)
VaLcur
g(8, a, Φ 0) + (pROPs
No
g(s, a, Φ, 0)
V
g(8,a, 0,
EPROPS
Yes
(1 <PROPS)
Table 2: Behavior of PROPS's clipped surrogate objective (Fq.
C PROPS Implementation Details
In this appendix, we describe two relevant implementation details for the PROPS update (Algorithm additionally summarize the bchavior of PROPS's clipping mechanism in Table 2
2 We
1. PROPS update: The PROPS update adapts the behavior policy to reduce sampling error in the buffer D. When performing this update with a full buffer, we exclude the oldest batch of data collected by the bchavior policy (i.e., the m oldest transitions in D); this data will be evicted from the buffer before the next behavior policy update and thus does not contribute to sampling error in D.
2. Behavior policy class: We compute behavior policies from the same policy class used for target policies. In particular, we consider Gaussian policies which output a mean yus) and a variance d²(s) and then sample actions a ~ T(-(s) = ((((s), In principle, the target and behavior policy classes can be different. However, using the same class for both policies allows us easily initialize the behavior policy equal to the target policy at the start of each update. This initialization is necessary to ensure the PROPS update increases the probability of sampling actions that are currently under-sampled with respect to the larget policy.
20

---

Published in Transactions on Machine Learning Research (2/2026)
Ic-18
GridWorld-5x5-vd 5x5-v0- Fixed Policy
10-3
GridWorld-5x5-vV. Fixed Policy
Oracle Adaptive Sampling
Oracle Adaptive Sampling
PROPS
PROPS
On-Policy Sampling
On-Policy Sampling
10-4
0.5
Variance estimate
Blas estimate
0.0
10-5
-0.5
10-6
-1.0
10-2
4000 6000
10000
10a
10a
10f
Timesteps
Timesteps
(a) Sampling error bias
(b) Sampling error variance
Figure 9: Sampling error bias and variance estimates of different sampling methods. Empirically, PROPS is unbiased and lower variance than on- on-policy sampling.
Additional Experiments
In this appendix, we include additional experiments and ablations.
D.1 Bias and Variance of PROPS
In Fig. we investigate the bias and variance of the empirical state-action visitation distribution dp(s, a) under PROPS, ROS, and on-policy sampling. We report the bias and variance averaged over all (8, a) E SxAA computed as follows:
bias
Σ (s.0)ESxA
(E [dv(s, a)] - d:o (s, a))
(13)
S A
variance
Σ
E (dd(s. a) dxo (s, (s,aa)² a))²
(14)
SS A
(a.0)ESxA
As shown in Fig. 9 the visitation distributions under PROPS and ROS empirically have near zero bias (note that the verlical axis has scale 10-15) and have lower variance than on-policy sampling.
21

---

Published in Transactions on Machine Learning Research (2/2026)
PROPS
ROS
On-Policy Sampling
Swimmer-y4
Sampling Error
Hopper-v4
Sampling Error
Walker2d-v4
Sampling Error
HalfCheetah-vv
Sampling Error
Ant-v4
Sampling Error
Humanoid-v4
Sampling
10
10
10
10
10
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
Samples Collected
Samples Collected
Samples Collected
Samples Collected
Sampies Collected
Samples Collected
Figure 10: Sampling error with fixed, expert target policy, Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
PROPS
PROPS, no regularization PROPS, no clipping, no regularization
ROS
PROPS, no clipping
On-Policy Sampling
Swimmer-v4
Hopper-v4
Walker2d-v4
10
Sampling Error
Sampling Error
Sampling Error
10
10
1024
2048
4096
8192
1024
2048
4096
B192
1024
2048
4096
8192
Samples Collected HallCheetan-wd
Samples Collected Ant-v4
Samples Collected Humanoid-v4
10"
Sampling Error
Sampling Eroo
Sampling Error
10"
10
1024
2048
4096
8192
1024
2048
4096
8192
1024
2048
4096
8192
Samples Collected
Samples Collected
Samples Collected
Figure 11: Sampling error ablations with a fixed, random target policy. Here, "no clipping" refers to setting EPROPS = co, and "no regularization" refers to setting À = 0. Solid curves denote the mean over 10 seeds, and shaded regions denote 95% bootstrap confidence intervals.
D.2 Correcting Sampling Error for a Fixed Target Policy
In this appendix, expand upon results presented in Section 6.2 of the main paper and provide additional experiments investigating the degree to which PROPS reduces sampling error with respect to fixed, ran- domly initialized target policy. We additionally include ablation studies investigating the effects of clipping and regularization.
We tune PROPS and ROS using a hyperparameter sweep. For PROPS, we sweep over learning rates in (10-3,10-4] and fix the remaining PROPS hyperparameters: regularization cocfficient À 0.1, target KL 6PROPS 0.03, and clipping coefficient éPROPS 0.3. For ROS, we sweep over learning rates in 10-4,1 We report results for hyperparameters yielding the lowest sampling error.
22

---

Published in Transactions on Machine Learning Research (2/2026)
PROPS, clipping
OS
0.016
0.018
3.020
Swimmer vd
8
0.034
0.030
0.032
Hopper-44 94
0.087E
0.0834
0.0826
HelCCheetah 14
0.084
0.090
Welker2d vi
0.15
0.18
Ant.v
0.040
044
Humaneld vb
3 0.006
4.001
0.61
0.005
0.010
0.028
0.222
0.024
0.0000
0.0578
0.0528
0.0030
0.050
0.045
0.030
0.13
0.12
2.114
6 0.698
0.030
0.002
0.034
0.028
0.4 0.0 1.0
0.0 C.4
1.0
2.0
Times/eep
extep
166
Timeslep
Timeotep
Timeslop
Figure 12: Sampling error throughout RL training without clipping the PROPS objective. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
PROPS, no regularization
On-Policy Sampling
Hopper
HeilChesiah v4
Humanole
0.3380
0.084
0.096
0.080
00013
④ 0.0000
3 0.0525
20.0275
@ 0.000
0.062
0.054
Sampte tag Emgi
0.070
0.075
@ 0..00
9 0.034
3 0.040
0.036
mmm
0.008
0.009
0.0225
0,0000
0.0220
0.3176
0.052
0.000
0.936
0.054
0.844
0.842
0.043
0.046
0.065
0.080
@
0.000
0.028
0.028
0.039
0.0 0.2 0.4 0.6 0.0 1.0
0.0 0. 0.4 0.6
0.8 1.0
0.0
0.5
1.9
1.5
20
0.0
0.5
1.0
1.5
2.0
2
4
Timestap 1e5
Timaslap
1eB
Thr eelap
106
Timeslip
106
Timeolop
1c6
Timaslap 1e6
Figure 13: Sampling error throughout RL training without regularizing the PROPS objective. Solid curves denote the mean over 5 seeds. Shaded regions denote 95% confidence belts.
In Fig. 10 we see that PROPS achieves lower sampling error than both ROS and on-policy sampling across all tasks. ROS shows little to no improvement over on-policy sampling, again highlighting the difficulty of applying ROS to higher dimensional tasks with continuous actions.
Fig. I1 ablates the effects of PROPS's clipping mechanism and regularization on sampling error reduction. We ablate clipping by setting EPROPS oo, and we ablate regularization by setting À = O. We use fixed expert target policy and use the same tuning procodure described carlier in this appendix. In all tasks, PROPS achieves higher sampling error without clipping or regularization than it does with clipping and regularization, though this method nevertheless outperforms on-policy sampling in all tasks. Only removing clipping increases sampling error in most setups, and only removing regularization often increases sampling error for smaller batches of data e.g.. 1024 samples. These observations indicate that while regularization is helpful, elipping has a stronger effect on sampling error reduction than regularization when the target policy is fixed.
D.3 Correcting Sampling Error During RL Training
In this appendix, include additional experiments investigating the degree to which PROPS reduces sampling error during RI training, expanding upon results presented in Section 6.3 of the main paper. We include sampling error curves for all six MuJoCo benchmark tasks and additionally provide ablation studies investigating the effects of clipping and regularization on sampling error reduction and data efficiency in the RL setting. We ablate clipping by tuning RT. agents with éPROPS = 20, and we ablate regularization by tuning RL agents with X O. Fig. 12 and Fig. 13 show sampling error curves without clipping and without regularization, respectively. Without clipping, PROPS achieves larger sampling error than on- policy sampling in all tasks except Humanoid. Without regularization, PROPS achieves larger sampling error in 3 out of 6 tasks. These observations indicate that while clipping and regularization both help reduce sampling error during RL training, clipping has a stronger effect on sampling error reduction. As shown in Fig. 14 PROPS data efficiency generally decreases when we remove clipping or regularization.
Lastly, we consider training with larger buffer sizes b in Fig. [5] We find that data efficiency may decrease with a larger buffer size. Intuitively, the more historic data kept around, the more data that must be collected to impact the aggregate data distribution.
23

---

Published in Transactions on Machine Learning Research (2/2026)
PROPS
PROPS, no regularization
PPO
PROPS, no clipping
PPO-Buffer
Swimmer-v4
Hopper-v4
HalfCheetah-v4
3000
100
3000
Mean Return
2000
2000
50
1000
1000
0
0.0
0.5
1.0
0.0
0.5
1.0
0
1
2
Timestep
1e6
Timestep Ant-v4
1e6
Timestep
1e6
Walker2d-v4
Humanoid-v4
4000
6000
Mean Return
3000
4000
4000
2000
2000
2000
1000
0
0
0
1
2
0
2
4
0.0
2.5
5.0
Timestep
1e6
Timestep
1e6
Timestep
1e6
Figure 14: Mean return over 50 seeds of PROPS with and without clipping or regularizing the PROPS objective. Shaded regions denote 95% bootstrap confidence intervals.
b b= 2
b=3
b=4 4
b= 8
Swimmer-v4
Hopper-v4
HalfCheetah-v4 v4
Mean Return
Mean Return
Mean Return
3000
100
3000
2000
2000
50
1000
1000
0
0.0
0.5
1.0
0.0
0.5
1.0
0
2
Timestep
1e6
Timestep
1e6
Timestep
1e6
Walker2d-v4
Ant-v4
Humanoid-v4
Mean Return
4000
Mean Return
Mean Return
4000
3000
4000
2000
2000
2000
1000
0
2
0.0
2.5
5.0
Timestep 1e6
Timestep 1e6
Timestep 1e6
Figure 15: Mean return over 50 seeds for PROPS with different buffer sizes. We exclude b 8 for Humanoid- v4 due to the expense of training and tuning. Shaded regions denote 95% bootstrap confidence intervals.
24

---

Published in Transactions on Machine Learning Research (2/2026)
Swimmer-y
Hopper-v4
HallCheslah--v
Walker2d-v4
1500
PROPS, PPO Dutter PPO
3000
PROPS, PRO--uuter PP0
Duter
2000
Butte
600
PP10
FP1
1000
Runtime
Runlime
Runtime
Runlime
400
1000
500
200
500
Figure 16: Runtimes for PROPS, PPO-Buffer, and PPO. We report means and standard errors over 3 independent runs.
PPO learning rate
1024, linearly annealed to over training
10-3, 10
PPO batch size
8192
PROPS learning rate
10-3 10-4 (and 10 5 for Swimmer)
PROPS behavior update frequency
256, 512, 1024
PROPS KL cutoff ornors
0.03, 0.05, 0.1
PROPS regularizer coefficient
0.1,0.3
Table 3: Hyperparameters used in our hyperparameter sweep for RL training.
E Runtime Comparisons
Figure 16 shows runtimes for PROPS, PPO-Buffer, and PPO averaged over 3 runs. We trained all agents on a MacBook Air with an MI CPU and used the same tuned hyperparameters used throughout the paper. PROPS takes at most twice long as PPO-Buffer; intuitively, both PROPS and PPO-Buffer learn from the same amount of data but PROPS learns two policies.
We note that PPO-Buffer is faster than PPO in HalfCheetah-vd because, with our tuned hyperparameters, PPO-Buffer performs fewer target policy updates than PPO. In particular, PPO-Buffer is updating its target policy every 4096 steps, whereas PPO updating the target policy every 1024 steps.
F Hyperparameter Tuning for RL Training
For all RL experiments in Section 6.3 and Appendix D.3 we tune PROPS, PPO-Buffer, and PPO separately using a hyperparameter sweop over paramcters listed in Table 3 and fix the hyperparameters Table 3 across all experiments. Since we consider wide range hyperparameter values, we ran 10 independent training runs for each hyperparameter setting. We then performed 50 independent training runs for the hyperparameter settings yielding the largest returns at the end of RL training. We report results for these hyperparameters in the main paper.
PPO Batch Size
PPO Learning Rate
PROPS Update Freq.
PROPS Learning Rate
PROPS KL Cutoff
PROPS
Environment
Regularization
Swimmer-v4
2048
10
1024
10-5 3
0.03
0.1
Hopper-v4
2048
10-3 3
256
10-3
0.05
0.3
HalfCheetah-v4
1024
10-4
512
10-3
0.05
0.3
Walkor2d-v4
2048
10-3
256
10-3
0.1
0.3
Ant-v4
2048
10-4
256
10-3
0.03
0.1
Humanoid-v4
8192
10
256
10-4
0.1
0.1
Table 4: Hyperparameters selected from our hyperparameter sweep for RL training.
25

---

Published in Transactions on Machine Learning Research (2/2026)
PPO number of update epochs
10
PROPS number of update epochs
16
Buffer size
target batches (also 3, 4, and 8 Fig. 15
PPO minibatch size for PPO update
bm/16
PROPS minibatch size for PROPS update
bn/16
PPO and PROPS networks
Multi-layer perceptron
with hidden layers (64.66 (4.644)
PPO and PROPS optimizers
Adam KKnngma & Ba 2015
PPO discount factor
0.99
PPO gencralized advantage estimation (GAE)
0.95
PPO advantage normalization
Yes
PPO loss clip coefficient.
0.2
PPO entropy coefficient
0.01
PPO value function coefficient
0.5
PPO and PROPS gradient clipping (max gradient norm)
0.5
PPO KI. cut-off
0.03
Evaluation frequency
Every 10 target policy updates
Number of evaluation episodes
20
Table 5: Hyperparamcters fixed across all experiments.
We use the PPO implementation provided by
CleanRL Huang et al. 2022
G Hyperparameter Sensitivity Analysis
To assess how sensitive PROPS is to hyperparameter selection, we follow Patterson al. (2023) and plot RL training performance aggregated over a large subset of runs from our hyperparameter sweep. Although inchude the PPO target learning rate and PPO target batch size in our hyperparameter sweep, keep them fixed their tuned values in violin plots. Including these hyperparameters would tell us about the sensitivity of PPO, but we are only interested in understanding the sensitivity of PROPS. We run 10 seeds for each hyperparameter setting. Fig. I7 and Fig, show training curves and performance profiles for tuned PROPS and aggregated PROPS as well as our baselines. Across tasks, tuned PROPS achieves higher returns than aggregated PROPS, demonstrating the benefit of careful hyperparameter tuning. Nevertheless, aggregated PROPS still outperforms PPO and PPO-Buffer, suggesting that PROPS maintains degree of hyperparameter robustness.
26

---

Published in Transactions on Machine Learning Research (2/2026)
PROPS (tuned)
PROPS (aggregate)
PPO-Buffer
PPO
Swimmer-v4
Hopper-v4
HalfCheetah-v4
Average Return
100
3000
3000
2000
2000
50
1000
1000
0.0
0.5
1.0
0.0
0.5
1.0
2
Timestep
1e6
Timestep Ant-v4
1e6
Timestep Humanoid-v4
1e6
Walker2d-v4
4000
Average Return
4000
4000
2000
2000
2000
0
2
2
4
0.0
5.0 Timestep
2.5
Timestep
1e6
Timestep
1e6
1e6
Figure 17: Mean return over 50 sceds. Shaded regions denote 95% confidence intervals.
PROPS (tuned)
PROPS (aggregate)
PPO-Buffer
PPO
Swimmer-v4
Hopper-v4
HalfCheetah-v4
1.00
1.00
1.00
Fraction runs with score >
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
100
0
2000
4000
0
2000
4000
Return (T)
Return (T)
Return (T)
Walker2d-v4
Ant-v4
Humanoid-v4
1.00
1.00
1.00
Fraction of runs
with score >
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
2500
5000
0
2500
5000
2500
5000
Return (T)
Return (T)
Return (T)
Figure 18: Performance profiles over 50 seeds. Shaded regions denote 95% confidence intervals.
27


Reviews:
[{"review_id": "vZWsvdyWSG", "reviewer": "Reviewer_a4R9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper builds on the Robust On-Policy Sampling (ROS) algorithm to reduce variance in the sampling process and obtain more accurate policy gradient estimates. The proposed method, PROPS, incorporates several techniques used in PPO to stabilize the ROS updates. This additional stability is shown to be beneficial in gridworld and continuous control benchmark tasks by reducing sampling error and improving policy optimization. \nAblations show the necessity of the additions to ROS. \nAdditionally, a new theoretical result is proved, generalizing one for the original ROS paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims concerning the purported benefits of PROPS to reduce sampling error and its advantages in continuous control tasks over the original ROS algorithm are supported by proper experiments in a few environments. \nWhile the diversity of environments is fairly limited, the experimental methodology seems sound with sufficient number of seeds and appropriate ablations. \n\nI think the paper could benefit from some additional exposition for the ROS method since PROPS builds on it directly. It was initially unclear how the ROS gradient estimate is useful for addressing the undersampling/oversampling issue.", "audience_yn": "Yes", "audience_explanation": "The proposed algorithm is a generic method to reduce sampling error when one can control the sampling distribution. While the core of the algorithm consists of the original ROS method, PROPS showcases that the method can work in more complex environments with some additional stabilizing techniques.\nAs such, I think this work would of interest to some members of the community, potentially beyond the RL sphere.", "requested_changes": "As mentioned above, it would help if there was a more in-depth description of ROS to make the paper clearer.\n\nMinor changes/questions:\n- Fig. 2, what's the oracle smallest variance possible with a given number of samples? \n\n- Fig. 10, could the orange lines be changed to some different color? It's currently hard to tell the difference.\n\n-  The text refers to Fig. 6b. and mentions  \"5 out of 6 tasks\" but the figure only shows GridWorld.\n\n- In the paragraph under equation (7), it is mentioned that \"We stop the PROPS update\nearly when $D_{KL}(\\pi_\\theat||\\pi_\\phi)$ reaches a chosen threshold $\\delta{PROPS}$\". Since there is also a KL regularizer between the two policies, are both of these methods necessary? It seems redundant.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "48iSW5GprT", "reviewer": "Reviewer_JSKr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops the idea of using off-policy data collection to obtain an on-policy data distribution in order to improve the Monte-Carlo estimate in on-policy policy gradient methods, an idea initially formalized in [1]. This correction improves the gradient estimates and, in turn, the efficiency and performance of on-policy policy gradient methods. The key contribution is to scale this approach to high-dimensional, continuous problems. To achieve this goal, the authors\n- remove the strong DAG assumption from [1] in the theoretical results for discrete state and action spaces.\n- introduce PPO-like gradient clipping to the objective introduced in [1]\n- add a KL regularization between behavior and target policy to the objective introduced in [1]\n\nStrengths and Weaknesses:\n\nS1: The paper makes a valuable contribution by scaling a recent sampling error correction method to standard deep RL benchmark problems by modifying the objective function with commonly used clipping and regularization terms.\n\nS2: The paper is well written and discusses findings and limitations in a transparent manner.\n\nW1: The theoretical reasoning heavily builds on [1] and solely considers finite-horizon MDPs with discrete state and action spaces, while the method is applied to infinite-horizon MDPs with continuous state and action spaces. From my understanding, scaling the approach to these more challenging settings is the key contribution of the paper. Thus, theoretical reasoning should be adapted to this type of MDP.\n\n[1] Zhong et al., Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning, NeurIPS 2022", "claims_and_evidence_yn": "Yes", "claims_explanation": "Theoretical results:\nAll theoretical results discuss finite state and action spaces and finite-horizon MDPs. The method, however, is evaluated in MuJoCo locomotion tasks, which have continuous state and action spaces and, to the best of my knowledge, are formulated as infinite-horizon MDPs (truncation rather than termination at the end of an episode). The authors discuss in the limitations section that they do not focus on the theoretical properties of the method. Nonetheless, I think the technical reasoning is rather weak, given that the key contribution is scaling findings from [1] to continuous spaces. While it is probably hard to give the same convergence guarantees as in the tabular setting, I would have expected some reasoning on whether findings of [1] carry over to continuous state and action spaces (and infinite horizon MDPs) and what potential problems might be.\n\nEmpirical results:\nThe empirical results are mostly convincing. A clear reduction in sampling error, as well as moderate efficiency and performance gains, can be observed. Some experiments are solely run with 5 random seeds (e.g., Fig. 3, 5). In those cases, it would be preferable to have at least 10 seeds.\nA limitation is the fact that the influence of the off-policy data generation on the advantage estimate is not considered. This is made clear by the authors, and I assume it is unlikely that the efficiency and performance gains are due to some effect on the advantage estimation rather than the sampling error correction. It is not trivial to find an ablation where it is possible to show this definitively, so I think the experiments can remain unchanged. However, if the authors have an idea how to do such an ablation, the paper would benefit from it.\n\n[1] Zhong et al., Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning, NeurIPS 2022", "audience_yn": "Yes", "audience_explanation": "On-policy policy gradient methods are of major importance in RL research and are probably the most common RL approach in applied fields like robotics. Improving the stability and efficiency of these methods is of interest to a large audience.", "requested_changes": "C1: The finite MDP setting seems a bit limiting. As far as I know, the MuJoCo locomotion tasks you benchmark on are infinite-horizon tasks as they truncate rather than terminate trajectories at the end of an episode, so at least empirically, your method seems to work in the infinite-horizon setting. Please elaborate on why the finite horizon assumption is necessary or provide results for infinite horizon MDPs.\n\nC2: Please discuss a bit more how and if results from [1] are applicable to continuous state and action spaces. I understand that this paper is more application-oriented than [1] and that an in-depth analysis of continuous spaces is not trivial. Nonetheless, it would be preferable to have a clear distinction between results that are easily transferable to continuous spaces and findings that are hard to prove in the continuous setting. In the latter case, it is ok to rely on intuition from the tabular setting, as long as this is made transparent.\n\nC3: Please discuss the computational overhead of your method in the empirical results. E.g., discuss the average run time of PROPS vs. PPO on your hardware.\n\nMinor:\n\nC4: Please make sure you have at least 10 random seeds for all experiments.\n\nC5: (Section 3.1) It is a bit surprising that you introduce discrete state and action spaces since your contribution is scaling the method to continuous spaces. I would prefer the more general continuous case, which can be easily done by replacing the probability mass functions (p, \\pi, d) with probability density functions.\n\nC6: (Section 4) Please introduce the acronym DAG (I assume directed acyclic graph).\n\nC7 : (Section 5) Before reading [1], it was not immediately clear to me why the loss function for adapting to the collection policy looks this way. To make the paper more self-contained, it would be good to explain that a bit more.\n\n[1] Zhong et al., Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning, NeurIPS 2022", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3Yk1FSDODt", "reviewer": "Reviewer_gaQF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the on-policy policy gradient RL without on-policy sampling. In particular, the paper builds upon [A1], which shows that instead of on-policy evaluation, one can use the existing method with some carefully crafted behavioral policy (ROS), which can result in sampling efficiency.  The main contributions over [A1] are that the work is only related to a finite action space; here, the paper considers a continuous action space. In particular, the paper proposes a KL regularization objective and PPO PPO-based technique to ensure that (i) the behavioral policy should not reweight the updates using the data that might have been obtained using a very old policy, and (ii) the behavioral policy does not move much from the target policy. The paper has conducted extensive empirical evaluations to validate its proposed approach. \n\nStrengths:\n1. The paper has explained the approaches and the rationale well enough. \n\n2. The empirical evidence suggests that the proposed approach indeed has an advantage (even though minute) over the ROS. \n\nWeaknesses:\n\n1. Even though the empirical results show improvements, they are very minute. \n\n2. The novelties are also very limited. In particular, it is eventually the ROS with some KL regularization and PPO-based clipping. Neither is novel to the RL. \n\n 3. In Figure 5, the sampling error fluctuate a lot, hence, it might be disputed to conclude that it has a better advantage over the on-policy sampling. \n\n[A1].Rujie Zhong, Duohan Zhang, Lukas Schäfer, Stefano Albrecht, and Josiah Hanna. Robust on-policy sampling\nfor data-efficient policy evaluation in reinforcement learning. Advances in Neural Information Processing\nSystems, 35:37376–37388, 2022.", "claims_and_evidence_yn": "No", "claims_explanation": "While the empirical evidence suggests some improvement for a continuous action space. However, it is not clear whether the proposed approach can have a significant advantage over the existing approaches. Also, for some tasks, the sampling error is increased. Further, the paper did not make any comparisons with other off-policy algorithms.", "audience_yn": "Yes", "audience_explanation": "This paper would be of interest to the RL community in general.", "requested_changes": "1. The paper provides a lot of examples of why on-policy sampling might not be a good idea for policy evaluation; one needs to use the existing data. In my opinion, it is unnecessary, as we have a pretty good idea from Zhong et al. Rather, I would suggest moving some of the theoretical results and explanations to the main text. \n\n2. The algorithm is similar to the two-time-scale approach as the target distribution is updated in the outer loop, and the behavioral policy is updated in the inner loop. Can the authors comment on the learning rates for both these loops?\n\n3. I would suggest comparing this approach with off-policy evaluation approaches (E.g. [A2])\n\n[A2]. Liu, Shuze Daniel, Claire Chen, and Shangtong Zhang. \"Doubly optimal policy evaluation for reinforcement learning.\" arXiv preprint arXiv:2410.02226 (2024). https://arxiv.org/pdf/2410.02226", "broader_impact_concerns": "N/A", "additional_comments": "1. As a metric, the authors used sampling error. I would suggest defining this formally, so that everyone can understand the implications. \n\n2. Can the authors comment on the theoretical properties? I mean can the authors show the convergence, and sampling efficiency with respect to RPO in any manner?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)