Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach
Authors: Baturay Saglam
Doğan Can Çiçek
Furkan Burak Mutlu
Suleyman Kozat
Venue: Accepted by TMLR ()

Abstract: Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent’s policy and collected data increases. Although the well-studied importance sampling and off-policy policy gradient techniques were proposed to compensate for this discrepancy, they usually require a collection of long trajectories and induce additional problems such as vanishing/exploding gradients or discarding many useful experiences, which eventually increases the computational complexity. Moreover, their generalization to either continuous action domains or policies approximated by deterministic deep neural networks is strictly limited. To overcome these limitations, we introduce a novel policy similarity measure to mitigate the effects of such discrepancy in continuous control. Our method offers an adequate single-step off-policy correction that is applicable to deterministic policy networks. Theoretical and empirical studies demonstrate that it can achieve a “safe” off-policy learning and substantially improve the state-of-the-art by attaining higher returns in fewer steps than the competing methods through an effective schedule of the learning rate in Q-learning and policy optimization.

Published in Transactions on Machine Learning Research (01/2024)
Mitigating Off-Policy Bias in Actor-Critic Methods with One- Step Q-learning: A Novel Correction Approach
Baturay Saglam
haturay.soglam@@yle..d
Deportment of Electrical Engineering
Yale University
Dogan C. Cicck
cicekwee. beick&eee.ilkenn.. edu. fr burak. murakknutlu@ee.. bilkent. edu. Rozat@ee. bilkent. edu.tr
Furkan B. Mutlu
Suleyman S. Kozat.
Department of Electricol and Electronics Engineering Bilkent Universily
Reviewed on OpenReview:
Abstract
Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent's 's policy and collected data increases. Although the well-studiod importance sampling and off-policy policy gradient techniques were proposed compensate for this discrepancy, they usually require collection of long trajectories and induce additional problems such vanishing/cxploding gradients discarding many useful experiences, which eventually increases the computational complexity. Moreover, their generalization to either continuous action domains or policies approximated by deterministic deep neural networks is strictly limited. To overcome these limitations, we introduce novel policy similarity measure to mitigate the effects of such discrepancy in continuous control. Our method offers an adequate single-step off-policy correction that applicable to deterministic policy networks. Thcorctical and empirical studics demonstrate that it can achieve a "safc" off-policy learning and substantially improve the stale-of-the-arl by attaining higher returns in fewer steps than the competing methods through an effective schedule of the learning rate in Q-learning and policy optimization.
1 Introduction
In continuous action domains, deep reinforcement learning (RL) requires large amounts of data to develop optimal policies and scalable agents. Enhancing data and sampling efficiency often involves using buffer, known as the experience replay memory (Lin, 1992), to store and repeatedly use agents' experiences for training. The advantage of employing experience replay lies its off-policy nature, where the stored samples generated by various behavioral policies differ from the target policy being optimized. However, this off-policy learning approach comes with potential pitfalls. Despite agents' past experiences possibly aiding future learning stages, the process neglects the diversity of policies that generated the samples. This off-policy error or bias can lead to sudden policy divergence, especially when combined with function approximation and bootstrapping a problem emphasized by the deadly triad (Sutton & Barto, 2018, chap. 11). Addressing this off aff-policy bias commonly involves disregarding transitions produced by behavioral policies uncorrelated to the target policy's distribution. This known as off-policy correction (Harutyunyan et al., 2016), usually achieved through the importance sampling (IS) operator (Owen, 2013: Hesterberg, 2003). This operator assigns weights to each sample or trajectory within Lemporal-difference (TD) learning (Sutton, 1988; Precup

---

Published in Transactions on Machine Learning Research (01/2024)
et al., 2001) based on the likelihood of the actions generated by the target agent's policy. Consequently, this allows for the weighting of off-policy data when calculating gradients for approximated value functions and policies, thereby refining the overall learning process,
1.1 A Review of Prior Off-Policy Correction Methods
IS methods, when applied to TD learning, have demonstrated efficiency in RL, whether used for single transitions (Schmitt et al., 2020) or temporally correlated trajectories (Watkins & Dayan, 1992; Degris et al., 2012; Munos et al., 2016; Harutyunyan et al., 2016; Espeholt et al., 2018). Despite their effectiveness, several challenges are present. For instance, trajectory- as unnecessary trajectory terminations, high variance, biased estimates, and gradient instability (Munos et al., 2016; Yu et al., 2018). They can also increase the computational burden and risk of gradient explosion as the environment complexity scales (Fujimoto et al., 2019). Conversely, single-step methods, which were designed for and tested in discrete control, may produce inaccurate importance weights (Schmitt et al., 2020), neglecting policy optimization in weight derivation. Furthermore, they are ill-suited for deterministic policy networks as they typically assume each action has a probability to be selected (Saglam et al., 2022a), thereby leaving deterministic policy networks underexplored.
methods can over-treat samples, leading to issues such
In continuous action spaces, bootstrapped Q-learning algorithms, based on TD learning methods, are commonly used for learning value functions (Mnih et al., 2013; 2015). Off-policy corrections are typically achieved by leveraging off- off-policy policy gradient (PG) techniques (Gu et al., 2017; Zhang et al., 2019; Graves et al., 2023) or IS methods that weigh the PG (Humayoo & Cheng, 2019). Despite their utility, these techniques that were built on traditional on-policy deep PG methods often require modification for off-policy learning, implicitly prioritizing PG over off-policy data.
While the available correction techniques in contimuus control tackle the off- bias problem from policy optimization perspective, an alternative approach is to consider the off-policy data-following the practices in discrete action domains (Harutyunyan et al., 2016; Munos et al., 2016; Espcholt et al., 2018; Schmitt et al., 2020). This strategy mitigates the impact of off-policy data the learning process, and, furthermore, uses full-returns or collected trajectories to accomplish off- off-policy correction. The assumption here is that a stochastic policy is in place.
In this paper, we identify two main limilations of existing importance sampling methods for off-policy correction: their limited applicability to deterministic policy networks (as we formally show in the following sections) and tendency to undervalue the importance of the off-policy data. While these methods (i.c., IS in discrete control and off-po PG) have proven successful, they often focus too much the PG, which may overlook the significance of proposed approach aims to address them in Related Work.
data. We delve deeper into these limitations and discuss how our
1.2 AC-Off-POC: A Novel Approach
To address the identified limitations, we introduce the Actor-Critic Off-Policy Correction (AC-OF-POC) algorithm. AC-Off-POC offers a novel approach, shifting focus to the employment of previously collected off-policy data. This is achieved by adjusting their contributions to the training process through a state- independent weighting strategy. Specifically, we construct a multivariate Gaussian distribution based on the numerical deviations between actions chosen by the behavioral policies and the current policy. Each off-policy sample is then weighted according to the divergence from this Gaussian to a standard unit Gaussian. Our empirical results demonstrate that AC-Off-POC significantly enhances the performance of ofate-of-thheart models in a variety of tasks, outperforming existing off off-policy correction methods in challenging continuous control tasks from OpenAl Gym (Brockman et al., 2016). A comparative analysis of the improvements brought by AC-Of-POCC over previous studies is presented in Table 1. In the interest of reproducibility, we have made our source code available in our (currently anonymized) GitHub repository'.
2

---

Published in Transactions on Machine Learning Research (01/2024)
Table 1: Properties of the prior studies compared with AC-Off-POC, Note that properties are pertinent the techniques considered in this study excluding policy search methods.
Property
IS methods
Off-Policy PG methods
AC-Oft-POC
Imput type (single- multi-step)
multi-step
both
single-step
Applicable to continuous action spaces?
x
Applicable to deterministic policies?
x
x2
State-dependent?
X
x
Independent of the PG used?
X2
Hyperparameter-free?
x3
X
/
Except the study by Humayoo & Cheng (2019)
2 Since the PG is directly replaced
3 If variance is reduced
2 Related Work
Here, we examine the prior studies regarding either their approach or the target domain of operation. To this end, we thoroughly compare our results with respect the prior and detail our contributions.
2.1 Off-Policy Correction without Deep Function Approximation
The initial studies in importance sampling and off-policy correction have been done by Precup et al. (2000); Watkins (1989). They focused on eligibility traces and learning from delayed rewards, respectively. This evolved into more complex approaches such as weighted importance sampling with linear function approximators (Mahmood al., 2014), and multi-step expected return formulations employing Monte Carlo techniques for large state and action spaces (Degris et al., 2012). These developments further motivated variance reduction techniques such as the study of Precup al. (2000). Recently, Ek et al. (2023) derived an off-policy evaluation scheme under out-of-sample conditions, that is, their approach of policy using past obscrvational data, where large portion might be missing. This derivation looks out for practical cases in which reaccess to the training data is impossible due to critical or unethical concerns.
assess the performance
Our work differs from these traditional methods in three key aspects. Firstly, our focus on deep RL, while these methods are not developed or tested in the deep function approximation setting, which more challenging task due convergence issues. Secondly, our algorithm employs one-step off-policy correction individual transitions rather than trajectories, distinction from the works of Precup et al. (2000); Degris et al. (2012). This is significant since trajectory-basse methods may lead to unnecessary trajectory terminations, high variance, or biased action probability estimates (Schmitt et al., 2020; Yu et al., 2018). Furthermore, unlike Watkins (1989), we learn a policy from instantaneous, rather than delayed, rewards, which simplifies the learning process and enhances interpretability, efficiency, and real-time adaptation while reducing the dependence on precise reward discounting. Finally, we focus on a learning scheme instead of an evaluation framework that assesses the performance of the agent's policy, in contrast Ek et al. (2023).
2.2 Off-Policy Correction in Discrete Control Tasks
The integration of deep function approximators into RL has led to advancements in algorithms, such RETRACE (Munos et al., 2016), which employs full returns as in on-policy learning. However, this approach was found to introduce a zero-mean random variable at each step, resulting increased variance (Schmitt et al., 2020). To tackle this, the V-trace algorithm (Espcholt et al., 2018) was proposed for trading off the increased variance for biased return estimates. Later, LASER (Schmitt al., 2020) was developed, successfully eliminating V-trace's bias without. introducing unbounded variance.
3

---

Published in Transactions on Machine Learning Research (01/2024)
These advancements improved performance in traditional Atari games, yet many of them are not originally proposed nor evaluated in continuous action settings. Conversely, our method is specifically designed with continuous control considerations in mind, demonstrating its versatility. Moreover, our approach employs a one-step off-policy correction on individual transitions, unlike RETRACE and V-trace, which rely on full returns or trajectories. This approach enhances computational efficiency, avoids unnecessary trajectory terminations, and minimizes variance and biased action probability estimates that can occur when multi-step formulation is used (Schmitt et al., 2020; Yu et al., 2018).
2.3 Importance Sampling in Continuous Action Spaces
Importance sampling has been relatively underemployed in continuous action domains (Saglam et al., 2022a). Techniques, such as DISC (Han & Sung, 2019), address the bias issue of clipped IS weights by individually clipping each action dimension. Batch Prioritized Experience Replay via KL. Divergence (KLPER) (Cicek et al., 2021) prioritizes on-policy samples in actor-critic algorithms by manipulating the structure in sampling from the replay buffer. Lately, Ying et al. (2023) considered the bias in off-policy evaluation caused by the reuse of off-policy samples. The authors call this phenomenon Reuse Bias, and alleviate it using their algorithm, Bias- Regularzdd Importance Sampling (BIRIS). These techniques have produced impressive results in environments that were challenging due to the discrepancy caused by off-policy samples.
However, our study presents several distinct properties from the above IS methods. While the method by Han & Sung (2019) apply IS for on-policy learning to weight transitions or trajectories, our approach employs IS for off-policy learning to rectily off-policy bias. Also, the technique by Cicek et al. (2021) priorilizes on-policy samples as an experience replay sampling algorithm, yet our method is an off-policy correction scheme that cooperates with various experience replay sampling algorithms, as demonstrated by our empirical results. Finally, the BIRIS algorithm addresses the off-policy bias in policy evaluation instead of adjusting the contribution of off-policy samples optimization. Thus, our study differs from BIRIS in terms of the motivation and approach to the problems caused by off-policy learning.
2.4 Off-Policy Correction in Continuous Action Spaces
In PG-based off-policy learning, many methods have addressed the instability of the conventional PG techniques (Humayoo & Cheng, 2019; Graves et al., 2023; Gu et al., 2017; Zhang et al., 2019). In the field of offline RI, Bagatella et al. (2022) proposed an exploration framework in the sparse reward setting to overcome the discrepancy between the offline data that demonstrates task of interest. Although our work shares similarity with the motivation of the study of Bagatella et al. (2022), we focus on online learning instead of learning from a static dataset. Moreover, our approach centers around adjusting the impact of off-policy samples during the learning process, instcad of exploration. Lastly, we consider instant aneous rewards rather than a delayed one, unlike Bagatella et al. (2022).
excessively deviating task than the
Concerning online PG methods, which is the central scope of our work, the available methods use off- policy correction techniques for policy gradients. As an example, Relative Off-Policy Actor-Critic (RIS-Off- PAC) (Humayoo & Cheng, 2019) uses smoothed variant of relative IS with a parameter 3 E [0, 1] to govern the smoothness. Actor-Critic with Emphatic Weights (ACE) (Graves et al., 2023) applies an emphatic weighting scheme based on the study of Sutton et al. (2016), albeit with a misleading excursion objective. Generalized Off-Policy CCroti (Geoff-PAC) (Zhang et al., 2019) rectifies this issue by employing a counterfactual objective and an emphatic approach for the policy gradient. Moreover, Interpolated Policy Gradient (TPG) (Gu et al., 2017) interpolates on- and off-policy updates for actor-critic methods while satisfying performance bounds. These methods employ multi-step bootstrapped Q-learning and have demonstrated promising results. Thus, we benchmark our proposed approach against these methods in our empirical analysis.
These mentioned studies address the shortcomings of traditional PG methods when used in conjunction with ofl-policy learning, where it is assumed that the transitions used to compute PG are collected by the current policy, ie., on-policy. They employ various techniques, e.g., emphatic weighting (Sutton et al., 2016) or importance weights (Humayoo Cheng, 2019), to adapt the PG to off-policy learning. However, our approach deviates by focusing on mitigating the limitations of pollpo data, rather than modifying the policy gradient. The proposed technique employs traditional PG algorithms in correcting the off-policy
4

---

Published in Transactions on Machine Learning Research (01/2024)
samples, which are inherently on-policy. This aspect of our algorithm consolidates plug-and-play framework for various PG methods complementing the existing off-policy PG approaches.
3 Background and Notation
In this section, we delve into the technical underpinnings necessary to articulate the proposed algorithm. We commence with the preliminaries of deep RL, followed by our notation for the off-policy setting. Finally, we formally elucidate why many IS methods are not extendable to deterministic neural network policies.
3.1 Deep Reinforcement Learning
At each discrete time step I, the agent observes a state 8, chooses an action a, receives a reward r, and transitions to a next state s'. In fully observable environments, the RL problem is represented by a Markov decision process (MDP), a tuple (S,A, P,7) where S and A denotes the state and action spaces, respectively, P the transition dynamics such that s' P(s, a), and y the constant discount factor. The value is defined the discounted sum of rewards R2 Σi 7're+i, where 7 prioritizes the short-term rewards.
The goal of an agent is to find policy π that maximizes the expected return: EE aar,,, [Ro, where Pn is the distribution over the states visited by policy т. Policy of an agent is stochastic if it maps states to action probabilities π : S - p(A) or deterministic if it maps states to a single action क S - A. Note that we use the term "deterministic policy" to refer to the policics approximated by the deterministic noural networks throughout. the article.
In this study, we consider one-step bootstrapped Q-learning (Watkins & Dayan, 1992). There exists an action-value function (Q-function or critic) corresponding to each policy that represents the expected return while following the policy after laking action in state 8, which is computed through the Bellman operator T" (Bellman, 1957):
T= (s, a) = Pa + (Q"(s',
The Bellman operator is a contraction for 7 E [0, 1) with unique fixed point Q" a), i.e., the resulting error is bounded by y (Bertsekas & Tsitsiklis, 1996). The optimal action-value function, (s, a) max, Q (s,aa),, is obtained through the greedy actions of the corresponding policy, that is, actions that yield the highest action-value.
In the deep setting of RL, the action-value functions are modeled by deep neural networks (o a), parame- terized by 0, also known as Q-networks. The Q-network is trained by minimizing a loss (0) on the TD error 8, the difference between the network output Qo(s, a) and learning target y:
y = (s'(s'
8 = y Qa(s,a), J(0)
(107]
(1)
where 0' is the larget network parameters of the Q-network. A target network is practical approach that provides a fixed objective to optimize the network and ensure stability in the updates. They are usually updated by small proportion at each step, i.e., 0' -0 + (1 - -)00, called soft-update, or periodically to exactly match the behavioral networks called hard-update.
In continuous action spaces, finding the optimal action that maximizes the Q-function, i.c., max, Q" a), is not feasible due to the infinite number of actions. To overcome this, actor-critic methods use separate network, called the actor network (or policy network), that selects actions for the given states. The actor network ne, parameterized by Φ, is optimized by single-step gradient ascent on the policy gradient. The loss for the deterministic policies (der()) is based on directly maximizing the Q-value estimate of the Q-network:
Jact(O) = E.-P. [Qo(s, a)lla=ma(s))
(2)
To realize a differentiable loss Jato (Ø), stochastic policies employ the derivaiie trick (Williams, 1992) and scale the logarithm by the expected sum of discounted rewards while following the current policy, predicted
5

---

Published in Transactions on Machine Learning Research (01/2024)
by the Q-network:
(sto( = =E woollllggollo [o(als)Qo(s, a)laamas(1
(3)
3.2
Off-Policy Learning
The agents employ behaviooal policy to accumulate experiences within replay buffer. These experiences, however, do not fully represent the agent's current policy as they are collected through its older versions, thus termed ofF-policy samples. Conversely, experiences gathered through the current policy are labeled on-policy samples. During each update step, the agent selects a batch of transitions from the replay buffer employing a sampling algorithm, which may incorporate both on- and off-policy data:
(s, a, r,,, R,
(4)
where and are used to denote experience replay and the sampled batch, respectively. We consistently apply boldface to refer to the batch (matrix) of each entity, while non-boldface terms represent vectors (for states and actions) or scalar values (for TD error and rewards).
3.3 Importance Sampling in Continuous Control
The current IS methods that incorporate action probabilities to perform off-policy correction demonstrate strong theoretical guarantees and exhibit good performance in practical applications (Espeholt al., 2018; Schmitt et al., 2020; Munos al., 2016; Harutyunyan et al., 2016). However, these methods are not well-suited for off-policy actor-critic algorithms that rely deterministic neural networks to represent their policies. To illustrate, the RETRACE algorithm computes the importance weights of transitions using the following approach:
RETRACE($) Amin(1, x(als) u(aas))
(5)
where and μ are the current and behavioral policies, respectively, and À is a fixed parameter that specifies the maximum allowed distance between the behavioral and evaluation policy. When considering a deterministic network to represent the bchavioral and evaluation policies, as observed in studies by Fujimoto et al. (2018) and Lillicrap el al. (2016), it is important to note that the resulting importance weight will consistently yield value of 1. This is because both policies functions are many-to-one mappings, where states are deterministically mapped to unique actions with a probability of 1. Consequently, the existing studics on IS are not feasible to be applied to deterministic actor-critic methods, since no weighting would occur in the end.
4 Mitigating Off-Policy Bias: A Single-Step Approach
We aim to improve learning efficiency by minimizing off-policy bias arising from divergent behavioral policies. To achieve this, we introduce Actor-Critic Off-Policy Correction (AC-OIE-OOC), with variants for deterministic and stochastic neural network policies. The AC-Off-POC framework applies Jensen-Shannon divergence to correct the impact of off-policy samples Q-network and policy optimization, requiring no action probability estimates for deterministic policies. Off-policy bias can be remedied by weighting actor network optimization transitions with similarity weights. This approach respects the MDP assumption, which suggests that collected data originate from the actions generated by behavioral policies. Moreover, TD methods provide direct discrepancy correction between target and behavioral policies through cumulative reward objective-basdd value functions (Harutyunyan et. al., 2016), necessitating weighting in Q-network update transitions as well. To this end, we represent policy similarity on a scale from 0 (uncorrelated) to (identical). Next, we culminate in comprehensive framework fcaturing AC-Off-POC for actor-critic with one-step Q-lcarning, validated by theoretical derivations. We show that AC-Off-POC delivers safe off-policy correction via bounded racton mapping. Key insights highlighted and discussed throughout.

---

Published in Transactions on Machine Learning Research (01/2024)
4.1 Deterministic Policies
We delve inlo the AC-OI-POC for deterministic policies, addressing it in two parts. First, Section 4.1.1 provides a technical description of the algorithm. Subsequently, Section 4.1.2 discusses the rationale behind the design choices made in developing this algorithm.
4.1.1 Methodology
We first assume that the multi-dimensional continuous actions each step are samples from multivariate Gaussian distribution. Let À denote the similarity between the distributions of the bchavioral policies that collected the sampled off-policy transitions and the current agent's ' policy (larget or evaluation policy). Then, given a sampled batch of transitions as defined by Equation 4, we can express the deterministic policy loss weighted by the similarity coefficient as:
181
Ja (0)
Σ Qa(s,, a.).
(6)
We refer to the À term as similarity weights or off-policy coefficients throughout. If the majority of behavioral policies within the sampled batch differ significantly from the current policy, the gradient step on the policy and Q-network parameters will be minimal, as both À and loss are computed over the batch. Extending this and using Equation 6, we define the X-scaled TD learning loss as:
j(() x 1101/ B
To gauge the similarity between the current policy and the policies that enacted the off-policy transitions from sampled batch B, we initially forward pass the states from B through the actor network corresponding to the current policy:
â = (s).
We now have the batch of current policy's ' action decisions å on the states from the off-policy transitions, and
the batch of action decisions of past policies a E Let å be the batch of numerical differences in the action decisions:
á.
Recalling the multivariate Gaussian assumption on the actions, we then construct a multivariate Gaussian distribution from the action difference batch Wi, Σ) where
B
1
A
Σ
à;
(8)
B i=1
[8]
Σ
2
à; 11) (á. -A).
[
As empirical validation, the Gaussian interpolation outlined in the latter equation has been previously employed by Cicek et al. (2021), where it has been proven to enhance overall empirical performance. Notice that the computation of the mean across numerical action deviations, and eonsequently the covariance matrix, does not depend directly on state information. Rather, it is iaae--independent This independence arises because the weights are assigned based on actions resulting from the application of the policy to states within the off- off-policy data. This approach emphasizes the outcomes of actions rather than the states themselves. Next, define the dissimilarity measure as:
p JSD(N(A, ()),
where o the standard deviation of the exploration noise and I the identity matrix. In addition, JJD divergence, the symmetrized and smoothed version of the Kullback Leibler (KT.
denotes the
7

---

Published in Transactions on Machine Learning Research (01/2024)
divergence, defined by
M M= 1 (P KL(PIIM + (LIM)
JSD(PIIQ) 1
where P and Q are random variables.
Naturally, if all transitions in the sampled batch match the distribution under the current policy, we have / and p (0, xo) otherwise in Equation 9. To project the similarity measure into the interval [0, 1] for bounded weights, non-linear transformation can be applied:
A p
(10)
Notice that for two identical policies we have À = 1, and for completely distinct policios we have À = 0, making Equation 10 a similarity measure between two policies.
4.1.2 Methodological Justifications and Motivations
The Gaussian assumption. As the agent continually learns, each action chosen by different behavioral policy may originate from separate Gaussian distribution. Importantly, dimensions within an action vector are often correlated (Todorov et al., 2012; Parberry, 2013), reflecting real-world scenarios where the action space dimensions interact. Suppose a robot attempts to stand from a prone position. Here, each angle in the robot's 's n-dimensional joint-angle action space can influence others, underscoring cross-dimensional correlation. A multivariate Gaussian distribution effectively represents this action vector, with the mean vector denoting the deterministic action chosen by the policy, and the covariance matrix embodying the zero-mean Gaussian-distributee additive exploration noise (Williams, 1992), which is standard method for introducing exploration in leading deterministic PG algorithms (Hollenstein et al., 2022).
Using Jensen-Shannon divergence. There are various divergence options to choose from when quantifying the importance of transitions in Equation 9, For instance, one can employ the Rényi divergence (Rényi, 1961; van Erven & Harremos, 2014) due to its convenient expression for the moments of importance weights. However, choose to begin with KL divergence, derived from Shannon entropy (Shannon, 1948, chap. 6), which quantifies the information contained in probability distribution. From the RL perspective, the Shannon entropy of a policy distribution reflects the policy's uncertainty, as actions conditioned on observed states are sampled from this distribution. For instance, an optimal deterministic policy that always selects the same action has low environmental uncertainly (e.g., partial observability, adversarial behavior), reflected in low Shannon entropy. Therefore, the KT. divergence between the distributions of two policies indicates their differential environmental uncertainty. A large KL divergence implies that the policies are dissimilar, choosing different actions for each state, suggesting different levels of environmental uncertainty.
infornation-theorttiin intuition provides basis for employing KT. divergence as off-policy correction metric (Cicek et al., 2021). However, due to its harsh penalization of significantly differing distributions, a symmetric measure like Jensen-Shannon (JS) divergence becomes a better choice. JS divergence offers a smoothed version of KT, divergence and does not penalize distribution differences as severely. Thus, considcring the unlikelihood of two behavioral policies of an agent the same environment being entirely distinct (Sutton & Barto, 2018, chap. 11), JS divergence should be used to gauge the similarity between two behavioral policies during training.
Morcover, we note that rather than directly comparing the action difference with zero multivariate Gaussian in Equation 9, we compute JSD using a reference matrix of zero-mean Gaussian with a diagonal covariance matrix that contains the standard deviation of the exploration noise in the diagonal entries. Otherwise, policies closer to the current policy that exocuted a transition with exploration noise could be rejected as the action decisions would numerically deviate from the current policy. Hence, we couple the standard deviation of the exploration noise with the reference matrix to form parameter-free algorithm.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Employment of the exponential operator. The chosen exponential operator provides continuous link between unbounded similarity values to bounded similarity weights without requiring any quantization, and runs in constant-time, i.e. O(1). Furthermore, its practical runtime can be reduced using SIMD (Single Instruction Multiple Data) instructions that modern CPUs and compilers offer. SIMD instructions facilitate vector operations, such as computing ex for each element. x of given vector of dimension n. A naive implementation would require n exponentiation operations, leading to a runtime of O(n). However, SIMD instructions can optimize this operation by allowing the same operation to be executed on multiple data elements in parallel (Buyya et al., 2013; Cardoso et al., 2017; Marinescu, 2023). Thus, the exponential operator also provides the advantage of quick and efficient projection.
Finally, we chose not to include a coefficient to modulate decay (i.e., e-ap) to keep the algorithm parameter- free. Our empirical results indicate that using e already achieves satisfactory performance across various environments. However, it is possible introduce task-specific modulation coefficient to accelerate learning, if required.
Analysis of the impact of minibatch size. The deterministic variant of AC-Off-POC computes the numerical discrepancy between actions of the target and behavioral policies, essentially comparing the distributions under the current and in the minority of batch transitions executed by policies highly similar to the current one. The average-based similarity weight computation, as per Equation 8, may assign a near-zero fixed weight to these transitions, causing information loss. We address this issue when we introduce the stochastic variant. Additionally, we provide an intuitive analysis on the impact of minibatch size in Observation 1.
transition-executing policies. However, a possible concern lies
Observation 1. Let [B] and [R] are the fized minibatch and experience replay buffer sizes used in training. IB] + [R], then the mean indices of the sampled transitions will be 0.5 in the expectation, and determénistic AC-Off-POC weights become slightly less than 0.5 due to the non-linear transformation. If [B] + O, then the AC-OfJ-POC weights camnol be accuralely estimated. Therefore, very large or small minibalch sizes may result in poor performance.
4.2 Stochastic Policies
The application of AC-OfE-POC to stochastic policies is straightforward, given that the policies are represented by probability distributions. While these known probability distributions can be directly used traditional methods via eligibility traces, they often necessitate long trajectories executed by each policy, as demonstrated in prior work. Instead, stochastic AC-Off-POC offers an off-policy correction on randomly selected off-policy transitions.
In this variant, we can forgo assumptions on the specific distribution from which actions are sampled. Thus, we can store the parameters of the policy distributions in the replay buffer. For instance, if an agent's 's stochastic policy is represented by a Beta distribution B(a, 3), we can store the paramcters a and 33 in the experience replay buffer, sampling these alongside states, actions, and rewards during each update step. Hence, the agent samples batch of off-policy transitions along with the parameters that define the policy distribution:
a, a...s'. (1s)) R.
In this way, each behavioral policy TR(-) that executed the ith transition can be represented with the corresponding distribution parameters: (-). Instead of comparing the difference batch with a reference Gaussian, we directly measure the similarity between the distribution under the current policy (-)) and noo (-):
P: = JSD((T()]lmaa) (-)),
7 [ Si)
A = min
P.].
17. (a. )
In the latter equation, we clip the similarity weights to obtain a 7-contraction mapping around the optimal Q-value, which we examine in the proof of Theorem 3. We then construct a similarity weight vector for the sampled batch of transitions:
A
A1
d2
AB
9

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm Actor-Critic Off-Policy Correction (AC-Off-POC)
1: Input: Tá. B
2: Output: A
if ı is deterministic then
Obtain the minibatch of transitions: (s,
Compute the current policy's action decisions the sampled states: å â= To(ss) Compute the numerical action difference batch: à = a Construct the multivariate Gaussian distribution (i2 D):
7:
B
Σ à Ε
à.
-a) jb)
B
8:
Compute the similarity coefficient x: = JSD E)I o1) = x=
clsc
10:
Obtain the minibatch of transitions containing the policy distribution parameters:
(s, a, a.. .18))))))))))))))))))))))))).. B.
11:
Compute the similarity coefficients A= [11
AB]]
Pi JS)))(())) (-)) = X min Neo [x]] e ePi]
12: end
13: return A AVA
Similar to Equations 6 and 7, we can derive the weighted policy and critic loss for stochastic actor-critic:
1
(8)
A log no (a Qu a
(11)
|
j(0) = | 0112
(12)
|
where ||. represents the L, norm and o is the Hadamard product.
the stochastic variant each off-policy transition carries a unique weight, as it can be computed through the corresponding policy distribution. Hence, we compute the loss for actor and critic parameters using a weighted of the off-policy transitions, which contrasts the deterministic case. This approach circumvents the potential issue of information loss in the deterministic variant. Nevertheless, even in the deterministic case, such loss of information can be largely disregarded, as the sampled batch is generally expected to be similar to the current policy on average. With this understanding, we extend our discussions on deterministic policics to the stochastic variant of AC-Of-POCC Finally, we provide a pscudocode for AC-Off-POC in A Algorithm
4.3 AC-Off-POC for Actor-Critic with One-Step Bootstrapped Q-learning
Algorithm 2 elucidates general framework incorporating AC-Off-POC. Subsequently, we address the potential limitations of AC-Off-POC and its advantages over previous work. We then conduct an intuitive complexity analysis of our method. Lastly, our theoretical analysis, specifically Theorem 3, establishes that our off-policy correction method can generate a 7-contraction mapping around the optimal Q-value with the TD( algorithm (Sutton, 1988). This analysis leverages the tabular setting, routinely used in the deep RL literature offer thcoretical assurances for non-tabular and deep function approximation contexts (Munos et al., 2016; Schmitt et al., 2020; Espeholt et al., 2018; Fujimoto et al., 2018; 2019).
4.3.1 Potential Failure Cases
Mcan difference is zero. Even when the behavioral policies in a sampled batch are quite different from the current policy, the computed mean difference could be zero. To see this, suppose that the action range is [-10, 10), and the current. policy outputs action 0, where half of the sampled past actions are -10, and
10

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm 2 A General Framework of AC-Off-POC for Actor-Critic, with One-Step Bootstrapped Q-learning Off- fff-Policy
Initialize the agent with actor T and critic Qo networks with parameters o,
2: Initialize larget and additional networks if required
3: Initialize the experience replay buffer R
4: for each exploration time step do
5: Explore the environment and collect a transition tuple (s, (s,a,r,s))
if the policy stochastic then
Include the policy distribution parameters a to the collected tuple end
Store transition tuple into R
10: end for
11: for each training iteration do
12: Sample a batch of transitions B from R by a sampling algorithm
13: Obtain the similarity (s):
X AC-Of--PC((T), B)
14:
Update actor and critic networks with an actor-critic algorithm with ono-stop bootstrapped Q-lcarning using the losses weighted by À V X, e.g,, Equations 6, 7, 11, and 12
15:
Update target networks if required
16:
end for
the other half 10. The fitted normal distribution will have zero mean even though the policies differ considerably, In this particular example, the variance in the fitted normal distribution would mitigate this issue, that is, the empirical variance would be greater than the exploration noise added to the actions, and thus, the similarity weights will be small.
Different policies selecting the same action. Relatively different policies might select the same action for particular state. Yet, their probability distributions may not similar to each other. Then, we would give low importance weights to the corresponding tuple. However, optimization of stochastic policies gradients is not similar to the deterministic policies since the distribution-related parameters are included in the training. For example, in the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018), (log-) probabilities of the actions rather than only the selected actions and the entropy of the policy distributions are also included in calculating the policy gradient. Hence, the distributions themselves should also contribute to computing the similaritics. Thus, even if two policies choose the same actions, we give low similarity weights to the corresponding tuple unless their distributions are also similar.
4.3.2 Advantages Over the Prior Works
AC-Of-POC addresses the issues associated with trajectory-based IS, including high variance build-up and vanishing gradients. It avoids these issues by employing individual transitions that are not temporally correlated, with a maximum weight of one. Thus, the weights of the sampled transitions in single update do not influence the weights in subsequent updates. This makes AC-Off-POC a more favorable choice for off-policy corrections compared to standard IS methods.
Recall that off-policy weighting TD learning methods is crucial to adjust. the expected return under the assumption that off-policy data was generated by the current policy. The existing off-policy correction studies in continuous control focus off- policy PG. However, these approaches incur high computational complexity, especially with increasing state and action dimensions, they necessitate lengthy trajectory collection and task-specific hyperparameter tuning. Their adaptability to deterministic policies is also constrained, as they rely on action probabilities, infeasible with deterministic policy approximations via neural networks. An alternative off-policy correction method should also consider off-policy bias in relation to off-policy data, rather than policy gradients, and be compatible with deterministic policies. Our presented solution effectively and safely addresses off- off-policy correction, both theoretically and practically, as demonstrated through the following rigorous analyses and empirical studies.
11

---

Published in Transactions on Machine Learning Research (01/2024)
4.3.3 Complexity Analysis
Primarily, AC-Off-POC necessitates various arithmetic matrix operations and one forward pass through the actor work The forward pass includes multiple matrix multiplicatiins, linear inpul size, over- shadowing the complexity of other matrix operations within AC-Off-POC. Thus, the complexity here is O(m). Conventionally, a simple deep actor-critic algorithm employs two networks for the actor and critic. Backpropagation, similar to the forward pass, is linear in imput size, yielding the same complexity if trained over a single iteration. Consequently, each learning step per iteration demands ((m) + O(m + n)) for the actor and critic networks respectively. Given that the complexity brought by AC-Off-POC is O(m), it is inferred that AC-ONT-POC escalates the complexity of actor-critic algorithms by slightly less than 50% (at most).
4.3.4 Theoretical Analysis
In our analysis, we adopt the framework established by Munos et al. (2016). This approach enables us explain the requirement of clipping the importance weights in AC-OЛ-POC. Furthermore, by adapting the principles from Munos et al. (2016) to the context of single-step returns on randomly sampled batch of transitions, we demonstrate that our method functions as a 7-contraction mapping around the true Q-function Q". Such a property ensures that iterative application of the TD operator moves the estimated value function closer to the true value function, i.e., convergence to the true value function.
Definition 1. AC-Off-POC can be regarded a special case of the RETRACE algorithm (Munos et al, 2016). Therefore, the general operator for single-step off-policy correction by AC-Off-POC, which is the reduction of RETRACE to one-slep backup selling, is defined by
HQ(s, = E.
Q(s,a)),
for any behavioral policy n that collected the off-policy samples, ... where we define En]:] Es-P, ara PH] and (s,) = E. (ss)s)Q(s.a)) a), token over the dataset of interest containing the transitions.
A Assumption (coverage assumption). There a non-zero probability that the current agent's or evaluation policy executes an action with a non-zero probability of being executed under a behavioral policy:
π (als) > 0 = plals) > 0, A V.
(13)
Observation 2 (Implication of Assumption 1). Assumption 5 guarantees that the similarity weights fall within the range [0, a(gle')] This properly emerges from our unique compulation melhod, independent of direct action probabilities under the behavioral and target policies. This independence, coupled with the inherent upper bound on the simslaarity weights, ensures controlled variance of the IS eslimator even when the behavioral policy's 's action probability is small, thereby contributing to the robustness and efficiency of AC-Off-POC.
Lemma 2. The difference between HQ and its fixed point Q" is
HQ(s,a) (((ss, = BBsr (Q- (²) -
where X' is the similarity coefficient computed for the subsequent transition of (s, (s,
Proof. See Appendix A.
Theorem 3. If for each action selected by the current policy ai, ~ π and sampled batch of transitions By R, we have the similarity coefficient X; = A(ai,,, Bj) min [(a..))s.) ete]] for the transition tuple (s, r,);) E Bj collected by the behavioral policy ти- Then, considering that Assumplion 1 is satisfied, for any Q-function Q, we have H. being a ry r-contraction mapping around Q=:
=ss))= e
IIQQ - 1 S <111Q= loc.
where 1loo is the supremum norm.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Proof. The proof considers stochastic policies and reduction to deterministic policies is straightforward, we show. From Lemma 2, the difference between HQ and its fixed point Q" is expressed by
HQ(s, - Q" (s. = "Bx' Pa E, XXQQ(',
= =P E. AQ 1) Ea, N' -' (N 15) AQQ [8)
=TBo'P, BR ( (7)
AT) ,y),
9
where B' is the batch containing the subsequent transitions of (s, a, s'), ie., (s', a', s") Ji=l E B', and X A(a,, B") is the corresponding similarity coefficient. Additionally, to obtain non-negative probabilities in the latter equation, A(y, < "(gls")
x(vls" must be satisfied. This why we clip d A(a, B,) for the corresponding pair (st, 4z) in the stochastic variant. For deterministic policies, we can have either nyls'" or ryyls')= = This is because the target and behavioral policies are many-to-one functions, mapping each state with a single action that has a probability of 1. Morcover, since represents the action chosen by the larget deterministie policy; we always have π (yls') = 1. Therefore, the constraint for the first case is trivially satisfied for any À E [0, 1]. The constraint for the second case also holds, can be shown by substituting Tylls") with 0 and following the rest of the proof.
Having :tyls") n A(y, B²) 2 satisfied, we have
HQ(s, a) (s, Σ Wz,y AQ(
2,21
which is a linear combination &Q(x,)) weighted by non-negative coefficients Wayi
Wa.y == TB. B
nlvls'))(y (') = x}},
where I(() is the indicator function. The sum of those coefficients over and y
Σ Way 7B B,, B' Σ (ylls' - A(y,
A(a', B'']B"]
Bo,,B&RRT - B'']
Clearly, we have Ex., Way I y since A(a',, B*) < 1 for Va B'. Therefore, ((Q(saa)- (", a) a sub-convex
combination of AQ y) weighted by non-negative coefficients Wz.y which sum to at most y. Hence, the operator H. which reduces the RETRACE operator to one-step return-based off-policy algorithms with AC-OfE-OCC, is y-contraction mapping around Q=.
J
5
Experiments
We conduct experiments to evaluate the effectiveness of the proposed approach. In Scction 5.2, we present the simulation results for the comparative evaluation with different IS methods and off-policy PG techniques. We conduct experiments under different batch and replay memory sizes, experience replay sampling methods, and divergence measures in Section 5.3 as sensitivity analysis. In addition, weights produced by AC-Off-POC are visualized and discussed in Section 5.4. We use the MuJoCo (Todorov et al., 2012) and Box2D (Parberry, 2013) benchmarks interfaced by OpenAT Gym (Brockman al., 2016). For reproducibility, we did not modify the environment dynamics and reward functions. The computing infrastructure used produce the reported results is summarized in our repository!.
5.1 Experimental Setup and Implementation
We apply our method to three baseline algorithms: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016), Soft. Actor-Critic (SAC) (Haarnoja et al., 2018), and Twin Delayed DDPG (TD3) (Fujimoto
13

---

Published in Transactions on Machine Learning Research (01/2024)
et al., 2018). For comparative evaluation, consider each baseline with and without AC-Off-POC, the continuous IS method, RIS-Off-PAC, and off-policy PG techniques with multi-step bootstrapped Q-learning: ACE, Geoff-PAC, and IPG. In the sensitivity analysis, we use the experience replay sampling methods, Combined Experience Replay (CER) (Zhang & Sutton, 2017), Experience Replay Optimization (ERO) (Zha et al., 2019), and Prioritized Experience Replay (PER) (Schaul et al.; 2016). The stochastic variant. of AC-Of--PCC is applied to SAC, while DDPG and TD3 use deterministic policies.
The environmental parameters used are in accordance with the OpenAI Baselines3 Zoo (Raffin, 2020). Due to the initial exploration steps sampling actions from the environment's 's action space, policy distribution parameters remain inaccessible for stochastic AC-Off-POC. Therefore, we only use them once they become available after a given number of exploration time steps. In our DDPG implementation, we adhered closely to the parameters outlined in the originating paper. We added an initial 1000 frozen exploration time steps and amplified the batch size to 256 for a more substantial off-policy sample exposure during updates. To facilitate exploration and align with the reference Gaussian distribution, the exploration noise was replacod with a Gaussian with zero mean and standard deviation 0.1. The TD3 implementation uses the finetuned parameter setting found in the author's GitHub repository". The precise implementation details of the bascline algorithms can be accessed via our code
The multi-step PG methods, ACE, GeofE-PAC, and IPG are applied to the baselines by replacing the PG computation. The IS method RIS-Off-PACE is directly applied on top of the baseline algorithms. We follow the original papers to implement ACE, IPG, and RIS-OR-PAC. Our implementation of Geoff-PAC is based on the code from the authors' GitHub repository*. We follow the exact structure given in the original papers to implement CER and FRO. Additionally, the repository is used to implement PER.
The competing algorithms introduce hyperparameters that either control the bias-variance trade-off or the on- on-poiccyness of the updates. We use the tuned hyperparameters provided in the original articles. In particular, we use the bias-variance tradc-off value of c = 1 for IPG. As reported by Zhang et al. (2019), ACE is not sensitive for the bias parameter ê on OpenAl Gym benchmarks and \ =0 0 can produce sufficiently good results. Hence, we set A = 0 for ACE. For Geoff-PAC, however, we employ the tuned hyperparameters of A1 = 0.7, A2 0.6, and 9 = 0.2 since they are reported to result in decent empirical performance. Finally, use 0.2 for the smoothing control parameter 3 in RIS-Off-PAC.
For all simulations, each algorithm is run for 1 million time steps with evaluations occurring every 1000 time steps, where an evaluation of an agent records the average reward over ten episodes in a distinct evaluation environment without exploration noise and updates. We report the average evaluation return of ten random seeds for initializing networks, simulators, and dependencies. Uniform sampling and an experience replay buffer of size million are used in the comparative evaluations.
5.2 Comparative Evaluation
Evaluation results under the DDPG, SAC, and TD3 algorithms are reported in Figures 1, 2, and 3, respectively. As learning curves are intertwined and hard to follow for some of the environments, we also report the average of the last ten evaluation returns, iee. where the algorithms converge, Tables and 3. Note that for some of the tasks, the baselines have worse performance than reported in the original articles. This is due to the stochasticity of the simulators and the used random seeds. Nevertheless, the performance difference between competing methods would be consistent if we used different sets of random seeds, regardless of where the basclines converge.
Both variants of AC-OIE-POC demonstrate significant performance gains over the baseline algorithms and match or surpass the competing methods across all tested domains. This underscores the scalability of our method. In particularly challenging tasks, such as Ant, BipedalWalker, Humanoid, and Swimmer, where baselines often become trapped al local optima, the benefits of AC-OS-POC are especially evident. The complexities of these environments often originate from the high dimensional state-action spaces or the
http:///ggg..aaaaaaaaaaaaaaaaaaa Phttps: h/tg://aauub.cc/////iia////nn//a
5 haaat:///ggitabb...oo////t///aaa
14

---

Published in Transactions on Machine Learning Research (01/2024)
DDPG
DDPG + AC-OR-POC
DDPG + ACE DDPG + RIS-OIEPAC
- DDPG + Geoff-PAC
DDPG + IPG
3ipadalWalkarw33
HallCheetah-v2
Hopperu2
-40
6000
3000
Evalliaton Reware
11000
5000
2500
1000
-80
4000
1500
700
-120
2000
1000
800
1000
500
500
-160
0
0.6
0.0
0.6
0.0
0.2
0.8
1.0
Total time steps (x10%)
Total time steps (x10*)
Total time steps (x10%)
Total time steps (x10*)
Humanaldiv2
LumorLandetCondiuuuuee v2
Swimmar-v2
Walkcer2d-22
1600
250
3000
Evalustion Revtrrs
1400
40
1000
800
M W
-750
5500
-250
70
2000
1500
1000
1250
200
-1500
500
0.2
0.4
0.6
0.8
0.2
0.4
0.8
1.0
Total time stops (x10*)
Total time steps (x10'))
steps (x109)
Total steps (x10%)
Figure 1: Evaluation curves for the set of OpenAl Gym continuous control tasks for 1 million training time steps over 10 random seeds under the DDPG algorithm. The shaded region represents a 95% confidence interval over the trials. A sliding window of size 10 smoothes the curves for visual clarity.
internal simulation dynamics, which consequently intensify the detrimental effects of off-policy samples. Therefore, the improvements granted by effective off-policy correction techniques become more significant. We note a substantial improvement offered by AC-Off-POC in environments such as BipedalWalker, Hopper, Swimmer, and Walker2d. It has been observed by Henderson et al. (2018) that on-policy algorithms tend to significantly outperform off off-policy methods in these tasks. This implics that these environments may heavily rely on on-policy samples for stable and elficient learning. Therefore, our method of correcting off- off-policy transitions maximizes the improvement, especially in environments where off- off-policy algorithms often excel, such as Ant, HalfChoctah, and Humanoid. Despite this, we still notice enhanced performanco in these domains. This could be attributed to the effective elimination of highly divergent off-policy samples collected throughout the learning process, as the JSD) measure only assigns low scores to transitions reflecting very distinct behavioral policies. Although the transitions reflecting different behavioral policies may not be similar, they are still incorporated into Q-learning and policy loss. This insight underscores the discussions regarding the use of JSD and the crucial role of off-policy samples in solving certain environments.
While the enhancements in performance under the SAC and TD3 algorithms remain relatively consistent, a marginally greater improvement noticed for stochastic policies over deterministic ones when DDPG is included. However, the augmentation of deterministic policy performance becomes pronounced with the application of minibatch lcarning, as AC-Off-POC entirely discounts transitions within the sampled batch. These results suggests that more substantial improvement can be expected for stochastic policies. Still, if the distribution of sampled transitions aligns on average with the current policy distribution, the performance can still be enhanced.
Competing methods display significant stability in unstable environments such as Ant, Hopper, and Walker2d. However, the corresponding performance improvements fall short as they directly discard off ofl-cy samples through policy gradients or IS. Detailed examination reveals that Gcoff-PAC and IPG yield the best, comparable results among competitors, while RIS-Off-E PAC underperforms, and ACE ranks midway. RIS- Oft-PAC, using the action-value generated from the behavioral policy to train the algorithm, is typically
15

---

Published in Transactions on Machine Learning Research (01/2024)
SAC
SAC + AC-OI-POC
SAC + ACE
SAC + GeolE-PC
SAC + IPG
SAC + RIS-OI-PAC
BipadalWalkarva
12000
HolfCheetahwy2
Hopperu2
8000
4000
Evalliatoo Reware
5000
10000
4000
200
8000
3000
100
6000
2000
4000
1000
2000
1100
0.0
0.6
0.0
0.4
1.0
Total time steps (x109)
Total time steps (x10*)
Total time steps (M10*)
Total time steps (x10*)
Humanaldiv2
300
Lumertanderrooniiuuusvzz2
140
Swimmar-v2
6000
Walkcerdd-42
Evaligtion Revlard
5000
200
120
100
5000
100
3000
2000
2000
-100
20
1000
1000
-00
0
-20
0.0
0.6
0.2
0.4
0.6
0.8
0.2
0.4
1.0
time steps (x10')
Total steps (x109)
steps (x109)
Total steps (x10%)
Figure 2: Evaluation curves for the set of OpenAl Gym continuous control tasks for 1 million training time steps over 10 random seeds under the SAC algorithm. The shadod region represents a 95% confidence interval over the trials. A sliding window of size 10 smoothes the curves for visual clarily.
suboptimal during the learning process, which results in below-par baselines. The empirical performance of ACE is confined to simpler domains such as simple Markov chains or cart-pole balancing, as it was initially proposed for lincar function approximation. Geoff-PAC, its deep function approximation extension, produces better baseline performance, as expected. However, we posit that the PG estimation of Geoff-PAC. PACC despite its unbiased sample from the policy gradient, is still inaccurate and insufficient for optimal performance. Lastly, IPG results in an on-policy deterministic actor-critic method when = 1 is employed (Gu et al., 2017). Our earlier discussion proposes that complete on-policy PG can completely disregard off- off-policy samples. Nevertheless, off-policy samples can sometimes prove beneficial, depending on the task. Therefore, the maximal performance observed is not by IPG, but rather by AC-Off-POC.
Consequently, off-policy samples usually degrade the performance due to the underlying distribution that substantially diverges from the current agent's policy. Nonetheless, off-policy methods may still require off- off-policy samples to learn the environment (Cicek et al., 2021). AC-Off-POC solves this issue by employing the Jensen-Shannon divergence, which obtains smooth similarity measurement prior to the non-linear transformation. Its symmetric similarity measurement prevents the off-policy transitions from being heavily penalized and allows them to contribute to the learning progress even with a small proportion. Considoring that the hyperparameters introduced by the competing approaches increase the computational complexity and parameter-free AC-Off-PAC attains superior performance, we believe that our method provides significant. gains over the prior work.
5.3 Sensitivity Analysis
The batch and replay memory sizes and different experience replay sampling algorithms can considerably impact the learning. Therefore, we perform experiments on both variants of PC-Of-POC under different experience replay buffer and minibatch sizes and sampling algorithms. We evaluate each bascline and AC-Off-POC with minibatch sizes of 32, 256, 1024, and 2048 and the replay memory of sizes 1 million (1M) and 100,000 (100K). We also investigate the performance improvement by AC-ON-POC under the sampling
16

---

Published in Transactions on Machine Learning Research (01/2024)
TD3
TD3 + AC-OI-POC
TD3 + ACE
TD3 + Geolt-PAC
TD3 + IPG
TD3 + RIS-OI-PAC
3ipadalWalkarva
HollCheetahwy2
4000
Hopper42
12000
300
3500
Evalusaion Reware
10000
3000
4000
8000
2500
100
0000
2000
0
4000
1500
1000
1000
500
-100
0
0.6
0.0
0.6
0.4
Total time steps (x109)
Total time steps (x10*)
Total time steps (H10*)
Total time steps (x10*)
Humanaidiv2
Lumertanderrooniiuuusszz2
Swimmar-v2
Wallcer2d-42
300
Evaluction Reviard
5000
5000
200
4000
4000
100
3000
0
2000
1100
1000
-200
0
-20
0.0
0.6
0.2
0.4
0.6
0.8
0.2
0.4
1.0
time steps (x10')
Total steps (x109)
steps (x109)
Total steps (x10%)
Figure 3: Evaluation curves for the set of OpenAl Gym continuous control tasks for 1 million training time steps over 10 random seeds under the TD3 algorithm. The shaded region represents a 95% confidence interval over the trials. A sliding window of size 10 smoothes the curves for visual clarily.
algorithms of CER, ERO, and PER. The same experimental setting given in Section 5.1 is used. Unless otherwise stated, minibatch size of 256, replay memory of size 1 million transitions, and uniform sampling are used as the default setting.
5.3.1 Impact of the Minibatch Size
Table demonstrates the resulting performances for different batch sizes. Firstly, all methods show optimal performance with a batch size of 256, both in terms of mean and confidence, as they were originally tuned (Haarnoja et al., 2018; Fujimoto et al., 2018). When applying stochastic AC-Off-POC to the SAC algorithm, it is observed that the relative performance improvement brought by AC-Off-POC remains consistent across different batch sizes. This can be attributed to the transition-wise similarity measurement unique to the stochastic variant, meaning that when implemented with stochastic policies, AC-Off-POC is not influenced by minibatch learning and size.
However, the batch-wise similarity measurement in the deterministic variant for TD3 is considerably affected by the minibatch size. For all environments, an increase batch size negatively impacts the performance of AC-Off-POC and falls short of the baseline performance. Similar performance degradation occurs for smaller batch sizes due to the inaccurate estimation of the JSD between two Gaussians with insufficient number of samples. In summary, our batch size sensitivity analysis empirically verifies Observation 1.
5.3.2 Impact of the Experience Replay Buffer Size
Table shows the effect of varying replay memory sizes, implying substantial shifts in performance in terms of both mean rewards and confidence. In the HalfChoetah environment, decrcasing the replay memory size significantly impairs the performance across all methods. This can be attributed the environment dynamics of HallCheetah, which typically necessitates off-policy samples for efficient solution. We consider a First-In,
17

---

Published in Transactions on Machine Learning Research (01/2024)
First-Out (FIFO) buffer in all experiments, meaning that smaller replay memory will contain more on-policy samples compared to buffer of size 1 million transitions.
In Hopper and Walker2d, a smaller memory size leads to better performance, supporting the evaluation results from Section 5.2. This suggests that off-policy correction in these environments produces more substantial performance enhancement compared to HalfCheetah, as the policy trained more on-policy manner. Similarly, find that the deterministic variant of AC-Off-POC with smaller buffer underperforms the baseline in HalfCheetah, but markedly improves the deterministic bascline algorithm in the Hopper and Walker2d environments.
5.3.3 Impact of Different Experience Replay Sampling Algorithms
Table 6 reports the resulting performances across different sampling algorithms. We first observe that CER is superior among all the sampling methods. This is due to the most recent collected transition, which is included in each update. Hence, all updates always occur with at least a single on-policy sample. A performance improvement can still be obtained when AC-Of-POCC applicd to the basclines while sampling is performed through CER. However, AC-Off-POC slightly underperforms the baselines the HalfCheetah environment under CER because of the discussed off-policy sample requirement. ERO performs poorly when applied to the baseline algorithms due to the internal structure of the algorithm, which is not in the scope of this study. Nevertheless, AC-Off-POC still improves ERO when applied to the baseline algorithms. In addition, PER usually underperforms when applied to actor-critic method. This is also expected result. due to the widely known suboptimal empirical performance of PER (Saglam et al., 2022b). Although ERO and PER exhibit poor performance due to the discussed drawbacks, AC-OS-POC can still atlain higher rewards. Therefore, we infer that AC-Of-POC can readily adapt to various experience replay sampling algorithms.
Finally, the results obtained under different. experience replay sampling schemes support our motivation behind concentrating on one-step returns. Specifically, the recent advances in experience replay research use
Table 2: Average return of last 10 evalnations over 10 trials of million time steps for Ant, Bipedal Walker, HalfChectah, and Hopper. = captures a 95% confidence interval over the trials. Bold values represent the maximum under each baseline algorithum and environment.
Method
Ant
Bipeeal
HalfCheetah
Hopper
DDPG
816.99 + 165.29
1139.73 = 36.10
3798.16 + 1586.74
2137.20 + 537 67
DDPG + AC-OF-POC
1052.28 + 239.11
-104.79 = 38.39
5012.32 + 2155.64
2335.00 + 765.72
DDPG + ACE
829.96 133.42
-122.38 33.63
4043.05 1 1672.13
1742.74 1 488.32
DDPG + Geoff-PAC
784.67 + 133.77
-125.71 + 36.54
4660.93 + 1951.33
1767.06 + 676.08
DDPG + IPG
833.51 + 88.32
1121991 = 39.11
4248.09 + 1796.15
1905.45 + 731.71
DDPG + RIS-OR-PAC
837.10 + 107.21
-10..97 + 31.83
4171.61 + 1753.42
1819.95 + 450.00
SAC
4864.24 296.94
285.37 + 50.74
9871.42 + 273.98
3351.43 + 317.07
SAC + AC-OR-POC
5983.03 + 341.04
343.96 + 6.93
11320.82 + 262.23
3846.05 +: 777.89
SAC + ACE
5019.17 549.69
307.18 + 17.98
9375.53 + 410.10
3367.48 + 313.79
SAC + Geoff-PAC
5281.92 471.66
297.24 29.89
10604.40 173.26
2636.44 + 791.47
SAC + IPG
5173.16 274.65
318.80 9.41
9993.52 + 399.77
2906.38 + 609.73
SAC | RIS-OI-PAC
4613.55 1 430.51
313.67 1 12.80
10267.73 1 325.33
3036.12 1 534.42
TD3
4846.66 + 418.23
271.77 + 79.60
10050.95 + 168.03
3174.56 1 401.05
TD3 + AC-O#-POC
5472.19 276.49
322.38 6.92
11735.35 + 196.97
3658.85
354.52
TD3 + ACE
5064.31 376.07
306.96 + 15.10
10543.87 148.98
3215.57
379.31
TD3 + Geoff-PAC
5261.21 + 416.70
319.52 + 15.19
11467.90 + 148.70
3041.78
516.34
TD3 IPG
5332.47 367.49
295.09 44.95
11064.32 143.90
3524.52 1 247.26
TD3 + RIS-OfF-PCC
5045.43 + 488.06
307.22 22 + 16.90
10811.51 + 166.86
3532.53 + 128.64
18

---

Published in Transactions on Machine Learning Research (01/2024)
Table Average return of last 10 evaluations over 10 trials of 1 million time steps for Humanoid, LunarLan- derContinnous, Swimmer, and Walker2d. + captures 95% confidence interval over the trials. Bold values represent the maximum under each baseline algorithm and environment.
Method
Humanoid
LunarLanderCCntinuuuss
Swimmer
Walker2d
DDPG
206.80 105.63
26.69 73.32
19.71 5.90
1270.86 + 332.23
DDPG + AC-OI-POC
1115.81 + 641.76
173.02 + 85.59
28.95 = 7.88
2854.85 + 555.45
DDPG + ACE
204.14 = 100.09
161.61 + 58.27
20.82 = 12.27
1742.84 + 414.89
DDPG + Geoff-PAC
255.75 115.61
210.53 70.60
27.80 5.45
2030.25 + 625.53
DDPG + TPG
208. 46 = 84.78
207.11 + 69.60
24.55 + 5.47
1601.75 + 516.69
DDPG + RIS-OREPAC
198.50 = 81.86
196.59 + 66.94
21.93 6.86
164148 + 162.71
SAC
2689.22 1499.29
271.37 +
53.11 4.68
4103.54 805.03
SAC + AC-OI-POC
5098.33 + 529.73
284.83 + 7.09
130.29 = 19.46
5366.81 + 819.00
SAC ACE
45455663 415.48
277.18 + 7.91
60.89 5.46
3239.37 + 1193.46
SAC + Geoff-PAC
4897.81 = 508.99
279.75 + 9.46
96.24 = 14.80
4367.23 + 1217.94
SAC IPG
4768.96 457.70
276.29 + 11.56
84.20 + 16.11
3456.34 + 1486.13
SAC + RIS-Of-PAC
4646.68 = 459.60
276.06 + 11.19
61.90 + 9.43
4032.21 + 976.43
TD3
4929.35 = 331.30
277.36 +:
52.18 4.45
4107.08 349.81
TD3 + AC-Of-POCC
5297.80 + 308.16
308.05 = 15.24
114.02 = 21.51
5219.89 + 583.36
TD3 ACE
5115.29 219.01
279.49 + 5.27
57.18 5.30
4179.44 267.09
TD3 + Geoff-PAC
4805.35 397.55
268.35 1 6.36
80.36 + 16.91
4795.22 + 232.31
TD3 IPG
4908.21
482.69
280.46 1 5.32
59.94 8.01
4751.66 + 153.17
TD3 + RIS-OI--PAC
5015.17 = 340.12
278.38 + 6.68
56.18 + 5.13
442131 + 194.74
Table 4: Average return over the last 10 evaluations over 10 trials million time steps, comparing the
impact of minibatch sizes (32, 256, 1024, 2048). captures a 95% confidence interval over the trials. Bold values represent. the maximum under each baseline algorithm and environment.
Setting
HalfCheetah
Hopper
Walker2d
SAC (32)
9092.21 312.47
3304.77 + 348.16
3707.69 + 844.41
SAC (256)
9871.42 273.98
3351.43 + 317.07
4103.54 = 805.03
SAC (1024)
8766.95 + 290.90
2700.17 + 342.32
3848.68 = 927.21
SAC (2048)
8477.68 + 322.43
2930.11 t 324.41
3222.36 + 930.57
SAC AC-Of-POCC (32)
11022.58 + 276.83
3470.82 + 777.71
5135.93 + 860.30
SAC + AC-OR-POC (256)
11320.82 + 262.23
3846.05 +\t 777.89
5366.81 + 819.00
SAC + AC-OF-POC (1024)
10459.58 = 284.13
3269.79 = 833.98
5242.90 = 823.80
SAC AC-OfI-POC (2048)
9215.60 303.78
3558.28 893.85
4300.63 + 918.97
TD3 (32)
8955.77 + 181.85
2961.18 + 434.44
3966.54 +: 377.57
TD3 (256)
10050.95 + 168.03
3174.56 + 401.05
4107.08 = 349.81
TD3 (1024)
9963.48 + 185.11
2847.66 + 408.16
3881.20 + 351.81
TD3 (2048)
8712.29 + 194.69
2630.64 + 455.97
4033.33 + 351.51
TD3 + AC-Of-PPCC (32)
9104.90 249.96
8112.20 + 411.33
4951.56 = 698.94
TD3 + AC-OffPPOC (256)
11735.35 196.97
3658.85 + 354.52
5219.89 + 583.36
TD3 AC-Of-POC (1024)
8791.17 252.00
3126.42 + 426.60
5040.96 - 607.94
TD3 AC-OЛ-PPCC (2048)
6982.50 + 248.80
2176.53 + 469.73
4713.68 + 730.17
temporally uncorrelated transitions sampled batches. Although our algorithm can be generalized to the multi-step returns (i.c., trajectories), we focus on one-step algorithms for our method be compatible with the current experience replay sampling techniques and the ones that will be introduced in the future.
19

---

Published in Transactions on Machine Learning Research (01/2024)
Table 5: Average return over the last 10 evaluations over 10 trials of million time steps, comparing the impact of replay buffer sizes (100000, 1000000}. = captures 95% confidence interval over the trials. Bold values represent the maximum under each baseline algorithm and environment.
Setting
HalfCheetah
Hopper
Walker2d
SAC (100K)
8596.65
326.68
3446.68 + 368.41
4117.61 t 937.39
SAC (1M)
9871.42
273.98
3351.43 = 317.07
4103.54 + 805.03
SAC + AC-OR-POC (100K)
8290.28
309.43
3959.37 + 744.74
5471.21
+ 788.24
SAC + AC-OR--POC (IM)
11320.82 + 262.23
3846.05 + 777.89
5366.81
+ 819.00
TD3 (100K)
9343.67 = 188.52
3200.30 + 403.49
4183.48
350.12
TD3 (1M)
10050.95 168.03
3174.56 + 401.05
4107.08
349.81
TD3 AC-ON-POC (100K)
9068.78 255.80
3790.57 + 494.83
5302.83
670.45
TD3 + AC-OftPOC (IM)
11735.35 = 196.97
3658.85 + 354.52
5219.89 + 583.36
Table 6: Average return over the last 10 evaluations over 10 trials of million time steps, comparing the impact of uniform, CER, ERO, and PER sampling methods. + captures a 95% confidence interval over the trials. Bold values represent the maximum under each bascline algorithm and environment.
Setting
HalfCheetah
Hopper
Walker2d
SAC (uniform)
9871.42 273.98
3351.43 317.07
4103.54 + 805.03
SAC (CER)
10539.72 +:2 285.12
3442.93 311.60
4105.39 + 712.35
SAC (ERO)
9497.97 284.17
3393.43
315.16
4033.84 843.72
SAC (PER)
8953.30 + 296.70
2891.34
317.16
3844.24 + 832.49
SAC + AC-Off-POC (uniform)
11320.82 = 262.23
3846.05 777.89
5366.81 = 819.00
SAC + AC-OR-POC (CER)
9642.60 + 209.10
3905.79 + 760.39
5449.23 + 802.84
SAC + AC-OfF-POC (ERO)
11006.71 + 272.90
3865.37 + 765.77
5409.00 = 827.40
SAC + AC-Of-POCC (PER)
10388.39 + 297.94
3493.84
840.43
4402.69 = 963.22
TD3 (uniform)
10050.95 + 168.03
3174.56
401.05
4107.08 349.81
TD3 (CER)
10628.20 +'2 173.55
3190.81
400.69
4151.20
342.95
TD3 (ERO)
9591.56 = 168.77
3187.00
403.67
4013.53 = 361.32
TD3 (PER)
9578.80 + 172.61
2637.58 + 407.95
3447.57 + 350.60
TD3 + AC-Of-POCC (uniform)
11735.35 + 196.97
3658.85 = 354.52
5219.89 + 583.36
TD3 + AC-ON-POC (CER)
9851.44 + 177.24
3692.22 + 335.98
5367.62 + 486.86
TD3 + AC-OF-POC (ERO)
11345.45 + 205.72
3664.95
352.06
5039.16 = 589.69
TD3 + AC-OR-POC (PER)
10648.33 + 231.66
3467.8) 415.79
4357.27 622.60
5.3.4 Impact of the Divergence Measure
We test the resulting performance of AC-Off-POC when JSD is replaced with KL divergence. These results are provided in Table 7. We obtain higher results with JSD in all of the environments. However, for the HalfChootah environment, KL divergence significantly degrades the performance. As it severely penalizes transitions under which the policy distribution differs from the policy of interest, most olf-policy transitions are not included in the gradient computation due to the corresponding small weights. As discussed, the HalfChectah environment usually requires those off-policy transitions, and since the agent does not effectively use them, the performance drops. Overall, these empirical studies also verify our discussions on the advantages of JSD over KT divergence.
20

---

Published in Transactions on Machine Learning Research (01/2024)
Table 7: Average return over the last 10 evaluations over 10 trials of million time steps, comparing the measures of the Jensen-Shannon divergence and KI. divergence. + captures a 95% confidence interval over the trials. Bold values represent the maximum under each baseline algorithm and environment.
Setting
HalfCheetah
Hopper
Walker2d
SAC AC-OS--POC (KL)
10747.20 + 219.16 11320.82 = 262.23
3567.88 + 373.95
4980.37 + 620.16 5366.81 = 819.00
SAC + AC-OftPOCC (JSD)
3846.05 = 777.89
TD3 + AC-Of--POC (KL)
10112.36 + 200.21
3449.99 + 362.96
4915.94 + 612.65
TD3 + AC-Off-POC (JSD)
11735.35 = 196.97
3658.85 = 354.52
5219.89 + 583.36
5.4 Weight Analysis
To validate the accuracy of the weights produced by deterministic AC-Oft-POC, we conduct experiments employing the experiences of an actor trained under the TD3 algorithm in the HalfCheetah and Hopper tasks, over million time steps. During the training phase, the transitions executed by the agent were stored in temporal order. Then, we start by sampling the transitions with window size equivalent to the minibatch size centered each transition. For each sampled batch, the similarity weight was computed with the deterministic variant of AC-OT-PPOC with respect to the trained agent's 's policy.
Figure 4(a) presents the normalized values for time steps and similarity weights produced by AC-Off-POC. From the first transition executed by random policy to the last transition executed by the expert agent, the sampled batches increasingly become on-policy in relation to the expert agent's policy. It can be observed that as the center of the sliding window nears the last transition, the AC-Off-POC weights approach the value 1. Therefore, the weights rise as the on-policyness of the sampled batch augments, which is observed from the linear trajectory followed by the (C-Off-POC weights. Overall, it can be inferred that the deterministic variant of AC-Oft-POC yields accurate estimates with negligible error. However, it does not employ temporally correlated trajectories or any action probability estimate.
We perform the same set of experiments to verify the accuracy of the stochastic variant of AC-Of-PPCC. As the stochastic variant assigns unique weights to each off-policy sample, we directly depict the weights assigned to each transition contained in the expert agent's 's experience replay buffer. In Figure 4(b), we observe that the stochastic variant also produces accurate similarity weights to schedule the contribution of each loss
Time step of the sliding window center AC-Off-POC weights produced by the trained agent's 's policy
Hopper-v2
Hopper-v2
HaliCheetah v2
HalfChestah
1.0
g 1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.3
1.0
0.0
0.2
0.8 1.0
Time steps (le6)
Time steps (1e6)
Time steps (1e6)
Time steps (1e6)
(a) Deterministic (TD3)
(b) Stochastic (SAC)
Figure4 4: Temporally ordered batches of transitions versus AC-Off-OCC weights produced by the trained agent's policy under the TD3 and SAC algorithms for the set of OpenAl Gym continnous control tasks over million training time steps. The shaded region represents 95% confidence interval of the normalized values over the trials.
21

---

Published in Transactions on Machine Learning Research (01/2024)
component with respect to off-policy samples. Moreover, the confidence of the weights is more robust, and weights are more accurate due to the direct use of the policy distributions in computing the dissimilarity between the target and behavioral policies. In conclusion, as discussed, AC-Off-POC can produce accurate similarity weights by proventing vanishing or exploding gradients and the employment of trajectories due to the transition-we wise or Lemporally uncorrelated batch-wise similarities.
6 Future Work
Our technique also has some limitations that open up interesting directions for future work. One limitation is that the choice of Jensen-Shannon divergence can be restrictive as it requires choosing parametric distributions over the action set and storing the distribution parameters along with the (s, a, s') tuple in the buffer. This contrasts with the importance weights based on propensity ratio that do not enforce any parametric assumption on the distribution and do not require storing any distribution parameters. A possible extension of our tochnique is explore other divergence measures that can relax the parametric assumption and reducc the storage requirement.
Another limitation is that our technique relies on sampling actions from the target policy to estimate the divergence term. This can introduce variance and bias in the estimation, especially when the target policy is very different from the behavioral policy. A possible improvement of our technique is to leverage the entire probability distribution of the target policy instead of sampling actions from it. This can allow us to employ a policy search algorithm to directly learn a policy for solving the underlying MDP of the environment by reducing the variance substantially through aulti-importance sampling. Alternatively, if we want to maintain value function and use its values to learn policy for given MDP; i.e,, actor-critic methods, then our approach remains credible off-policy correction technique that also corrects the expected return estimated by the Q-network. We hope that our work will inspire further research off-policy correction techniques for deep Q-learning variants and their applications to various RL problems.
7 Conclusion
We discuss off-policy correction, which aims mitigate the detrimental effects induced by the mismatch between the underlying distributions of the agent's policy and the previously collected data. While off-policy correction is usually performed through off-policy policy gradients (PG) in actor-critic methods, we address that they cannot apply to the policies approximated by the deterministic neural networks since they require action probabilities. Moreover, previous studies usually approach the off-policy correction from the PG side, that is, they adapt the conventional PG techniques that are on-policy by construction to off-policy learning. To this end, we introduce the AC-Off-POC algorithm as a complementary alternative to the prior works. In contrast to off-policy policy gradients, our method approaches the problem from the off-policy data side without modifying the policy gradients used. AC-Off-PÓC can achieve an efficient one-step off-policy correction and improve the data efficiency by reweighting the contribution of each off-policy sample in optimizing the value function and policy. We support our claims with theoretical analysis to show that it obtains a bounded contraction mapping that offers safe single-step off-policy correction.
An extensive set of empirical studies demonstrates that AC-Off-OOC improves the state-of-the-art and outperforms the competing off- off-policy correction methods by a considerable margin. By alleviating the bias induced by the off-policy data, our method can attain faster convergence and optimal policies by disregarding the transitions executed by behavioral policies that highly deviate from the current policy in terms of the numerical action decisions. Our method also resolves computational cost issues by not introducing hyperparameters additional networks. Moreover, we show that a generic approach can readily apply AC-Off-POC to actor-critic method with one-step bootstrapped Q-learning. Lastly, we provide an open-source repository containing all the code and results to further support research on off-policy deep RI.
References
Marco Bagatella, Sammy Joe Christen, and Otmar Hilliges. SFP:
State-free priors for exploration in off- policy reinforcement learning. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL
22

---

Published in Transactions on Machine Learning Research (01/2024)
https::/openreeview http://wwww......oo.oooooottttaa
Richard Bellman. Dynamic Programmingg. Dover Publications, 1957. ISBN 97804864280993.
D. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, Belmont, MA, 1996.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/
2016. CRL attp://arxiv. wwt.:/ggg//aeo.o.eeeo http:/abss160...000 01540.
Rajkumar Buyya, Christian Vecchiola, and Thamarai Selvi. Chapter -
principles of parallel and dist ristributed computing. Rajkumar Buyya, Christian Vecchiola, and Thamarai Selvi (eds.), Mastering Cloud Computing, pp. 29 70. Morgan Kaufmann, Boston, 2013. ISBN 978-0-12-411454-5, doi: tttps://doi.org 1-11:///97----2-411------------- 8.00000. URL 1B770024114558000022
João M.P. Cardoso, José Gabriel F. Coutinho, and Pedro C. Diniz. Chapter 2 high-performance embedded computing. In João M.P. Cardoso, José Gabriel F. Coutinho, and Pedro C. Diniz (eds.), Embedded Computing for High Performance, pp. 17-56. Morgan Kaufmann, Boston, 2017. ISBN 978-0-12-804189- 5. doi: science/
12000000000220001055000000000000 URL https::/www..eiieneedireee wwt.:.sccieneedrect.coom
Dogan C. Cicek, Enes Duran, Baturay Saglam, Furkan B. Mutlu, and Suleyman S. Kozat. Off-policy
correction for deep deterministic policy gradient algorithms via batch prioritized experience replay. In 2021 TEFE 33rd International Conference on Tools with Artificial Intelligence (ICTAT), pp. 1255 2555-1262, 2021. doi: D0.108//CTAA15222522210.000
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp. 179 186, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
Sofia Ek, Dave Zachariah, Fredrik D. Johansson, and Peter Stoica. Off-policy evaluation with out-of- sample guarantees. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: //openreview.cc/////////// //opeeneeriss.nnt//fooorm/r//e www.:e/orrriiiidtttteeeeeee
Expert Certification.
Lasse Espcholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukeuoglu. IMPALA: Scalable distributed deep-RL. with importance weighted actor-learner architectures. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416. PMLR, 10-15 Jul 2018. URL attps://proceett espeholt188 html.
mlr, press/v80
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018. URL
wtw.pppres//v//fffee wtto/////ffiffoiieeeee html.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 96th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-13 Jun 2019. URL stttps://proceedingg wwrhgggsmmr...m www..p..es.eee...ee http/////////jjj///// /ttf///jjioiaaaaaaaaaa html.
Eric Graves, Ehsan Imani, Raksha Kumaraswamy, and Martha White. Off-policy actor-critic with emphatic weightings. Journal of Machine Learning Research, 24(146):1-66 63, 2023. URL httm://jmlr org/papers/ v24/21-1350.ne html.
Shixiang (Shanc) Gu, Timothy Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on-policy and off- policy gradient estimation for deep reinforcement learning. In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus,
23

---

S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https://proceedinng. a277111f281122677711..1111600002 Published in Transactions on Machine Learning Research (01/2024)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861 1870. PMLR, 10 15 Jul 2018. URL https: https://proot /tppa//////occogps.e/ mlr www..ppres///e... press/v80/ haarnojala aww.b.a1.b...... html.
Seunguul Han and Youngchul Sung. Dimension wise importance sampling weight clipping for sample-efficient reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2586 2595. PMLR, 09 15 Jun 2019. URL https://proceeddngg/ mlr. prwss//97/haa html.
Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. Q(A) with off-policy corrections. ALT, pp. 305 320, 2016. URL https://doi. http:://ab...rrg/1..000///////// 00001100000//11-3----6---7-----
Peter Henderson, Riashat Islam, Philip Bachman, Joolle Pincau, Doina Procup, and David Meger. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. URL https://ojs aaai. aaai..rg//inii http:///p//..//attarr///////////
Timothy Hesterberg. Advances in importance sampling. 01 2003.
Jakob Hollenstein, Sayantan Auddy, Matteo Saveriano, Erwan Renaudo, and Justus Piater. Action noise in off-policy deep reinforcement learning: Impact on exploration and performance. Thansactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: net htt.://nteeoooommrra/--111111111 Survey Certification.
Mahammad Humayoo and Xueqi Cheng. Relative importance sampling for off-policy actor-critic in deep reinforcement learning, 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016. URL http: arxiv wwtp:/o////////aeeeeeeee httss//109.000 02971.
Long-Ji Lin. Self- laffiimproving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3):293-33 321, May 1992. ISSN 1573-0565. doi: 10.1007/BF009996699 URL https://don //dp:/or//////////////o 10. 10.1007/BF009926990
A. Mahmood, H. V. Hasselt, and R. Sutton. Weighted importance sampling for off-policy learning with linear function approximation. In NIPS, 2014.
Dan C. Marinescu. Chapter 3 parallel processing and distributed computing. In Dan C. Marinescu (ed.), Cloud Computing (Third Editionn) pp. 41 41-94. Morgan Kaufmann, third edition edition, 2023. URL https://wee
ISBN 978-0-323-85277-7. doi: http:///bboor//0000/////82222222 1312.5602. cite http:////caaoorrrroaaaa/////////
sciencedirect.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 2013. URL http://arxxx org/abs/ NIPS Deep Learning Workshop 2013.
Volodymyr Muih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-38 33, 02 2015. doi: 10. 10.1038/naturel4/ 4236.
24

---

Published in Transactions on Machine Learning Research (01/2024)
Remi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy rein- forcement, learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances
in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:
//proceedings.none www.sneeerrrrppppppoese hwa.raa.c.pppeeoot../////saasaaa
Art B. Owen. Monte Carlo theory, methods and ezamples. 2013.
lan Parberry. Introduction to Game Physics with Bor2D. CRC Press, Inc., USA, 1st edition, 2013. ISBN 1466565764.
Doina Precup, Richard Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. Compuler Science Department Faculty Publication Series, 06 2000.
Doina Precup, Richard Sutton, and Sanjoy Dasgupta. Off-policy temporal-diiference learning with function approximation. Proceedings of the 18th International Conference on Machine Learning, 06 2001.
Antonin Raffin. RI basclines3 zoo. https://gghhuub..oo////////M//L2
2020.
Alfréd Rényi. On Measures of Entropy and Information. The Regents of the University of California, 1961.
Baturay Saglam, Dogan C. Cicek, Furkan B. Mutlu, and Suleyman S. Kozat. Safe and robust experience sharing for deterministic policy gradient algorithms, 2022a. URL https://arxii: http::://artt..or//a/////0000// org/abs/22077
13453.
Baturay Saglam, Furkan B. Mutlu, Dogan C. Cicek, and Suleyman S. Kozat. Actor prioritized experience replay, 2022b. URL https://arxiv. https://arxiv.org/abb//// wwtpg///g//ggo...///////// 00532.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR (Poster), 2016. URL setp://arxixz org/arr./////oooooooooo.... hts//bbs/15111. 05952.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay.
In Hal Daumé ITT and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8545-8554. PMLR, 13-18 Jul 2020. .ww.
URL btctps:/pproceediii
html.
Claude Elwood Shannon. mathematical theory of communication. The Bell System Technical Journal, 27: 379-423, 1948. URL http:// btttp://plan9 http://lnnn.eelllabb.baaa honmmmmmmass/satttttaannnonaaa shannon 1etioonn991.... pdf...........................
bell-labr
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1) (11):9 9944, Aug 1988. ISSN 1573-0565, doi: 10.1007/BF00115000 URL https://doi. wttp://0...00///0000000101111110
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, A. Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy cemfoorl-differrnce learning. Journal of Machine Learning Research, 17(73):1-29, 2016. URL http:// /ttp:://mmrrr wttp://pppsss//////eeepe html...mmm.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS, RSSJ International Conference on Intelligent Robols and Systems, pp. 5026 5033, 2012. doi: 10.1109/TROS.... TROS.2012. 6386109.
Tim van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797 3820, 2014. doi: 10.1109/TIT 2014..320500.
Christopher Watkins. Learning from delayed rewards, 01 1989.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279-292, May 1992. ISSN 1573-0565. doi: 10.1007/BF0099220 URL
Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229 256, 1992.
25

---

Published in Transactions on Machine Learning Research (01/2024)
Chengyang Ying, Zhongkai Hao, Ximning Zhou, Hang Su, Dong Yan, and Jun Zhu. On the reuse bias in off-policy reinforcement. learning. In Edith Elkind (ed.), Proceedings of the Thirly-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 4513 4521. International Joint Conferences on Artificial Intelligence Organization, 8 2023. doi: 10.24963/ /ijeaiiiiiiie /020 2023/502. URL https: http::/ddoi.rroo org/10. 24963/ijcai. 2023/502 502.. Main Track.
Huizhen Yu, A. Rupam Mahmood, and Richard S. Sutton. On generalized bellman equations and temporal- difference learning. Journal of Machine Learning Research, 19(48):1-44 49, 2018. URL attp://jmla /trr//rg//////////////////
papers/v19/17- html.
Daochen Zha, Kerng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. In Proceedings of the tveety-Eighhth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 4243 4249. International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 1t..4996//jcai.2219////
URL https: /toii org/ 24963/ijcai. 2019/589.
Shangtong Zhang and Richard Sutton. A deeper look at experience replay. orXiv preprint 2017.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor-critic.
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances
in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: .df.
// wttppppp....p.p.......//o/////a.
A Proof of Lemma
Lemma 8. The difference between HQ and its fized point QT is:
HQ(ss, a) - BBa [B, (Q -Q-))-0)))))
where X' is the similarity coefficient computed for the subsequent transition of (s, a.)'s)
Proof. The proof reduces the proof of Lemma 1 by Munos et al. (2016) to single-step TD learning when off- ofl-policy correction by AC-Off-POC is involved. Munos et al. (2016) consider the trajectories starting from = O. Thus, we consider 0 and denote the entities in the next step by superscript, e.g., a' a++1, since we focus on one-step returns. The general RETRACE operator for multi-step return-based off- off-policy algorithms is given by
RQ(s,a)== = Q(s,a) +
Σ
c) + +nB,Q(sHH,)) ai))
(14)
120
1=1
where e is the importance weight corresponding to the time step i. Setting in the definition of the operator R yiclds
RQ(s,a)= Q(s, E,
Σr Σ
c)( 7e ((0.((1)))))))) Q(s,,
120
==1
a(s, E, (r T -(s.a))
= =)Q(s,a).
where 8 80, = 0o, r ro, s' 81, and II IT when / = 0 (Munos et al., 2016). Notice that reduction to single-step off-policy correction yields the AC-Of-POC operator defined in Definition 1. As done by Munos et al. (2016), we rewrite Equation 14:
RQ(s,a) = ETE
[I cr) (B= c41 (000::s1))))(]((ss1)==)))))) ((4))))))))))1)
120
26

---

Published in Transactions on Machine Learning Research (01/2024)
which is obtained by bootstrapping. Setting t 0 reduces to the following alternative of H obtained by bootstrapping:
a)(s,a) = Σ BBn
c.)(F + [II ==1
c++1 ((4 (.,1))
120
t=
= My [r + (E x'Q(s',
where we replace 6j with the next similarity weight X, which is computed for the subsequent transition of (s,a,r,s'). The reduction from multi-step IS one-return off-policy correction theoretically implies that off-policy correction occurs for the expected return of the next state-action pair (8',a') (Harulyunyan el. al., 2016). As Q is the fixed point of R, and hence, H, we have
HQT a) Q" (s, a) Enfr + - -xQ"),,'))
Subtracting the fixed point (* a) from HQ(s, a), get.
HQ(s, 0) (= a) E, Ir + 7E -Q( (" P A,, -XXQ(((''))
where define AQ = Q
27


Reviews:
[{"review_id": "g4pf6W8XoQ", "reviewer": "Reviewer_3Y3g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n\nThe paper is interesting in that it provides empirical performance over continuous control baselines, using ideas from off-policy learning literature. The empirical improvements seem significant compared to the DDPG baseline.\n\n=== Weaknesses ===\n\nThough the empirical improvements are clear, I think much of the paper can be improved. For example, the theoretical motivation of the paper is unclear -- though the weighting idea is inspired from off-policy learning literature, it is only peripherally related to principled off-policy evaluation literature such as Retrace. It is not clear whether the proposed method converges at all or if converging to the optimal policy. As a result, the proposed method does not have enough theoretical grounding and does not offer much insight into the off-policy learning problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new empirical approach to off-policy RL. They propose new way of weighting actor-critic losses using ideas inspired from off-policy evaluation literature. They show the algorithmic procedure and display performance improvements over baseline such as DDPG over continuous control tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "=== weighted loss vs. off-policy evaluation ===\n\nI can see the motivation of the weighted loss -- weighting the actor-critic losses less if the behavior action is different from target action, and more otherwise. This weighting somehow helps the learning process to focus more on the more on-policy data rather than off-policy data.\n\nHowever, it is not clear what the actual underlying connection is between the proposed weighting loss and off-policy evaluation method such as Retrace, from which the paper draws inspirations. The connection feels very tangential in that Retrace is provably convergent in the off-policy evaluation and optimization settings, whereas the proposed method here feels more like an empirical trick. \n\nThe choice of the Retrace form $\\min(\\pi/\\mu,c)$ is justified by convergence theory, how about here? For example, the decaying form $e^{-\\rho}$ feels rather arbitrary and why do we choose exponent as the decreasing function of $\\rho$ rather than something else? Do we need an extra coefficient to modulate the decay such that the function is actually $e^{-\\alpha\\rho}$?\n\nThe definition of Gaussian interpolation in Eqn 8 is also a purely empirical approach that does not have any theoretical grounding.\n\n=== analysis ===\n\nThe theoretical analysis in 4.3.4 does not feel corresponding to the empirical loss proposed in the paper.\n\nFor starters, $HQ$'s behavior policy $\\eta$ does not serve much purpose in the expectation that defines the operator because it is an one-step transition. Indeed, fixing $s,a$, the only randomness in the expectation will be the one-step transition $s'\\sim p(\\cdot|s,a)$. Given a fixed target policy $\\pi$, the behavior policy $\\eta$ does not play a role in the definition. In Retrace the behavior policy and trace cutting is only meaningful when it is more than 1 step.\n\nMost importantly, the proposed method is **weighting** the loss, rather than **weighting** the back-up target. There is a fundamental difference here. The proposed method weights a mixture of one-step loss functions for critic, which ultimately just carries out one-step bootstrapping. This differs from carrying out a weighted target and regress against it, which leads to the classic Retrace type of approach (and also general dynamic programming to TD-learning kind of approach).\n\nFurther I do not see how the theory can meaningfully reduce from stochastic to deterministic policy, since in the latter case the problem of off-policy evaluation with multi-step operator is ill-defined. Indeed, if the target policy is deterministic, it can only be one-step and hence no multi-step learning is involved; if the behavior policy is deterministic, $\\pi/\\mu$ will blow up and importance sampling is ill-defined in that context. \n\n=== Empirical result ===\n\nThe gains are interesting but I think the empirical improvements alone cannot justify the approach being applied. It'd be nice and useful in general to see the sensitivity of the approach to various hyper-parameter settings such as the choice of the decay function $e^{-\\rho}$ vs. others, or just simple constant $c$ like in Retrace. \n\nInterestingly, Ant-v2 + DDPG seem to perform poorly and likewise for all the methods considered. This might be an issue with the baseline method itself. In openai spinningup baseline, they show that DDPG works definitely worse on Ant-v2 compared to other more sophisticated methods such as TD3 and SAC [1], I wonder if the authors have any thoughts regarding this -- since the baseline shown in the paper is even worse, its performance keeps dropping since the beginning.\n\n[1] https://spinningup.openai.com/en/latest/spinningup/bench.html", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OIchnF4FJ3", "reviewer": "Reviewer_3DcE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Importance weights are normally not applicable to deterministic policies without meanningless importance weights of $0$ or $\\infty$, so this does present a way to handle that scenario. The weighting is also suprirsingly good on the MuJoCo benchmark. It does require some assumptions of a Gaussian exploration noise (in order to define the base $N(0, \\sigma I)$), but this is common enough in continuous control that I don't think it's so bad to add that assumption.\n\nTo me the main weakness is that..it doesn't seem very motivated or theoretically valid? The authors mention that the 1-step correction done here avoids issues of vanishing importance weights that occur if you take the product over the trajectory. Normally the reason you take the product over the trajectory is because, from a fixed initial state $s_0$, policies $\\pi_1$ and $\\pi_2$ will visit subsequent states $s_1$, $s_2$, etc. with different probability, since $\\pi_1$'s actions will differ from $\\pi_2$. So the most theoretically correct thing to do is to multiply the importance weights over all timesteps, to account for difference in state visitation frequency.\n\nWhat this work essentially does is say, \"state visitation doesn't matter. Just assume it's fine and only handle the action difference.\"This works out because both DDPG and SAC converge on off-policy data, even without correction, in the tabular case, as long as you eventually visit every $(s,a)$ pair.\n\nSo, in short it feels more like an optimization trick that has been dressed up in more theory than it deserves. It's really just showing that you can get faster convergence if you reweight examples by how different their actions are in continuous space. And that's fine, but it overall felt like the paper had a lot of theory that it didn't really need.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aim to mitigate the challenges of off-policy data by including a corrective term for reweighting past data.\n\nThis is commonly done with importance weights. In this work, the authors focus on a deep RL setting where the policy is deterministic. This can create some challenges with naive importance weights $\\pi(a|s) / \\mu(a|s)$ where $\\pi$ is a one-hot action.\n\nTo get around this, the authors will model all difference between policies as following a multivariate Gaussian. More specifically:\n\nFor any minibatch $\\mathcal{B}$, we have the set of actions $a \\in \\mathcal{B}$ from $(s,a)$ examples. For each state $s$ we find $\\hat{a} = \\pi(s)$, which gives us a batch of action differences $\\dot a = a - \\hat{a}$, and within each minibatch we can fit a Gaussian to the action diffs, with mean $\\mu = 1/N \\sum \\dot a$ and covariance $\\Sigma$.\n\nAssuming that policies are run with exploration noise $\\sigma$, as is common in methods like DDPG, if $\\mu = \\pi$, even for fixed $s$ we will see varying actions in our replay buffer, so the similarity $\\rho$ is defined as the Jenson-Shannon divergence $JS(N(\\mu, \\Sigma) || N(0, \\sigma I))$. This makes $\\rho = 0$ in the limit of inifnite batch size when $\\mu = \\pi$. Each transition is then weighted by $\\lambda = e^{-rho]$. Since $\\rho \\ge 0$ this gives a weight of 1 for identical policies that trends to 0 as JS divergence rises.\n\nThe authors additionally propose an extension of this idea to stochastic policies (based on saving the distribution parameters of the behavior policy in the replay buffer), but fundamentally it computes similarity + reweights in the same way.\n\nThese are used to reweight the actor-critic update. The authors present some results showing this is still a contraction (based on RETRACE's treatment) and show it improves upon DDPG and SAC.\n\nThe difference between policies is then defined as the Jenson-Shannon divergence betwee", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No changes requested - I am not a fan of the presentation but I think the paper is presented clearly and am willing to believe this is just up to interpretation of the work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zz0SNYYKF4", "reviewer": "Reviewer_uJTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is easy to implement\n- The proposed method makes sense, and the paper is relatively clear\n- The results are very encouraging\n- The potential impact of this paper is very high. The proposed method improves DDPG, SAC and TD3, 3 SOTA algorithms that are extensively used. The method does so without introducing yet another neural network, or complicated mathematical approaches.\n\nWeaknesses:\n\n- The structure of the paper is sometimes a bit difficult to follow, with the readers often having to keep a queue of questions in their mind, with the questions being answered 1 or 2 pages later. The paper is to me a bit too \"top to bottom\", first giving results and then only explaining how we got to them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method to compute importance weights used to train actor and critic networks in off-policy Reinforcement Learning settings, focusing on continuous actions (but with both stochastic and deterministic actors). The core of the contribution is a method to compute similarity measures between an unknown behavior policy and the current actor, for each sample in a minibatch. The interesting aspect of the proposed method is that the similarity measure uses both local (the difference between the action selected by the behavior policy and the actor), and minibatch-global (some computed Gaussian distribution) to produce the weights.\n\nTheoretical and empirical evidence is provided that the proposed method makes sense and leads to good results. The empirical evaluation is particularly thorough and uses state-of-the-art baselines. The results seem very good in every case, illustrating the robustness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is no critical requested change. I would simply suggest that the paper is slightly reorganized to allow the readers to more easily understand the contribution and its motivation:\n\n- Section 1.2 seems a bit too long and superfluous. It provides an \"advertisement\" for the proposed method but does not mention the gist of the contribution, the fact that a similarity measure based on $\\mathcal{B}_a - \\pi(\\mathcal{B}_s)$ and a minibatch-specific Gaussian distribution is used. I would suggest to make Section 1.2 smaller, maybe by putting the list of contributions back in Section 1, and replacing Section 1.2 by some sort of table, that succintly compares the hypotheses and properties of AC-Off-POC with families of related work.\n- Section 4.1 could be split in two sections. Currently, it both (quite nicely) presents the contribution, equation after equation, but then it spends almost one page justifying the use of the JSD versus other distance metrics. Some other aspects of the contribution, such as convincing the reader that computing per-minibatch state-independent covariance matrices is a good idea, are a bit lacking. When reading that section and Equation 8, I was surprised that the state disappears. The later explanations motivate this approach and somewhat (90%) convince me that the state can safely be ignored, but more arguments would be nice. I would suggest to split the section in 2: first a section that succintly presents the contribution (and the pseudocode of Algorithm 1 at the same time), then a section that motivates every aspect. I don't know in which order the sections should be: either first the motivations and then the algorithm, or first the algorithm and then the motivations.\n- Section 4.3.1, \"Addressing potential concerns\", could have a more informative title. For increased readability, I would suggest to use a Latex \"definition\" in that section, or subsubsections, to list each concern and then explaining their mitigation.", "broader_impact_concerns": "There does not seem to be any risk of negative broader impacts, and as such no discussion of them seems necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary

This paper proposes an approach to mitigate off-policy bias in actor-critic reinforcement learning algorithms. Off-policy bias arises when the data used to train the agent's policy differs from the policy it is intended to optimize. This can lead to instability and poor performance. The proposed method uses a novel policy similarity measure to identify and correct off-policy data. This method applies to deterministic policy networks and can be used with one-step Q-learning. The authors show the proposed approach improves the performance over state-of-the-art methods on various continuous control tasks. 

## Decision

Overall, the method is novel, and the reviewers found it novel. The initial version of this paper was badly written, and the authors carefully rewrote the paper based on the reviews submitted and significantly improved it. Now, the current version of the paper paper reads well. As a result, the majority of the reviewers lean towards accepting. However, reviewer 3Y3g pointed out the paper is built upon somewhat shaky theoretical foundations because the theory of importance sampling for off-policy correction does not apply to the deterministic policy. However, I think the authors clarified that in the rebuttal.
Decision: Accept (poster)