Year: 2021

Abstract:
Popular off-policy deep reinforcement learning algorithms compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose a novel learnable penalty to enact such pessimism, based on a new way to quantify the critic’s epistemic uncertainty. Furthermore, we propose to learn the penalty alongside the critic with dual TD-learning, a strategy to estimate and minimize the bias magnitude in the target returns. Our method enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. Empirically, by integrating our method and other orthogonal improvements with popular off-policy algorithms, we achieve state-of-the-art results in continuous control tasks from both proprioceptive and pixel observations.

heading: 1 INTRODUCTION
text: Sample efficiency and generality are two directions in which reinforcement learning (RL) algorithms are still lacking, yet, they are crucial for tackling complex real-world problems (Mahmood et al., 2018). Consequently, so far, many RL milestones have been achieved through simulating conspicuous amounts of experience and tuning for effective task-specific parameters (Mnih et al., 2013; Silver et al., 2017). Recent off-policy model-free (Lee et al., 2021; Chen et al., 2021) and modelbased algorithms (Chua et al., 2018; Janner et al., 2019), pushed forward the state-of-the-art sampleefficiency on several benchmark tasks (Brockman et al., 2016). We attribute such improvements to two main linked advances: more expressive models to capture uncertainty and better strategies to counteract detrimental biases from the learning process. These advances yielded the stabilization to adopt more aggressive optimization procedures, with particular benefits in lower data regimes.
Modern off-policy algorithms learn behavior by optimizing the expected performance as predicted by a trained parametric deep model of the environment. Within this process, overestimation bias naturally arises from the maximization performed over the model’s performance predictions, and consequently, also over the model’s possible errors. In the context of model-free RL, such model is trained to predict the agent’s future returns via temporal difference (TD-) learning and is referred to as the critic. A common strategy to counteract overestimation is to parameterize the critic with multiple, independently-initialized networks and optimize the agent’s behavior over the minimum value of the relative outputs (Fujimoto et al., 2018). Empirically, this strategy consistently yields pessimistic target performance measures, avoiding overestimation bias propagating through the TDlearning target bootstraps. However, this approach directly links the critic’s parameterization to bias counteraction, making improvements in each of these components hard to pursue independently.
Based on these observations, we propose a more general formulation for counteracting overestimation bias independently. In particular, we compute the critic’s target performance predictions by replacing the ubiquitous minimization procedure with an explicit penalty that is agnostic to the critic’s parameterization. Our proposed penalty is the output of a linear model of the epistemic uncertainty, computed as the expected Wasserstein distance between the return distributions predicted by the critic. Based on this formulation, we derive Generalized Pessimism Learning (GPL), a new strategy that learns an accurate penalization throughout the RL process. Within this strategy, we propose optimizing the penalty’s weight with dual TD-learning, a new procedure minimizing the estimated bias in the critic’s performance predictions with dual gradient descent. GPL is the first effective method able to freely learn an unbiased performance objective throughout training.
Furthermore, we also extend GPL by introducing a new pessimism annealing procedure, motivated by the principle of optimism in the face of uncertainty (Brafman & Tennenholtz, 2002). This procedure leads the agent to adopt a risk-seeking behavior policy by utilizing a purposely biased estimate of the performance in the initial training stages. Hence, it trades-off expected immediate performance for directed exploration, incentivizing the visitation of states with high epistemic uncertainty from which the critic would gain more information.
We incorporate GPL with modern implementations of the Soft Actor-Critic (SAC) (Haarnoja et al., 2018a;b) and Data-regularized Q (DrQ) (Yarats et al., 2021b;a) algorithms, yielding GPL-SAC and GPL-DrQ, respectively. On the Mujoco environments from the OpenAI Gym suite (Todorov et al., 2012; Brockman et al., 2016), GPL-SAC outperforms prior, more expensive, model-based (Janner et al., 2019) and model-free (Chen et al., 2021) state-of-the-art algorithms. For instance, in the Humanoid environment GPL-SAC is able to recover a score of 5000 in less than 100K experience steps, more than nine times faster than regular SAC. Additionally, on pixel-based environments from the DeepMind Control Suite (Tassa et al., 2018), GPL-DrQ provides significant performance improvements from the recent state-of-the-art DrQv2 algorithm. These results highlight the effectiveness and applicability of GPL, in spite of introducing only negligible computational overheads. We release our implementations to facilitate future comparisons and extensions.
In summary, we make several contributions towards improving off-policy reinforcement learning:
• We propose a novel penalty to counteract overestimation bias, disentangling the critic’s parameterization from the enforced pessimism.
• We propose the first optimization method to estimate and precisely counteract overestimation bias throughout training with dual gradient descent.
• We propose a pessimism annealing strategy that exploits epistemic uncertainty to actively seek informative states in the early training stages.
• Integrating our method with SAC and DrQ, we achieve new state-of-the-art performance results with trivial overheads on both proprioceptive and pixel observations tasks.

heading: 2 RELATED WORK
text: Modern model-free off-policy algorithms utilize different strategies to counteract overestimation bias arising in the critic’s TD-targets (Thrun & Schwartz, 1993; Pendrith et al., 1997; Mannor et al., 2007). Many approaches combine the predictions of multiple function approximators to estimate the expected returns, for instance, by independently selecting the bootstrap action (Hasselt, 2010). In discrete control, such technique appears to mitigate the bias of the seminal DQN algorithm (Mnih et al., 2013), consistently improving performance (Van Hasselt et al., 2016; Hessel et al., 2018). In continuous control, similar strategies successfully stabilize algorithms based on the policy gradient theorem (Silver et al., 2014; Lillicrap et al., 2015). Most notably, Fujimoto et al. (2018) proposed to compute the critic’s TD-targets by taking the minimum over the outputs of two different actionvalue models. This particular minimization strategy has become ubiquitous, being employed in many popular follow-up algorithms (Haarnoja et al., 2018b; Yarats et al., 2021b). To better trade-off optimism and pessimism, Zhang et al. (2017) proposed using a weighted combination of the original and minimized targets. Instead, Kuznetsov et al. (2020) proposed to parameterize a distributional critic and drop a fixed fraction of the predicted quantiles to compute the targets. Moreover, as in our approach, several works also considered explicit penalties based on heuristic measures of epistemic uncertainty (Lee et al., 2013; Ciosek et al., 2019). Recently, Kumar et al. (2020) proposed to complement these strategies by further reducing bias propagation through actively weighing the TD-loss of different experience samples. Aleatoric uncertainty is also an additional source of bias in TD-learning, due to the practical inability of considering multiple transition samples in stochastic environments (Baird, 1995). This phenomenon is known as the double-sample issue, but has been rarely addressed in prior off-policy literature (Dai et al., 2018).
Within model-based RL (Atkeson & Santamaria, 1997), recent works have achieved remarkable sample efficiency by learning large ensembles of dynamic models for better predictions (Chua et al., 2018; Wang & Ba, 2019; Janner et al., 2019). In the model-free framework, prior works used large critic ensembles for more diverse scopes. Anschel et al. (2017) proposed to build an ensemble using several past versions of the value network to reduce the magnitude of the TD-target’s bias.
Moreover, Lan et al. (2020) introduced a sampling procedure for the critic’s ensemble predictions to regulate underestimation in the TD-targets. Their work was later extended to the continuous setting by Chen et al. (2021), which showed that large ensembles combined with a high update-to-data ratio enable to outperform the sample efficiency of contemporary model-based methods. Ensembling has also been used to achieve better exploration following the principle of optimism in the face of uncertainty (Brafman & Tennenholtz, 2002) in both discrete (Osband et al., 2016; Chen et al., 2017) and continuous settings (Ciosek et al., 2019). Lee et al. (2021) further showed the effectiveness of combining several of these strategies in a unified framework.
In the same spirit as this work, multiple prior methods attempted to learn components and parameters of underlying RL algorithms. Several works have approached this problem by utilizing expensive meta-learning strategies to obtain new learning objectives based on the multi-task performance from low-computation environments (Bechtle et al., 2021; Oh et al., 2020; Xu et al., 2020; Co-Reyes et al., 2021). More related to our work, Moskovitz et al. (2021) recently proposed to use an adaptive binary controller that switches on or off a bias correction penalty for the TD-targets. In particular, they treat the controller optimization task as a multi-armed bandit problem performed throughout different training iterations to maximize immediate performance improvements. Instead, GPL makes effective use of dual gradient descent to minimize bias directly, similarly to how Haarnoja et al. (2018a;b) learns the exploration temperature parameter in the Soft Actor-Critic (SAC) algorithm.

heading: 3 PRELIMINARIES
text: In RL, we aim to autonomously recover optimal agent behavior for performing a particular task. Formally, we describe this problem setting as a Markov Decision Process (MDP), defined as the tuple (S,A, P, p0, r, γ). At each time-step of interaction, the agent observes some state in the state space, s ∈ S, and performs some action in the action space, a ∈ A. The transition dynamics function P : S × A × S → R and the initial state distribution p0 : S → R describe the evolution of the environment as a consequence of the agent’s behavior. The reward function r : S × A → R quantifies the effectiveness of each performed action, while the discount factor γ ∈ [0, 1) represents the agent’s preference for earlier rewards. A policy π : S ×A → R maps each state to a probability distribution over actions and represents the agent’s behavior. An episode of interactions between the agent and the environment yields some trajectory τ containing the transitions experienced, τ = (s0, a0, r0, s1, a1, r1, ...). The RL objective is then to find an optimal policy π∗ that maximizes the expected sum of discounted future rewards:
π∗ = argmax π Epπ(τ) [ ∞∑ t=0 γtr(st, at) ] , (1)
where pπ(τ) represents the distribution of trajectories stemming from the agent’s interaction with the environment. Off-policy RL algorithms commonly utilize some critic model to evaluate the effectiveness of the agent’s behavior. A straightforward choice for the critic is to represent the policy’s action-value function Qπ : S × A → R. This function quantifies the expected sum of discounted future rewards after executing some particular action from a given state:
Qπ(s, a) = Epπ(τ |s0=s,a0=a) [ ∞∑ t=0 γtr(st, at) ] . (2)
Most RL algorithms consider learning parameterized models for both the policy, πθ, and the corresponding action-value function, Qπϕ. In particular, after storing experience transitions (s, a, s
′, r) in a replay data buffer D, we learn Qπϕ by iteratively minimizing a squared TD-loss of the form:
JQ(ϕ) = E(s,a,s′,r)∼D [ (Qπϕ(s, a)− y)2 ] ,
y = r + γEa∼π(s′) [ Q̂πϕ′(s ′, a) ] .
(3)
Here, the TD-targets y are obtained by computing a 1-step bootstrap with a target action-value estimator Q̂πϕ′ . Usually, Q̂ π ϕ′ is a regularized function of action-value predictions from a target critic model using delayed parameters ϕ′. Following the policy gradient theorem (Sutton et al., 2000; Silver et al., 2014), we can then improve our policy by maximizing the expected returns as predicted by the critic, e.g., by minimizing the negated action-value estimates:
Jπ(θ) = −Es∼D,a∼πθ(s) [ Q̂πϕ(s, a) ] . (4)

heading: 4 ADDRESSING OVERESTIMATION BIAS

heading: 4.1 BIAS IN Q-LEARNING
text: In off-policy RL, several works have identified an accumulation of overestimation bias in the actionvalue estimates as a consequence of TD-learning (Thrun & Schwartz, 1993; Pendrith et al., 1997; Mannor et al., 2007). Formally, we quantify the target action-value bias B(s, a, s′) as the difference between the actual and estimated TD-targets for a given transition:
B(s, a, s′) = r + γEa′∼π(s′) [ Q̂πϕ′(s ′, a′) ] − ( r + γEa′∼π(s′) [Qπ(s′, a′)] ) = γEa′∼π(s′) [ Q̂πϕ′(s ′, a′)−Qπ(s′, a′) ] .
(5)
As discussed by Fujimoto et al. (2018), positive bias arises when the target action-values are obtained directly from the outputs of a parameterized action-value function, i.e., Q̂πϕ′ = Q π ϕ′ . The reason for this phenomenon is that the policy is trained to locally maximize the action-value estimates from Eqn. 4. Hence, its actions will exploit potential model errors to obtain higher scores, implying that Es,a∼π(s)[Qπϕ′(s, a)] > Es,a∼π(s)[Qπ(s, a)]. Instabilities then arise as the errors can quickly propagate through the bootstrap operation, inherently causing the phenomenon of positive bias accumulation. To counteract this phenomenon, Fujimoto et al. (2018) proposed clipped double Q-learning. This technique consists in learning two separate action-value functions and computing the target action-values by taking the minimum over their outputs:
Q̂πϕ′min(s, a) = min ( Qπϕ′1(s, a), Q π ϕ′2 (s, a) ) . (6)
The role of the minimization is to consistently produce overly pessimistic estimates of the target action-values, preventing positive bias accumulation. This approach is an empirically effective strategy for different benchmark tasks and has become standard practice.

heading: 4.2 THE UNCERTAINTY REGULARIZER
text: In this work, we take a more general approach for computing the target action-values. Particularly, we use a parameterized function, the uncertainty regularizer pβ(s, a, ϕ, θ), for trying to approximate the bias in the critic’s action-value predictions for on-policy actions. Thus, we specify an actionvalue estimator that penalizes the action-value estimates via the uncertainty regularizer:
Q̂πϕ′(s, a) =Q π ϕ(s, a)− pβ(s, a, ϕ, θ),
where pβ(s, a, ϕ, θ) ≈Qπϕ(s, a)−Qπ(s, a), for a ∼ πθ(s). (7)
A consequence of this formulation is that as long as pβ is unbiased for on-policy actions, so will the action-value estimator. Hence, the expected target action-value bias will be zero, preventing the positive bias accumulation phenomenon without requiring overly pessimistic action-value estimates. Based on these observations, in the next section, we specify a new method that learns an unbiased uncertainty regularizer and continuously adapts it to reflect changes in the critic and policy.

heading: 5 GENERALIZED PESSIMISM LEARNING
text: Generalized pessimism learning (GPL) entails learning a particular parameterized uncertainty regularizer pβ , defined in Eqn. 8. Our method makes pβ adapt to changes in θ and ϕ throughout the RL process to keep the target action-values unbiased. Hence, GPL allows preventing positive bias accumulation without overly pessimistic targets. With any fixed penalty, we argue that it would be infeasible to maintain the expected target action-value bias close to zero due to the number of affecting parameters and stochastic factors in different RL experiments.

heading: 5.1 UNCERTAINTY REGULARIZER PARAMETERIZATION
text: We strive for a parameterization of the uncertainty regularizer that ensures low bias and variance estimation of the target action-values. Similar to prior works (Ciosek et al., 2019; Moskovitz et al., 2021), GPL uses a linear model of some measure of the epistemic uncertainty in the critic. Epistemic uncertainty represents the uncertainty from the model’s learned parameters towards its possible predictions. Hence, assuming enough model expressivity, the areas of the state and action spaces where the critic’s epistemic uncertainty is elevated are the areas in which the agent did not yet observe enough data to reliably predict its returns and, for this reason, the magnitude of the critic’s error is expectedly higher. Consequently, if a policy yields behavior with high epistemic uncertainty in the critic, it is likely exploiting positive errors and overestimating its expected returns. As we use the policy to compute the TD-targets, the higher the uncertainty, the higher the expected positive bias. Several prior works further discuss this relationship (Fujimoto et al., 2018; Ciosek et al., 2019).
We propose measuring epistemic uncertainty with the expected Wasserstein distance between the critic’s predicted return distributions Zπ , as defined by Bellemare et al. (2017). In the usual case where we parameterize the critic with multiple action-value functions, we interpret each action-value estimate as a Dirac delta function approximation of the return distribution, Zπϕ (s, a) = δQπϕ(s,a). Our uncertainty regularizer then consists of linearly scaling this measure via a learnable parameter β:
pβ(s, a, ϕ, θ) = β × Ea∼πθ(s),ϕ1,ϕ2 [ W (Zπϕ1(s, a), Z π ϕ2(s, a)) ] . (8)
We estimate the expectation in Eqn. 8 by learning a critic ensemble of N ≥ 2 independent models with parameters {ϕi}Ni=1, and averaging the distances between the corresponding predicted return distributions. Notably, the Wasserstein distance has easy-to-compute closed forms for many popular distributions. For Dirac delta functions, it is equivalent to the distance between the corresponding locations, hence, W (δQπϕ1 (s,a), δQ π ϕ2 (s,a)) = |Qπϕ1(s, a)−Q π ϕ2 (s, a)|.
Our quantification of epistemic uncertainty is an interpretable measure for any distributional critic. Moreover, for some fixed β, increasing the number of critics decreases the estimation variance but leaves the expected magnitude of the uncertainty regularizer unchanged. This is because the sample mean of the Wasserstein distances is always an unbiased estimate of Eqn. 8 for N ≥ 2. Under reasonable assumptions, it is proportional to the standard deviation of the distribution of actionvalue predictions. We can also restate clipped double Q-learning using our uncertainty regularizer with N = 2 and β = 0.5, allowing us to replicate its penalization effects for N > 2 by simply fixing β. In contrast, Ciosek et al. (2019) proposed the sample standard deviation of the action-value predictions to quantify epistemic uncertainty. However, the sample standard deviation does not have a clear generalization to arbitrary distributional critics and its expected magnitude is dependent on the number of models. We provide formal derivations for these claims in Appendix A.
Algorithm 1 GPL-SAC 1: procedure TRAININGLOOP(N,UTD, ρ, γ, Ĥ) 2: Initialize πθ, {Qπϕi} N i=1, {Qπϕ′i} N i=1, β ← 0.5, α← 0.0, D ← ∅
3: loop 4: Observe s, execute a ∼ πθ(s), collect s′, r 5: Store D ← D ∪ (s, a, s′, r) 6: for j = 1, 2, ...,UTD do 7: Sample minibatch {(s, a, s′, r)} ∈ D, a′ ∼ π(s′) 8: Compute Q̂πϕ′(s
′, a′)← REGULARIZEDVALUE(s′, a′, ϕ′, θ, β) 9: Compute the TD-targets y ← r + γ ( Q̂πϕ′(s ′, a′)− α log π(a′|s′) )
10: for i = 1, 2, ...,N do 11: Compute the TD-errors for the ith critic ei ← Qπϕi − y 12: Update ϕi to minimize J(ϕ) = e2i ▷ Learning the critic 13: Update ϕ′i ← ρϕ′i + (1− ρ)ϕi ▷ Updating the target critic 14: Update β to minimize J(β) = β × ∑N i=1 ei ▷ Learning the bias (dual TD-learning) 15: Sample a ∼ π(s), compute Q̂πϕ(s, a)← REGULARIZEDVALUE(s, a, ϕ, θ, β) 16: Update θ to minimize J(θ) = −Q̂πϕi(s, a) + α log π(a|s) ▷ Learning the policy 17: Update α to minimize J(α) = α× (− log π(a|s)− Ĥ) ▷ Learning the entropy bonus 18: procedure REGULARIZEDVALUE(s, a, ϕ, θ, β) 19: Compute pβ(s, a, ϕ, θ)← βN2−N ∑N i=1 ∑N j ̸=i |Q π ϕi (s, a)−Qπϕj (s, a)| ▷ Sample mean of Eqn. 8
20: Compute Q̄πϕ(s, a)← 1N ∑N i=1 Q π ϕi (s, a) 21: return Q̄πϕ(s, a)− pβ(s, a, ϕ, θ)

heading: 5.2 DUAL TD-LEARNING
text: The expected bias present in the action-value targets is highly dependent on several stochastic factors stemming from aleatoric uncertainty and the learning process. We empirically show this by running multiple experiments with simple extensions of the SAC algorithm in different Gym environments and periodically recording estimates of the action-value bias by comparing the actual and estimated discounted returns (as described in Appendix B). As shown in Figure 2, the bias in the predicted action-values notably varies across environments, agents, training stages, and even across different random seeds. These results reinforce our thesis that there is no fixed penalty able to account for the many sources of stochasticity in the RL process, even for a single task. Hence, they show the necessity of learning pβ alongside the policy and critic to accurately counteract bias.
GPL optimizes β as a dual variable, enforcing the expected target action-value bias to be zero:
argmin β
−β × Es,a,s′ [B(s, a, s′)] . (9)
For arbitrary actions, the action-value estimates are not directly affected by the positive bias induced by the policy gradient optimization. Consequently, we can make the reasonable assumption of the critic itself providing an initially unbiased estimate of the expected returns, i.e., Es,a[Qπϕ(s, a)] ≈ Es,a[Qπ(s, a)]. Thus, to approximate B(s, a, s′), we propose to use the differences between the current TD-targets and action-value predictions for off-policy actions:
B(s, a, s′) ≈ B̂(s, a, s′) = r + γEa′∼π(s′) [ Q̂πϕ′(s ′, a′) ] −Qπϕ(s, a). (10)
In practice, GPL alternates the optimizations of β for the current bias, and both ϕ and θ, with the corresponding updated RL objectives. This procedure is similar to the automatic exploration temperature optimization proposed by Haarnoja et al. (2018b), approximating dual gradient descent (Boyd & Vandenberghe, 2004). We can estimate the current bias according to Eqn. 10 at minimal extra cost by negating the errors from the TD-loss. Thus, we name this procedure dual TD-learning.
Unfortunately, the unbiasedness assumption of Eqn. 10 does not necessarily hold when using deep networks and approximate stochastic optimization. In particular, given initially biased targets, some of the bias might propagate to the critic model, influencing the approximation in Eqn. 10. Nonetheless, GPL’s performance and the optimization dynamics of dual TD-learning appear to be empirically robust to moderate levels of initial target bias. We show this in Appendix D.1 by analyzing the
behavior and performance of GPL with different initial values of β and, thus, different bias in the initial targets. An intuition for this robustness is that the relative difference between the off-policy and on-policy action-value predictions should always push β to counteract bias stemming from model errors in the policy gradient action maximization (Fujimoto et al., 2018). We further validate dual TD-learning in Appendix D.2 by comparing and discussing alternative optimization strategies. In particular, we evaluate optimizing β by minimizing the squared norm of the bias and by using the bandit-based optimization from Tactical Optimism and Pessimism (Moskovitz et al., 2021).

heading: 5.3 PESSIMISM ANNEALING FOR DIRECTED EXPLORATION
text: As described in Section 3, the policy learns to maximize the unbiased action-values predicted by the online estimator Q̂πϕ. Motivated by the principle of optimism in the face of uncertainty (Brafman & Tennenholtz, 2002), we also consider an alternative optimistic policy gradient objective:
Joptπ (θ) = −Es,a∼π(s)[Q̂ πopt ϕ (s, a)], where Q̂ πopt ϕ (s, a) = Q π ϕ(s, a)− pβopt(s, ϕ, θ). (11)
This objective utilizes an optimistic shifted uncertainty regularizer, pβopt , calculated with parameter βopt = β − λopt, for some decaying optimistic shift value, λopt ≥ 0. This new objective trades off the traditional exploitative behavior of the policy with directed exploration. In particular, as λopt is large, π will be incentivized to perform actions for which the outcome has high epistemic uncertainty. Therefore, the agent will experience transitions that are increasingly informative for the critic but expectedly sub-optimal. Hence, we name the process of decaying λopt pessimism annealing, striving for improved exploration early on without biasing the policy’s final objective.

heading: 6 EXPERIMENTS
text: To evaluate the effectiveness of GPL, we integrate it with two popular off-policy RL algorithms. GPL itself introduces trivial computational and memory costs as it optimizes a single additional weight, re-utilizing the errors in the TD-loss to estimate the bias. Moreover, we implement the critic’s model ensemble as a single neural network, using linear non-fully-connected layers evenly splitting the nodes and dropping the weight connections between the splits. Practically, when evaluated under the same hardware, this results in our algorithm running more than two times faster than the implementation from Chen et al. (2021) while having a similar algorithmic complexity.
We show that GPL significantly improves the performance and robustness of off-policy RL, concretely surpassing prior algorithms and setting new state-of-the-art results. In our evaluation, we repeat each experiment with five random seeds and record both mean and standard deviation over
the episodic returns. We report additional details of our experimental settings and utilized hyperparameters in Appendix B. Furthermore, we provide comprehensive extended results analyzing different parameters, alternative implementations, and training times in Appendix D.

heading: 6.1 CONTINUOUS CONTROL FROM PROPRIOCEPTIVE OBSERVATIONS
text: GPL-SAC. First, we integrate GPL with Soft Actor-Critic (SAC) (Haarnoja et al., 2018a;b), a popular model-free off-policy algorithms that uses a weighted entropy term in its objective to incentivize exploration. Specifically, we substitute SAC’s clipped double Q-learning with our uncertainty regularizer, initialized with β = 0.5. We also adopt additional practices inspired by recent related works (Chen et al., 2021; Bjorck et al., 2021). In particular, we parameterize ten independent action-value functions as ‘modern’ residual neural networks with spectral normalization (Miyato et al., 2018) and increase the critic’s update-to-data (UTD) ratio to twenty. We denote the resulting algorithm GPL-SAC and provide a summary of this integration in Algorithm 1, highlighting the novelties.
Baselines. We compare GPL-SAC with prior state-of-the-art model-free and model-based algorithms with similar or greater computational complexity, employing high UTD ratios:
• REDQ (Chen et al., 2021): State-of-the-art, model-free algorithm on the OpenAI Gym suite. This algorithm employs multiple parameterized action-value functions and utilizes clipped double Q-learning over a sampled pair of outputs to compute the critic’s targets.
• MBPO (Janner et al., 2019): State-of-the-art, model-based algorithm on the OpenAI Gym suite. This algorithm learns a large ensemble of world models and employs a Dyna-style (Sutton, 1991) optimization procedure to train the policy with short, generated rollout data.
• SAC-20: Simple SAC extension where we increase the UTD ratio to twenty.
Results. We evaluate GPL-SAC compared to the described baselines on five of the more challenging environments from the OpenAI Gym suite (Brockman et al., 2016), involving complex locomotion problems from proprioceptive observations. We evaluate the performance by collecting the returns over five evaluation episodes every 1000 environment steps. In Figure 3, we provide visualizations of the different performance curves. We extend this analysis in Appendix C.
GPL-SAC is consistently the best performing algorithm on all environments, setting new state-ofthe-art results for this benchmark at the time of writing. Moreover, the performance gap is greater for tasks with larger state and action spaces. We motivate this by noting that all baselines use fixed strategies to counteract overestimation bias, thus, requiring overly pessimistic estimates of the returns to avoid instabilities. Hence, the resulting policies are likely overly conservative, hindering exploration and efficiency, with larger effects on the more complex tasks. For instance, on the Humanoid task, GPL-SAC remarkably surpasses a score of 5000 by the 100K experience threshold, more than nine times faster than SAC and more than 2.5 times faster than REDQ.
Ablations. Furthermore, we provide results from ablating the main components that differentiate GPL-SAC from SAC-20 on a subset of environments. Namely, we evaluate the contributions from learning the uncertainty regularizer, increasing the number of action-value functions, and using the improved modern action-value network architecture. In Figure 4, we provide visualizations of the performance curves. Empirically, each component consistently improves performance, with higher contributions from learning the uncertainty regularizer and increasing the critic’s ensemble size. We provide many further in-depth ablation studies as part of Appendix D.

heading: 6.2 CONTINUOUS CONTROL FROM PIXELS
text: GPL-DrQ. We also integrate GPL with a recent version of Data-regularized Q (DrQv2) (Yarats et al., 2021a), an off-policy, model-free algorithm achieving state-of-the-art performance for pixelbased control problems. DrQv2 combines image augmentation from DrQ (Yarats et al., 2021b) with several advances such as n-step returns (Sutton & Barto, 2018) and scheduled exploration noise (Amos et al., 2021). Again, we substitute DrQv2’s clipped double Q-learning with our uncertainty regularizer. To bolster exploration in pixel-based environments, we also integrate pessimism annealing from Section 5.3, with λopt linearly decayed from 0.5 to 0.0 together with the exploration noise in DrQv2. We leave the rest of the hyper-parameters and models unaltered to evaluate the generality of applying GPL. We name the resulting algorithms GPL-DrQ and GPL-DrQ-Expl+, respectively.
Results. We evaluate GPL-DrQ and GPL-DrQ-Expl+ on the environments from the DeepMind Control Suite (Tassa et al., 2018) modified to yield pixel observations. We use the medium benchmark evaluation as described by Yarats et al. (2021a), consisting of 12 complex tasks involving control problems with hard exploration and sparse rewards. In Table 1 we report the returns obtained after experiencing 3M and 1.5M environment frames. For each run, we average the returns from 100 evaluation episodes collected in the 100K steps preceding each of these milestones. We highlight the top performances that are within half a standard deviation from the highest mean return. We provide visualizations of the relative performance curves in Appendix C.
Both GPL-DrQ and GPL-DrQ-Expl+ significantly improve the performance of DrQv2 in most tasks. In particular, DrQv2 sporadically yields underperforming returns, likely due to a lack of exploration from its overly pessimistic critic1. GPL generally appears to resolve this issue, while pessimism annealing further aids precisely in the tasks where under-exploration is more frequent. Overall, these results show both the generality and effectiveness of GPL to improve the current state-of-theart through simple integrations by providing a framework to better capture and exploit bias.

heading: 7 DISCUSSION AND FUTURE WORK
text: We propose Generalized Pessimism Learning (GPL), a strategy that adaptively learns a penalty to recover an unbiased performance objective for off-policy RL. Unlike traditional methods, GPL achieves training stability without necessitating overly pessimistic estimates of the target returns, thus, improving convergence and exploration. We show that simple integrations of GPL with modern algorithms yield state-of-the-art results for both proprioceptive and pixel-based control tasks. Moreover, GPL’s penalty has a natural generalization to different distributional critics and variational representations of the weights posterior. Hence, our method has the potential to facilitate research going beyond action-value functions and model ensembles for continuous control, two exciting extensions we leave for future work.
1This instability was also observed by DrQv2’s authors after re-collecting their results.

heading: ETHICS STATEMENT
text: This work aims to improve the efficiency and generality of reinforcement learning algorithms. Thus, it may contribute to future effective deployments of autonomous agents for real-world applications. Furthering automation has the potential to provide many economic and social benefits to humanity. However, if unregulated, such advancements could accentuate societal inequalities, worsen the consequences of harmful misuse, and have a tangible environmental impact.

heading: REPRODUCIBILITY STATEMENT
text: In the supplementary material, we provide a compact version of our source code that enables the reproduction of all experiments in this work. After review, we will open-source a comprehensive, documented repository for this project to facilitate future extensions.

heading: APPENDIX

heading: A PENALTIES FOR BIAS COUNTERACTION
text: Generalized Pessimism Learning makes use of the uncertainty regularizer penalty to counteract biases arising from the off-policy RL optimization process. Here, we further analyze some of the properties and relationships of our penalty with alternatives from prior works. We will consider the common case where the critic is parameterized as an ensemble of N ≥ 2 action-value functions {Qπϕi} N i=1. To simplify our analysis, we assume that the different action-value predictions for a given input are independently drawn from a Gaussian distribution with some variance σ2Q(s,a). We motivate this latter assumption by the Central Limit Theorem, given the many sources of stochasticity affecting the training process of each action-value model in the ensemble. To simplify our notation, we will drop input dependencies when doing so will not compromise clarity, e.g., σ2Q(s,a) ⇒ σ 2 Q.

heading: A.1 EXPECTED PENALIZATION OF THE UNCERTAINTY REGULARIZER
text: The uncertainty regularizer penalty pβ(s, a, ϕ, θ) is parameterized as a weighted model of epistemic uncertainty. Particularly, we propose to measure epistemic uncertainty with the expected Wasserstein distance between the return distributions predicted by the critic. This is formally described in Eqn. 8 from Section 5. In the common case where we parameterize the critic with an ensemble of action-value functions, we treat each as a Dirac delta function approximation of the return distribution. Consequently, we can estimate the expected Wasserstein distance by averaging over all the N2 − N absolute differences between different Dirac locations predicted by the action-value functions:
pβ(s, a, ϕ, θ) = β × Ea∼πθ(s),ϕ1,ϕ2 [ W (Zπϕ1(s, a), Z π ϕ2(s, a)) ] = β × Ea∼πθ(s),ϕ1,ϕ2 [ |Qπϕ1(s, a)−Q π ϕ2(s, a)|
] ≈ β
N2 −N N∑ i=1 N∑ j ̸=i |Qπϕi(s, a)−Q π ϕj (s, a)|.
(12)
From the independence assumption of the ensemble models, the differences in the action-value predictions will also follow a Gaussian distribution:
Dij = Q π ϕi(s, a)−Q π ϕj (s, a) ∼ N(0, 2σ 2 Q). (13)
Each absolute difference Wij = |Dij | will then follow a folded normal distribution (Leone et al., 1961), with moments given by:
E[W ] = µW = √ 2
π σD =
2 π σQ, var(W ) = σW = σ 2 D − µ2W =
( 2− 4
π2
) σ2Q. (14)
Consequently, the expected penalization of the uncertainty regularizer immediately follows:
E[pβ ] = βµW = 2β
π σQ. (15)

heading: A.2 EXPECTED PENALIZATION OF THE POPULATION STANDARD DEVIATION
text: Ciosek et al. (2019) and Moskovitz et al. (2021) make use of a weighted model of an alternative epistemic uncertainty measure to define a penalty as psβ(s, a, ϕ, θ) = β × s(s, a, ϕ, θ). In particular, they make use of the population standard deviation s(s, a, ϕ, θ), treating the critic’s ensemble predictions as independently sampled action-values:
s(s, a, ϕ, θ) = √√√√ 1 N N∑ i=1 ( Qπϕi(s, a)− µQϕ(s,a) )2 , where µQϕ = 1 N N∑ i=1 Qπϕi . (16)
We can rewrite the population standard deviation measure in terms of the square root of the sum of N squared difference terms Di:
s = √√√√ N∑ i=1 D2i , where Di = 1√ N ( Qπϕi − µQϕ ) . (17)
Moreover, we can rewrite each difference term as a weighted sum of the N different ensemble predictions:
Di = 1√ N
( Qπϕi − 1
N N∑ i=1 Qπϕi
) =
1√ N N − 1 N Qπϕi − ∑ j ̸=i Qϕj N  . (18) From the independence assumption of the ensemble models, it follows that each Di is also a Gaussian random variable with moments given by the basic properties for the sums of independent random variables:
E[D] = µD = 0, var(D) = σ2D = 1
N
( (N − 1)2
N2 σ2Q + N − 1 N2 σ2Q
) =
N − 1 N2 σ2Q. (19)
Hence, by simply scaling the population standard deviation, we can express it as a sum of standard normal random variables:
N
σQ √ N − 1 s = √√√√ N∑ i=1 ( N σQ √ N − 1 Di )2
= √√√√ N∑ i=1 Z2i , where Z ∼ N(0, 1).
(20)
This enables to relate s to a Chi distribution (Forbes et al., 2011) with parameter N :
N
σQ √ N − 1 s ∼ χN .
Consequently, the expected value of psβ can be obtained from scaling the expected value of a Chi distribution:
E[psβ ] = βE[s] = β √ N − 1 N E[χN ]σQ = β √ N − 1 N
× √ 2Γ ( N+1 2 ) Γ ( N 2
) σQ. (21) Comparing the expected value of this penalty with the expected value of the uncertainty regularizer penalty from Eqn. 15, we note a few key facts. Both quantities are linearly proportional to the standard deviation of the action-value predictions σQ. However, for the population standard deviation penalty, the scale of the proportionality is dependent on the number of action-value models used to parameterize the critic. Hence, unlike for the uncertainty regularizer penalty, the critic’s parameterization directly affects the expected penalization magnitude and should influence design choices regarding the parameter β.

heading: A.3 RELATIONSHIP WITH CLIPPED DOUBLE Q-LEARNING
text: We can rewrite the ubiquitous clipped double Q-learning penalization practice (Fujimoto et al., 2018) using the uncertainty regularizer penalty. In particular, we can specify the clipped double Q-learning action-value targets from Eqn. 6 in terms of the difference between the mean action-value prediction and a penalty:
Q̂ϕmin(s, a) π = min ( Qπϕ1(s, a), Q π ϕ2(s, a) ) = Q̄πϕ(s, a)− pmin(s, a, ϕ, θ),
where Q̄πϕ = 1
N N∑ i=1 Qπϕi , pmin = min ( Qπϕ1 , Q π ϕ2 ) − Q̄πϕ = 1 2 |Qπϕ1 −Q π ϕ2 |.
(22)
Similarly, for N = 2 our uncertainty regularizer reduces to:
pβ = β
2
( |Qπϕ1 −Q π ϕ2 |+ |Q π ϕ2 −Q π ϕ1 | ) = β|Qπϕ1 −Q π ϕ2 |. (23)
Thus, pmin = pβ for β = 0.5. As shown earlier in this section, the expected magnitude of the uncertainty regularizer penalty is not dependent on the number of action-value functions. Hence, our penalty allows us to extend the expected regularization induced by clipped double Q-learning for N > 2, simply fixing β = 0.5. This would not be possible using the population standard deviation penalty since its expected magnitude is dependent on the number of action-value functions employed.

heading: B ALGORITHMIC AND EXPERIMENTAL SPECIFICATIONS
text: In this section, we provide details regarding the experimental results from the main text. For further information about our efficient implementation, please, refer to the shared code.

heading: B.1 EMPIRICAL BIAS ESTIMATION
text: In Section 5, we record the evolution of the target action-value bias throughout the RL process by running experiments with two different extensions to the SAC algorithm on three OpenAI Gym environments. The first extension makes use of unpenalized optimistic target action-values, while the second extension makes use of a penalty with a magnitude equivalent to the one induced by the clipped double Q-learning targets. Both are implemented through the uncertainty regularizer with a fixed β, as derived in Appendix A. Moreover, both extensions use an update-to-data ratio of twenty and N = 4 action-value models to parameterize the critic. The rest of the hyper-parameters follow our implementation of GPL-SAC. We record estimates of the action-value bias by collecting transitions from ten evaluation episodes every 1000 environment steps. We obtain each estimate by taking the average difference between the observed discounted returns in the evaluation episodes and the discounted returns from the critic’s predictions. In particular, we correct the critic’s raw predictions by subtracting the discounted log probabilities of the performed actions, accounting for SAC’s modified action-value objective.

heading: B.2 PROPRIOCEPTIVE OBSERVATIONS EXPERIMENTS
text: Hyper-parameters. We provide a list of the utilized hyper-parameters for GPL-SAC in Table 2. The majority of the chosen hyper-parameters follow closely the seminal SAC papers (Haarnoja et al., 2018a;b), with some exceptions to improve performance and stability. For instance, consistently with Chen et al. (2021), we employ an ensemble of ten action-value models and increase the updateto-data ratio to twenty. Additionally, we employ a simplified version of the modern architecture from Bjorck et al. (2021) to parameterize the action-value models, as depicted in Figure 5. This
architecture follows many of the practices introduced by recent work to stabilize transformer training (Xiong et al., 2020). Specifically, we employ a single hidden residual block with layer normalization (Ba et al., 2016) followed by two fully-connected layers regularized with spectral normalization (Miyato et al., 2018). Throughout all architectures, we keep the hidden dimensionality fixed to 256. We initialize the uncertainty regularizer with β = 0.5 to reflect the penalization magnitude of clipped double Q-learning. Following Haarnoja et al. (2018b), the entropy coefficient α is adaptively updated at each training iteration based on a policy target entropy Ĥ . In particular, this follows a dual optimization procedure to keep the estimated average policy entropy close to Ĥ , as described in Line 17 of Algorithm 1. We utilize increased values of Ĥ to optimize α, following the choices of Janner et al. (2019). We learn β using dual TD-learning with the same optimizer used for adjusting the value of α.
Baseline results. For the continuous control experiments from proprioceptive states, we employ different baselines to ground our results and provide a comparison with current state-of-the-art algorithms. The reported results for REDQ come from re-running Chen et al. (2021)’s original implementation with the provided hyper-parameters. The reported results for MBPO come instead from the original paper, as publicly shared by Janner et al. (2019). The reported results for SAC20 come from running the same base implementation as GPL-SAC, with a few differences in the listed hyper-parameters. Namely, SAC-20 uses an ensemble of N = 2 action-value models with the classical 3-layer fully-connected architecture from Haarnoja et al. (2018b) and uses an uncertainty regularizer penalty with a fixed parameter β = 0.5.

heading: B.3 PIXEL OBSERVATIONS EXPERIMENTS
text: Hyper-parameters. We provide a list of the utilized hyper-parameters for GPL-DrQ in Table 3. All the hyper-parameters shared with DrQv2 follow the values provided by Yarats et al. (2021a), while the uncertainty regularizer and optimizer for β follow the same specifications as in GPL-SAC. Additionally, GPL-DRQ-Expl+ linearly decays the optimistic shift value λopt from 0.5 down to 0.0 with the same frequency as the exploration noise’s standard deviation.
Baseline results. For the experiments on the DeepMind Control Suite from pixel observations, we compare our extensions with the base DrQv2 algorithm. Since the provided results for DrQv2 had inconsistent numbers of repetitions, we recollected the results by running the experiments with Yarats et al. (2021a)’s original implementation. However, in our evaluation, we observed higher variances in the performance for some of the considered tasks than what Yarats et al. (2021a) reported. We shared this inconsistency with DrQv2’s authors, and they confirmed the validity of our empirical findings after recollecting the results themselves.

heading: C MAIN PERFORMANCE RESULTS

heading: C.1 OPENAI GYM RESULTS AFTER 100K EXPERIENCE STEPS
text: In Table 4 we compare the performances of the examined algorithms after the milestone of collecting 100K environment steps. We believe this to be an appropriate comparison point for recent algorithms, given their relative sample-efficiency improvements in the OpenAI Gym tasks. For each run, we record the average returns from 25 evaluation episodes collected in the preceding 5000 environment steps. These results highlight once again the superior sample efficiency of GPL-SAC, pushing the performance boundary of off-policy methods. GPL-SAC is also more consistent than the considered baselines in most tasks, as shown by the lower relative standard deviations. As mentioned in Section 6 of the main text, we attribute the performance gap mainly to the ability of GPL to prevent the accumulation of overestimation bias without requiring overly pessimistic targets. In particular, the principal identified downsides of overly pessimistic targets are two-fold. Firstly, they slow down reward propagation and introduce further errors in the TD-targets, potentially leading to suboptimal convergence of the relative action-value functions (Kumar et al., 2020). Secondly, they
induce overly conservative policies that prefer low-uncertainty behavior, hindering exploration in stochastic environments (Ciosek et al., 2019).

heading: C.2 DEEPMIND CONTROL SUITE PERFORMANCE CURVES
text: In Figure 6 we provide the performance curves for the pixel-based medium benchmark tasks from the DeepMind Control Suite, as specified by Yarats et al. (2021a). These visualizations complement the results summary provided in Section 6 and further highlight the effectiveness of GPL and the proposed pessimism annealing procedure. In accordance with our earlier analysis, our methods yield improved performance and robustness in the harder exploration environments, where overly pessimistic action-value targets are expectedly more detrimental.

heading: D EXTENDED EMPIRICAL ANALYSIS
text: We provide an extended empirical analysis of Generalized Pessimism Learning. In particular, we study the effects of different components and parameters on the stability and efficiency of our GPLSAC algorithm. We focus our analysis on a representative subset of the considered OpenAI Gym tasks. We follow the same experimental protocol as described in Section 6 of the main text.

heading: D.1 UNCERTAINTY REGULARIZER ANALYSIS
text: In all main experiments, we initialize the uncertainty regularizer with β = 0.5. This setting yields the same initial expected regularization magnitude as the ubiquitous clipped double Q-learning method. Hence, to better understand the optimization properties of GPL, we consider instantiating GPLSAC with three different initial values for β, namely, β = 0.25, β = 0.5, and β = 0.75. For these settings, we record both the evolution of β throughout learning and the performance of the resulting algorithms. This analysis aims to evaluate the sensitivity of GPL to the initial value of β and the effectiveness of the dual TD-learning procedure. In particular, this simple experimental setting should elucidate some of the properties of the actual implementation of GPL, where the initial unbiasedness assumption used to motivate dual TD-learning does not necessarily hold. Moreover, the learned values of β should provide insights regarding the bias arising from interactions of offpolicy reinforcement learning methods and the different examined tasks.
Parameter evolution. In Figure 7 (Top), we show the value of β collected throughout training in each experiment with the examined initial settings. For each task, β appears to follow a recognizable trend of convergence towards some particular distinct range of values. This range appears to be influenced by the environment’s complexity, with β converging to lower values for Walker2d and increasingly higher values for Ant and Humanoid, respectively. However, β appears to adapt rather slowly, seldom reaching stability for distant initialization values in the examined experience regime. Our intuition is that this phenomenon is mainly due to two characteristics of pessimism learning. The first characteristic is that bias in the target action-value predictions can arise simply due to the stochasticity of the RL process dynamics for any value of β. Hence, there will always be a stochastic component from the dual-TD learning signal introducing noise to β’s optimization. The second characteristic is that, in practice, dual TD-learning occurs at a slower rate than TD-learning itself. Hence, as discussed in Section 5, given an imperfect initialization of β, part of the target bias might leak into the online action-value models in the first iterations of training. While the iterative
interactions between dual TD-learning and TD-learning still pushes the value of β to mitigate bias, consistently with our inituition, training takes longer to reach expectedly unbiased targets from a dimmed-down signal in the ‘unbiasing’ learning direction.
Performance results. In Figure 7 (Bottom), we show the performance curves for the examined initial settings. While GPL-SAC recovers strong performance for most tested initializations, we can see robustness improve when we initialize β closer to its convergence range. Specifically, β = 0.75 appears to work best for Humanoid, β = 0.5 for Ant, and β = 0.25 for Walker. These results provide further evidence that one of the main factors determining optimal pessimism is task complexity, with harder tasks requiring more conservative targets for better stabilization. Thus, these results further highlight the importance of an adaptive strategy for off-policy RL to achieve proper bias counteraction in arbitrary environments.

heading: D.2 ALTERNATIVE UNCERTAINTY REGULARIZER OPTIMIZATIONS
text: To evaluate the relative empirical effectiveness of dual TD-learning, we implement and test two alternative optimization strategies for the uncertainty regularizer pβ . In particular, we replace the dual TD-learning procedure in GPL-SAC with each strategy and compare the resulting performances.
GPL-SAC-E2E. First, we consider learning β to minimize end-to-end the squared norm of the expected target action-value bias approximation from Eqn. 10. This optimization strategy could be seen as a more direct approach than dual TD-learning, resulting in the following optimization objective:
argmin β
Es,a,s [ B̂(s, a, s′)2 ] , where
B̂(s, a, s′) = r + γEa′∼π(s′) [ Q̄πϕ′(s ′, a′)− pβ(s′, a′, ϕ, θ) ] −Qπϕ(s, a).
(24)
In practice, we optimize this alternative objective in place of dual TD-learning with standard gradient descent. We name the resulting algorithm GPL-SAC-E2E.
GPL-SAC-TOP. We also consider the alternative optimization procedure from the Tactical Optimism and Pessimism (TOP) algorithm (Moskovitz et al., 2021). In particular, this optimization involves learning an adaptive binary controller that switches on or off a bias correction penalty based on the sample standard deviation of the critic’s action-value estimates. The TOP algorithm learns this controller as a multi-armed bandit problem, using the difference in consecutive episodic returns as feedback. We transpose this framework to GPL by optimizing β over a discrete choice of two possible values, β ∈ {0, √ 2}. With these values, the uncertainty regularizer yields the same expected penalization as the optimistic and pessimistic modes of TOP, respectively. TOP’s authors selected these penalization levels from a vast choice of settings evaluated on the OpenAI Gym tasks. We name the resulting algorithm GPL-SAC-TOP.
Results. In Figure 8 we show the performance curves for the evaluated alternative optimization procedures as compared to the original dual TD-learning in GPL-SAC:
GPL-SAC-E2E optimizes the uncertainty regularizer too aggressively, leading to instabilities and suboptimal performance across all tasks. We motivate these results based on two related inconsistencies of end-to-end bias minimization from Eqn. 24. Firstly, this alternative optimization does not consider the effects of the target action-value bias on the changes in the action distribution used for bootstrapping. In particular, modifying the uncertainty regularizer will affect the objective of the policy’s optimization from Eqn. 4 and the corresponding distribution of on-policy actions. Secondly, even considering a fixed policy, this end-to-end strategy assumes that β linearly affects the bias. However, this is not the case due to the non-linear compounding effects from the bootstrapping in the TD-recursions. Hence, as empirically confirmed by the superior performance, framing bias minimization as a dual problem is a more general and appropriate formulation.
On the other hand, GPL-SAC-TOP matches the performance of dual TD-learning on the Humanoid task but severely underperforms on the other two tasks. For Humanoid, TOP’s optimization strategy quickly converges to sampling β = √ 2 consistently, as β = 0 causes considerable instabilities. As observed in Appendix D.1, this task benefits from highly penalized targets for stabilization, making this early convergence to a high β effective. However, both Walker and Ant tasks benefit from moderate levels of pessimism, which the bandit optimization strategy fails to recover in the examined experience regime. This inefficiency stems from the fact that immediate episodic return improvement does not appear to correlate with overall learning efficiency for these tasks. Moreover, the slow update frequency and noisy nature of the controller feedback make GPL-SAC-TOP unable to dynamically address arising biases. These results further show the effectiveness of dual TD-learning to directly minimize a source of learning instability and enable consistently efficient learning.

heading: D.3 PESSIMISM ANNEALING FOR OPENAI GYM
text: We experiment with applying pessimism annealing to GPL-SAC for the OpenAI Gym tasks. In the same spirit as our previously described application for pixel observations tasks, we linearly decay λopt from 0.5 to 0.0 in the first 50000 environment steps. We leave the rest of the hyper-parameters unaltered. We name the resulting algorithm GPL-SAC-Expl+.
Results. In Figure 9 we provide the performance curves comparing GPL-SAC with GPL-SACExpl+. The shifted uncertainty regularizer marginally improves performance in only two out of three environments. These results indicate that the undirected Gaussian exploration from SAC combined with the unbiased targets from GPL are already sufficient to ensure effective exploration in the OpenAI Gym tasks. In contrast, pessimism annealing appears to have a more significant effect in our results for the DeepMind Control Suite from Section 6 of the main text, highlighting the harder exploration challenge introduced by these pixel-based tasks.

heading: D.4 ENSEMBLE SIZE
text: In the ablation study from Section 6 of the main text, the increased number of action-value models in the critic’s ensemble appeared to have the greatest overall effect on GPL-SAC’s performance. This parameter directly influences the ability to capture the critic’s epistemic uncertainty, thus, affecting both the accuracy of the action-value predictions and the variance in the Wasserstein distance
estimation for the uncertainty regularizer penalty. Therefore, we extend our empirical analysis by testing GPL-SAC with broader numbers of action-value models to understand the relevance of this parameter and the scalability of our algorithm. Namely, we compare instances of GPL-SAC with N = 2, N = 5, N = 10, N = 15, and N = 20 action-value models.
Results. In Figure 10 we provide the performance curves for the different ensemble sizes. Predictably, both learning efficiency and final performance monotonically improve with the ensemble size. Yet, these improvements appear to saturate at different values of N based on the complexity of the underlying environment, showing how harder tasks increasingly rely on accurately representing epistemic uncertainty. In particular, for Walker2d training with only N = 2 action-value models yields very similar results to training with N = 20 action-value models. However, the performance differences are increasingly noticeable for the other tasks that involve significantly larger state and action spaces. For instance, for Humanoid the agent is not able to recover meaningful behavior with N = 2 and performance saturates only around training with N = 10.

heading: D.5 UPDATE-TO-DATA RATIO
text: Following the examined prior state-of-the-art algorithms, GPL-SAC uses a critic UTD ratio of 20 across the different OpenAI Gym tasks. This aggressive optimization frequency ensures that the critic encapsulates most information from experience collected at any given point in the RL process. However, since the UTD ratio affects training time almost linearly, we examine its impact on performance. In particular, we compare instances of GPL-SAC with UTD ratios of 1, 5, 10, and 20.
Results. In Figure 11 we provide the performance curves for the different examined UTD ratios. Larger UTD ratios yield clear sample efficiency improvements for all tasks. Moreover, using a UTD ratio of 1 in the Ant and Humanoid environments appears to cause some learning instabilities towards the end of the examined experience regime. These instabilities are likely from the inability of the learning process to quickly incorporate new information from recently collected data into the critic, given the growing size of the replay buffer. Consequently, a slow learning process might produce
inaccurate return predictions for on-policy behavior, leading to a noisy policy gradient signal and, therefore, a degradation of the policy itself.

heading: D.6 DISTRIBUTIONAL CRITIC
text: The uncertainty regularizer penalty is a function of the expected Wasserstein distance between the return distributions predicted by the critic. Hence, it has a natural generalization to more expressive distributional critics beyond ensembles of action-value functions. Therefore, we evaluate the effectiveness of extending GPL-SAC by parameterizing the critic using an ensemble of N distributional models {Zπϕi} N i=1. Following Dabney et al. (2018), we let each model output M quantile locations values parameterizing a quantile distribution. We consider both 1-Wasserstein and 2-Wasserstein distances for the uncertainty regularizer (which for Dirac delta functions are equivalent). For quantile distributions, these distance measures are calculated as follows:
W1(Z π ϕi(s, a), Z π ϕj (s, a)) = M∑ k=1 1 M |Zπϕi(s, a)k − Z π ϕj (s, a)k|,
W2(Z π ϕi(s, a), Z π ϕj (s, a)) = √√√√ M∑ k=1 1 M ( Zπϕi(s, a)k − Z π ϕj (s, a)k )2 .
(25)
We propose applying the uncertainty regularizer to each quantile location to compute the target return distribution, Ẑπϕ (s, a) ∈ RM , as:
Ẑπϕ (s, a)j = Z π ϕ′(s, a)j − pβ(s′, a′, ϕ, θ), for all j ∈ {1, 2 . . .M}. (26)
We optimize the critic’s models with a distributional version of TD-learning (Bellemare et al., 2017) making use of Huber quantile regression (Koenker & Hallock, 2001). We still optimize the policy to maximize the expected returns by averaging over the quantile locations of the target return distribution. Moreover, we still perform dual TD-learning with minimal overheads by averaging all quantile errors computed for quantile regression during the distributional TD-learning updates. In our implementation, we approximate the return distribution with M = 10 quantile locations and set the Huber quantile regression coefficient to κ = 1. We keep all other hyper-parameters and model architectures consistent with GPL-SAC. We name the resulting algorithms GPL-QR-SAC (W1/W2).
Results. In Figure 12 we provide the performance curves comparing both versions of GPL-QR-SAC with the original GPL-SAC algorithm. GPL-QR-SAC (W1) appears to outperform GPL-QR-SAC (W2), indicating that the 1-Wasserstein distance is a more effective and stable measure of epistemic uncertainty for pβ . However, in the examined tasks, GPL-QR-SAC (W1) performs similarly to GPL-SAC, meaning there are no major benefits in parameterizing the critic with more expressive approximations to the return distribution. Yet, these results could also indicate that the returns for the considered OpenAI Gym environments have low stochasticity or that the action-value models already encapsulate all learnable information in the considered training regime. We leave further experimentation and analysis of using GPL with distributional critics for future work.

heading: D.7 COMPUTATIONAL SCALING
text: We analyze the computational scaling of our implementations of Generalized Pessimism Learning-based algorithms. Particularly, we record the average training time of executing the different considered algorithms for either 1000 environment steps for OpenAI Gym tasks or 10000 environment steps for DeepMind Control Suite tasks. We run each algorithm on an NVIDIA RTX 3090 GPU and an AMD Ryzen Threadripper 3970x CPU. As described in Section 6 of the main text, our implementation groups the parameters of the different models in the critic’s ensemble into a single network to achieve better scaling for inference and backpropagation with distributed hardware.
Training times. We report all average training times in Table 5. Comparing the training times with and without dual TD-learning and pessimism annealing clearly shows that both procedures introduce trivial computational overheads. Since the critic’s updates represent the bulk of the computation in off-policy RL, modifying the associated hyper-parameters and parameterizations has relevant effects on training times. In particular, using a classical action-value model architecture speeds up training time by approximately 30%, while using a distributional critic slows down training by approximately 20%. Thanks to our implementation and the efficiency of large tensor operations, increasing the number of action-value models affects training times sub-linearly. For instance, doubling the critic’s ensemble size to 20 results in less than 12% additional training overhead. On the other hand, increasing the UTD ratio increases the total training time almost linearly. As compared to the original REDQ implementation from Chen et al. (2021), GPL-SAC trains in less than half the time while having similar algorithmic complexity.
Considerations. The results from this Section can provide intuitions on selecting components and hyper-parameters when applying our implementation of Generalized Pessimism Learning for different problems. For instance, increasing ensemble size beyond N = 10 appears to affect training times only marginally. Thus, indicating that a viable criterion for selecting N in practical scenarios could be based on how many models fit into GPU memory. Multiple directions can be explored to further improve the time-efficiency and scalability of GPL. Since our uncertainty regularizer is compatible with variational representations of the critic’s parameters posterior, Bayesian parameterizations could be a viable option to capture epistemic uncertainty more efficiently than model ensembles. Moreover, recent work by Björck et al. (2021) shows that it is possible to train modern reinforcement learning algorithms in half precision, with non-trivial computation and memory benefits.

Review summaries:
This paper addresses the issue of  overestimation bias in TD learning by deriving a Generalized Pessimism Learning (GPL) framework. Authors have used a dual TD learning procedure to minimise the estimated bias in critic’s performance and adaptively learns a penalty to recover an unbiased performance objective. The idea has been integrated into SAC and DrQ on various tasks. 

This paper introduces GPL, an off-policy RL algorithm that aims to address the overestimation bias in Q learning. The authors propose to estimate the bias in Q learning by uncertainty quantification as a penalization to estimated Q functions. The authors also propose to learn the appropriate penalty through minimizing the bias. The authors further propose GPL, which penalizes the estimated Q functions based on uncertainty quantification. In addition, the authors propose to adjust the penalization parameter to conduct exploration. The authors conducted experiments on GPL incorporated with SAC and DrQ. Experimentations demonstrate better performance than the SAC and DrQ baselines.

The paper is motivated by the over/under-estimation bias in TD-learning for deep RL. The paper proposes to dynamically adjust the pessimism level of the learning rule, by adapting a multiplier parameter using dual gradient update. The paper provides some semi-theoretic motivations and empirical performance gains.