Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

1 INTRODUCTION . Sample efficiency and generality are two directions in which reinforcement learning ( RL ) algorithms are still lacking , yet , they are crucial for tackling complex real-world problems ( Mahmood et al. , 2018 ) . Consequently , so far , many RL milestones have been achieved through simulating conspicuous amounts of experience and tuning for effective task-specific parameters ( Mnih et al. , 2013 ; Silver et al. , 2017 ) . Recent off-policy model-free ( Lee et al. , 2021 ; Chen et al. , 2021 ) and modelbased algorithms ( Chua et al. , 2018 ; Janner et al. , 2019 ) , pushed forward the state-of-the-art sampleefficiency on several benchmark tasks ( Brockman et al. , 2016 ) . We attribute such improvements to two main linked advances : more expressive models to capture uncertainty and better strategies to counteract detrimental biases from the learning process . These advances yielded the stabilization to adopt more aggressive optimization procedures , with particular benefits in lower data regimes . Modern off-policy algorithms learn behavior by optimizing the expected performance as predicted by a trained parametric deep model of the environment . Within this process , overestimation bias naturally arises from the maximization performed over the model ’ s performance predictions , and consequently , also over the model ’ s possible errors . In the context of model-free RL , such model is trained to predict the agent ’ s future returns via temporal difference ( TD- ) learning and is referred to as the critic . A common strategy to counteract overestimation is to parameterize the critic with multiple , independently-initialized networks and optimize the agent ’ s behavior over the minimum value of the relative outputs ( Fujimoto et al. , 2018 ) . Empirically , this strategy consistently yields pessimistic target performance measures , avoiding overestimation bias propagating through the TDlearning target bootstraps . However , this approach directly links the critic ’ s parameterization to bias counteraction , making improvements in each of these components hard to pursue independently . Based on these observations , we propose a more general formulation for counteracting overestimation bias independently . In particular , we compute the critic ’ s target performance predictions by replacing the ubiquitous minimization procedure with an explicit penalty that is agnostic to the critic ’ s parameterization . Our proposed penalty is the output of a linear model of the epistemic uncertainty , computed as the expected Wasserstein distance between the return distributions predicted by the critic . Based on this formulation , we derive Generalized Pessimism Learning ( GPL ) , a new strategy that learns an accurate penalization throughout the RL process . Within this strategy , we propose optimizing the penalty ’ s weight with dual TD-learning , a new procedure minimizing the estimated bias in the critic ’ s performance predictions with dual gradient descent . GPL is the first effective method able to freely learn an unbiased performance objective throughout training . Furthermore , we also extend GPL by introducing a new pessimism annealing procedure , motivated by the principle of optimism in the face of uncertainty ( Brafman & Tennenholtz , 2002 ) . This procedure leads the agent to adopt a risk-seeking behavior policy by utilizing a purposely biased estimate of the performance in the initial training stages . Hence , it trades-off expected immediate performance for directed exploration , incentivizing the visitation of states with high epistemic uncertainty from which the critic would gain more information . We incorporate GPL with modern implementations of the Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018a ; b ) and Data-regularized Q ( DrQ ) ( Yarats et al. , 2021b ; a ) algorithms , yielding GPL-SAC and GPL-DrQ , respectively . On the Mujoco environments from the OpenAI Gym suite ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) , GPL-SAC outperforms prior , more expensive , model-based ( Janner et al. , 2019 ) and model-free ( Chen et al. , 2021 ) state-of-the-art algorithms . For instance , in the Humanoid environment GPL-SAC is able to recover a score of 5000 in less than 100K experience steps , more than nine times faster than regular SAC . Additionally , on pixel-based environments from the DeepMind Control Suite ( Tassa et al. , 2018 ) , GPL-DrQ provides significant performance improvements from the recent state-of-the-art DrQv2 algorithm . These results highlight the effectiveness and applicability of GPL , in spite of introducing only negligible computational overheads . We release our implementations to facilitate future comparisons and extensions . In summary , we make several contributions towards improving off-policy reinforcement learning : • We propose a novel penalty to counteract overestimation bias , disentangling the critic ’ s parameterization from the enforced pessimism . • We propose the first optimization method to estimate and precisely counteract overestimation bias throughout training with dual gradient descent . • We propose a pessimism annealing strategy that exploits epistemic uncertainty to actively seek informative states in the early training stages . • Integrating our method with SAC and DrQ , we achieve new state-of-the-art performance results with trivial overheads on both proprioceptive and pixel observations tasks . 2 RELATED WORK . Modern model-free off-policy algorithms utilize different strategies to counteract overestimation bias arising in the critic ’ s TD-targets ( Thrun & Schwartz , 1993 ; Pendrith et al. , 1997 ; Mannor et al. , 2007 ) . Many approaches combine the predictions of multiple function approximators to estimate the expected returns , for instance , by independently selecting the bootstrap action ( Hasselt , 2010 ) . In discrete control , such technique appears to mitigate the bias of the seminal DQN algorithm ( Mnih et al. , 2013 ) , consistently improving performance ( Van Hasselt et al. , 2016 ; Hessel et al. , 2018 ) . In continuous control , similar strategies successfully stabilize algorithms based on the policy gradient theorem ( Silver et al. , 2014 ; Lillicrap et al. , 2015 ) . Most notably , Fujimoto et al . ( 2018 ) proposed to compute the critic ’ s TD-targets by taking the minimum over the outputs of two different actionvalue models . This particular minimization strategy has become ubiquitous , being employed in many popular follow-up algorithms ( Haarnoja et al. , 2018b ; Yarats et al. , 2021b ) . To better trade-off optimism and pessimism , Zhang et al . ( 2017 ) proposed using a weighted combination of the original and minimized targets . Instead , Kuznetsov et al . ( 2020 ) proposed to parameterize a distributional critic and drop a fixed fraction of the predicted quantiles to compute the targets . Moreover , as in our approach , several works also considered explicit penalties based on heuristic measures of epistemic uncertainty ( Lee et al. , 2013 ; Ciosek et al. , 2019 ) . Recently , Kumar et al . ( 2020 ) proposed to complement these strategies by further reducing bias propagation through actively weighing the TD-loss of different experience samples . Aleatoric uncertainty is also an additional source of bias in TD-learning , due to the practical inability of considering multiple transition samples in stochastic environments ( Baird , 1995 ) . This phenomenon is known as the double-sample issue , but has been rarely addressed in prior off-policy literature ( Dai et al. , 2018 ) . Within model-based RL ( Atkeson & Santamaria , 1997 ) , recent works have achieved remarkable sample efficiency by learning large ensembles of dynamic models for better predictions ( Chua et al. , 2018 ; Wang & Ba , 2019 ; Janner et al. , 2019 ) . In the model-free framework , prior works used large critic ensembles for more diverse scopes . Anschel et al . ( 2017 ) proposed to build an ensemble using several past versions of the value network to reduce the magnitude of the TD-target ’ s bias . Moreover , Lan et al . ( 2020 ) introduced a sampling procedure for the critic ’ s ensemble predictions to regulate underestimation in the TD-targets . Their work was later extended to the continuous setting by Chen et al . ( 2021 ) , which showed that large ensembles combined with a high update-to-data ratio enable to outperform the sample efficiency of contemporary model-based methods . Ensembling has also been used to achieve better exploration following the principle of optimism in the face of uncertainty ( Brafman & Tennenholtz , 2002 ) in both discrete ( Osband et al. , 2016 ; Chen et al. , 2017 ) and continuous settings ( Ciosek et al. , 2019 ) . Lee et al . ( 2021 ) further showed the effectiveness of combining several of these strategies in a unified framework . In the same spirit as this work , multiple prior methods attempted to learn components and parameters of underlying RL algorithms . Several works have approached this problem by utilizing expensive meta-learning strategies to obtain new learning objectives based on the multi-task performance from low-computation environments ( Bechtle et al. , 2021 ; Oh et al. , 2020 ; Xu et al. , 2020 ; Co-Reyes et al. , 2021 ) . More related to our work , Moskovitz et al . ( 2021 ) recently proposed to use an adaptive binary controller that switches on or off a bias correction penalty for the TD-targets . In particular , they treat the controller optimization task as a multi-armed bandit problem performed throughout different training iterations to maximize immediate performance improvements . Instead , GPL makes effective use of dual gradient descent to minimize bias directly , similarly to how Haarnoja et al . ( 2018a ; b ) learns the exploration temperature parameter in the Soft Actor-Critic ( SAC ) algorithm . 3 PRELIMINARIES . In RL , we aim to autonomously recover optimal agent behavior for performing a particular task . Formally , we describe this problem setting as a Markov Decision Process ( MDP ) , defined as the tuple ( S , A , P , p0 , r , γ ) . At each time-step of interaction , the agent observes some state in the state space , s ∈ S , and performs some action in the action space , a ∈ A . The transition dynamics function P : S × A × S → R and the initial state distribution p0 : S → R describe the evolution of the environment as a consequence of the agent ’ s behavior . The reward function r : S × A → R quantifies the effectiveness of each performed action , while the discount factor γ ∈ [ 0 , 1 ) represents the agent ’ s preference for earlier rewards . A policy π : S ×A → R maps each state to a probability distribution over actions and represents the agent ’ s behavior . An episode of interactions between the agent and the environment yields some trajectory τ containing the transitions experienced , τ = ( s0 , a0 , r0 , s1 , a1 , r1 , ... ) . The RL objective is then to find an optimal policy π∗ that maximizes the expected sum of discounted future rewards : π∗ = argmax π Epπ ( τ ) [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) where pπ ( τ ) represents the distribution of trajectories stemming from the agent ’ s interaction with the environment . Off-policy RL algorithms commonly utilize some critic model to evaluate the effectiveness of the agent ’ s behavior . A straightforward choice for the critic is to represent the policy ’ s action-value function Qπ : S × A → R. This function quantifies the expected sum of discounted future rewards after executing some particular action from a given state : Qπ ( s , a ) = Epπ ( τ |s0=s , a0=a ) [ ∞∑ t=0 γtr ( st , at ) ] . ( 2 ) Most RL algorithms consider learning parameterized models for both the policy , πθ , and the corresponding action-value function , Qπϕ . In particular , after storing experience transitions ( s , a , s ′ , r ) in a replay data buffer D , we learn Qπϕ by iteratively minimizing a squared TD-loss of the form : JQ ( ϕ ) = E ( s , a , s′ , r ) ∼D [ ( Qπϕ ( s , a ) − y ) 2 ] , y = r + γEa∼π ( s′ ) [ Q̂πϕ′ ( s ′ , a ) ] . ( 3 ) Here , the TD-targets y are obtained by computing a 1-step bootstrap with a target action-value estimator Q̂πϕ′ . Usually , Q̂ π ϕ′ is a regularized function of action-value predictions from a target critic model using delayed parameters ϕ′ . Following the policy gradient theorem ( Sutton et al. , 2000 ; Silver et al. , 2014 ) , we can then improve our policy by maximizing the expected returns as predicted by the critic , e.g. , by minimizing the negated action-value estimates : Jπ ( θ ) = −Es∼D , a∼πθ ( s ) [ Q̂πϕ ( s , a ) ] . ( 4 )

Summary: This paper addresses the issue of  overestimation bias in TD learning by deriving a Generalized Pessimism Learning (GPL) framework. Authors have used a dual TD learning procedure to minimise the estimated bias in critic’s performance and adaptively learns a penalty to recover an unbiased performance objective. The idea has been integrated into SAC and DrQ on various tasks.