Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

1 INTRODUCTION . Sample efficiency and generality are two directions in which reinforcement learning ( RL ) algorithms are still lacking , yet , they are crucial for tackling complex real-world problems ( Mahmood et al. , 2018 ) . Consequently , so far , many RL milestones have been achieved through simulating conspicuous amounts of experience and tuning for effective task-specific parameters ( Mnih et al. , 2013 ; Silver et al. , 2017 ) . Recent off-policy model-free ( Lee et al. , 2021 ; Chen et al. , 2021 ) and modelbased algorithms ( Chua et al. , 2018 ; Janner et al. , 2019 ) , pushed forward the state-of-the-art sampleefficiency on several benchmark tasks ( Brockman et al. , 2016 ) . We attribute such improvements to two main linked advances : more expressive models to capture uncertainty and better strategies to counteract detrimental biases from the learning process . These advances yielded the stabilization to adopt more aggressive optimization procedures , with particular benefits in lower data regimes . Modern off-policy algorithms learn behavior by optimizing the expected performance as predicted by a trained parametric deep model of the environment . Within this process , overestimation bias naturally arises from the maximization performed over the model ’ s performance predictions , and consequently , also over the model ’ s possible errors . In the context of model-free RL , such model is trained to predict the agent ’ s future returns via temporal difference ( TD- ) learning and is referred to as the critic . A common strategy to counteract overestimation is to parameterize the critic with multiple , independently-initialized networks and optimize the agent ’ s behavior over the minimum value of the relative outputs ( Fujimoto et al. , 2018 ) . Empirically , this strategy consistently yields pessimistic target performance measures , avoiding overestimation bias propagating through the TDlearning target bootstraps . However , this approach directly links the critic ’ s parameterization to bias counteraction , making improvements in each of these components hard to pursue independently . Based on these observations , we propose a more general formulation for counteracting overestimation bias independently . In particular , we compute the critic ’ s target performance predictions by replacing the ubiquitous minimization procedure with an explicit penalty that is agnostic to the critic ’ s parameterization . Our proposed penalty is the output of a linear model of the epistemic uncertainty , computed as the expected Wasserstein distance between the return distributions predicted by the critic . Based on this formulation , we derive Generalized Pessimism Learning ( GPL ) , a new strategy that learns an accurate penalization throughout the RL process . Within this strategy , we propose optimizing the penalty ’ s weight with dual TD-learning , a new procedure minimizing the estimated bias in the critic ’ s performance predictions with dual gradient descent . GPL is the first effective method able to freely learn an unbiased performance objective throughout training . Furthermore , we also extend GPL by introducing a new pessimism annealing procedure , motivated by the principle of optimism in the face of uncertainty ( Brafman & Tennenholtz , 2002 ) . This procedure leads the agent to adopt a risk-seeking behavior policy by utilizing a purposely biased estimate of the performance in the initial training stages . Hence , it trades-off expected immediate performance for directed exploration , incentivizing the visitation of states with high epistemic uncertainty from which the critic would gain more information . We incorporate GPL with modern implementations of the Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018a ; b ) and Data-regularized Q ( DrQ ) ( Yarats et al. , 2021b ; a ) algorithms , yielding GPL-SAC and GPL-DrQ , respectively . On the Mujoco environments from the OpenAI Gym suite ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) , GPL-SAC outperforms prior , more expensive , model-based ( Janner et al. , 2019 ) and model-free ( Chen et al. , 2021 ) state-of-the-art algorithms . For instance , in the Humanoid environment GPL-SAC is able to recover a score of 5000 in less than 100K experience steps , more than nine times faster than regular SAC . Additionally , on pixel-based environments from the DeepMind Control Suite ( Tassa et al. , 2018 ) , GPL-DrQ provides significant performance improvements from the recent state-of-the-art DrQv2 algorithm . These results highlight the effectiveness and applicability of GPL , in spite of introducing only negligible computational overheads . We release our implementations to facilitate future comparisons and extensions . In summary , we make several contributions towards improving off-policy reinforcement learning : • We propose a novel penalty to counteract overestimation bias , disentangling the critic ’ s parameterization from the enforced pessimism . • We propose the first optimization method to estimate and precisely counteract overestimation bias throughout training with dual gradient descent . • We propose a pessimism annealing strategy that exploits epistemic uncertainty to actively seek informative states in the early training stages . • Integrating our method with SAC and DrQ , we achieve new state-of-the-art performance results with trivial overheads on both proprioceptive and pixel observations tasks . 2 RELATED WORK . Modern model-free off-policy algorithms utilize different strategies to counteract overestimation bias arising in the critic ’ s TD-targets ( Thrun & Schwartz , 1993 ; Pendrith et al. , 1997 ; Mannor et al. , 2007 ) . Many approaches combine the predictions of multiple function approximators to estimate the expected returns , for instance , by independently selecting the bootstrap action ( Hasselt , 2010 ) . In discrete control , such technique appears to mitigate the bias of the seminal DQN algorithm ( Mnih et al. , 2013 ) , consistently improving performance ( Van Hasselt et al. , 2016 ; Hessel et al. , 2018 ) . In continuous control , similar strategies successfully stabilize algorithms based on the policy gradient theorem ( Silver et al. , 2014 ; Lillicrap et al. , 2015 ) . Most notably , Fujimoto et al . ( 2018 ) proposed to compute the critic ’ s TD-targets by taking the minimum over the outputs of two different actionvalue models . This particular minimization strategy has become ubiquitous , being employed in many popular follow-up algorithms ( Haarnoja et al. , 2018b ; Yarats et al. , 2021b ) . To better trade-off optimism and pessimism , Zhang et al . ( 2017 ) proposed using a weighted combination of the original and minimized targets . Instead , Kuznetsov et al . ( 2020 ) proposed to parameterize a distributional critic and drop a fixed fraction of the predicted quantiles to compute the targets . Moreover , as in our approach , several works also considered explicit penalties based on heuristic measures of epistemic uncertainty ( Lee et al. , 2013 ; Ciosek et al. , 2019 ) . Recently , Kumar et al . ( 2020 ) proposed to complement these strategies by further reducing bias propagation through actively weighing the TD-loss of different experience samples . Aleatoric uncertainty is also an additional source of bias in TD-learning , due to the practical inability of considering multiple transition samples in stochastic environments ( Baird , 1995 ) . This phenomenon is known as the double-sample issue , but has been rarely addressed in prior off-policy literature ( Dai et al. , 2018 ) . Within model-based RL ( Atkeson & Santamaria , 1997 ) , recent works have achieved remarkable sample efficiency by learning large ensembles of dynamic models for better predictions ( Chua et al. , 2018 ; Wang & Ba , 2019 ; Janner et al. , 2019 ) . In the model-free framework , prior works used large critic ensembles for more diverse scopes . Anschel et al . ( 2017 ) proposed to build an ensemble using several past versions of the value network to reduce the magnitude of the TD-target ’ s bias . Moreover , Lan et al . ( 2020 ) introduced a sampling procedure for the critic ’ s ensemble predictions to regulate underestimation in the TD-targets . Their work was later extended to the continuous setting by Chen et al . ( 2021 ) , which showed that large ensembles combined with a high update-to-data ratio enable to outperform the sample efficiency of contemporary model-based methods . Ensembling has also been used to achieve better exploration following the principle of optimism in the face of uncertainty ( Brafman & Tennenholtz , 2002 ) in both discrete ( Osband et al. , 2016 ; Chen et al. , 2017 ) and continuous settings ( Ciosek et al. , 2019 ) . Lee et al . ( 2021 ) further showed the effectiveness of combining several of these strategies in a unified framework . In the same spirit as this work , multiple prior methods attempted to learn components and parameters of underlying RL algorithms . Several works have approached this problem by utilizing expensive meta-learning strategies to obtain new learning objectives based on the multi-task performance from low-computation environments ( Bechtle et al. , 2021 ; Oh et al. , 2020 ; Xu et al. , 2020 ; Co-Reyes et al. , 2021 ) . More related to our work , Moskovitz et al . ( 2021 ) recently proposed to use an adaptive binary controller that switches on or off a bias correction penalty for the TD-targets . In particular , they treat the controller optimization task as a multi-armed bandit problem performed throughout different training iterations to maximize immediate performance improvements . Instead , GPL makes effective use of dual gradient descent to minimize bias directly , similarly to how Haarnoja et al . ( 2018a ; b ) learns the exploration temperature parameter in the Soft Actor-Critic ( SAC ) algorithm . 3 PRELIMINARIES . In RL , we aim to autonomously recover optimal agent behavior for performing a particular task . Formally , we describe this problem setting as a Markov Decision Process ( MDP ) , defined as the tuple ( S , A , P , p0 , r , γ ) . At each time-step of interaction , the agent observes some state in the state space , s ∈ S , and performs some action in the action space , a ∈ A . The transition dynamics function P : S × A × S → R and the initial state distribution p0 : S → R describe the evolution of the environment as a consequence of the agent ’ s behavior . The reward function r : S × A → R quantifies the effectiveness of each performed action , while the discount factor γ ∈ [ 0 , 1 ) represents the agent ’ s preference for earlier rewards . A policy π : S ×A → R maps each state to a probability distribution over actions and represents the agent ’ s behavior . An episode of interactions between the agent and the environment yields some trajectory τ containing the transitions experienced , τ = ( s0 , a0 , r0 , s1 , a1 , r1 , ... ) . The RL objective is then to find an optimal policy π∗ that maximizes the expected sum of discounted future rewards : π∗ = argmax π Epπ ( τ ) [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) where pπ ( τ ) represents the distribution of trajectories stemming from the agent ’ s interaction with the environment . Off-policy RL algorithms commonly utilize some critic model to evaluate the effectiveness of the agent ’ s behavior . A straightforward choice for the critic is to represent the policy ’ s action-value function Qπ : S × A → R. This function quantifies the expected sum of discounted future rewards after executing some particular action from a given state : Qπ ( s , a ) = Epπ ( τ |s0=s , a0=a ) [ ∞∑ t=0 γtr ( st , at ) ] . ( 2 ) Most RL algorithms consider learning parameterized models for both the policy , πθ , and the corresponding action-value function , Qπϕ . In particular , after storing experience transitions ( s , a , s ′ , r ) in a replay data buffer D , we learn Qπϕ by iteratively minimizing a squared TD-loss of the form : JQ ( ϕ ) = E ( s , a , s′ , r ) ∼D [ ( Qπϕ ( s , a ) − y ) 2 ] , y = r + γEa∼π ( s′ ) [ Q̂πϕ′ ( s ′ , a ) ] . ( 3 ) Here , the TD-targets y are obtained by computing a 1-step bootstrap with a target action-value estimator Q̂πϕ′ . Usually , Q̂ π ϕ′ is a regularized function of action-value predictions from a target critic model using delayed parameters ϕ′ . Following the policy gradient theorem ( Sutton et al. , 2000 ; Silver et al. , 2014 ) , we can then improve our policy by maximizing the expected returns as predicted by the critic , e.g. , by minimizing the negated action-value estimates : Jπ ( θ ) = −Es∼D , a∼πθ ( s ) [ Q̂πϕ ( s , a ) ] . ( 4 )

Summary: This paper introduces GPL, an off-policy RL algorithm that aims to address the overestimation bias in Q learning. The authors propose to estimate the bias in Q learning by uncertainty quantification as a penalization to estimated Q functions. The authors also propose to learn the appropriate penalty through minimizing the bias. The authors further propose GPL, which penalizes the estimated Q functions based on uncertainty quantification. In addition, the authors propose to adjust the penalization parameter to conduct exploration. The authors conducted experiments on GPL incorporated with SAC and DrQ. Experimentations demonstrate better performance than the SAC and DrQ baselines.