Tactical Optimism and Pessimism for Deep Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) has begun to show significant empirical success in recent years , with value function approximation via deep neural networks playing a fundamental role in this success [ 37 , 49 , 5 ] . However , this success has been achieved in a relatively narrow set of problem domains , and an emerging set of challenges arises when one considers placing RL systems in larger systems . In particular , the use of function approximators can lead to a positive bias in value computation [ 53 ] , and therefore systems that surround the learner do not receive an honest assessment of that value . One can attempt to turn this vice into a virtue , by appealing to a general form of the optimism-underuncertainty principle—overestimation of the expected reward can trigger exploration of states and actions that would otherwise not be explored . Such exploration can be dangerous , however , if there is not a clear understanding of the nature of the overestimation . ⇤Work mostly completed at the Gatsby Unit . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This tension has not been resolved in the recent literature on RL approaches to continuous-control problems . On the one hand , some authors seek to correct the overestimation , for example by using the minimum of two value estimates as a form of approximate lower bound [ 20 ] . This approach can be seen as a form of pessimism with respect to the current value function . On the other hand , [ 14 ] have argued that the inherent optimism of approximate value estimates is actually useful for encouraging exploration of the environment and/or action space . Interestingly , both sides have used their respective positions to derive state-of-the-art algorithms . How can this be , if their views are seemingly opposed ? Our key hypothesis is the following : The degree of estimation bias , and subsequent efficacy of an optimistic strategy , varies as a function of the environment , the stage of optimization , and the overall context in which a learner is embedded . This hypothesis motivates us to view optimism/pessimism as a spectrum and to investigate procedures that actively move along that spectrum during the learning process . We operationalize this idea by measuring two forms of uncertainty that arise during learning : aleatoric uncertainty and epistemic uncertainty . These notions of uncertainty , and their measurement , are discussed in detail in Section 5.1 . We then further aim to control the effects of these two kinds of uncertainty , making the following learning-theoretic assertion : When the level of bias is unknown , an adaptive strategy can be highly effective . In this work , we investigate these hypotheses via the development of a new framework for value estimation in deep RL that we refer to as Tactical Optimism and Pessimism ( TOP ) . This approach acknowledges the inherent uncertainty in the level of estimation bias present , and rather than adopt a blanket optimistic or pessimistic strategy , it estimates the optimal approach on the fly , by formulating the optimism/pessimism dilemma as a multi-armed bandit problem . Furthermore , TOP explicitly isolates the aleatoric and epistemic uncertainty by representing the environmental return using a distributional critic and model uncertainty with an ensemble . The overall concept is summarized in Figure 1 . We show in a series of experiments that not only does the efficacy of optimism indeed vary as we suggest , but TOP is able to capture the best of both worlds , achieving a new state of the art for challenging continuous control problems . Our main contributions are as follows : • Our work shows that the efficacy of optimism for a fixed function approximator varies across environments and during training for reinforcement learning with function approximation . • We propose a novel framework for value estimation , Tactical Optimism and Pessimism ( TOP ) , which learns to balance optimistic and pessimistic value estimation online . TOP frames the choice of the degree of optimism or pessimism as a multi-armed bandit problem . • Our experiments demonstrate that these insights , which require only simple changes to popular algorithms , lead to state-of-the-art results on both state- and pixel-based control . 2 Related Work Much of the recent success of off-policy actorcritic algorithms build on DDPG [ 34 ] , which extended the deterministic policy gradient [ 49 ] approach to off-policy learning with deep networks , using insights from DQN [ 37 ] . Like D4PG [ 8 ] , we combine DPG with distributional value estimation . However , unlike D4PG , we use two critics , a quantile representation rather than a categorical distribution [ 10 ] , and , critically , we actively manage the tradeoff between optimism and pessimism . We also note several other success stories in the actor-critic vein , including TD3 , SAC , DrQ , and PI-SAC [ 20 , 21 , 58 , 33 ] ; these represent the state-of-theart for continuous control and will serve as a baseline for our experiments . The principle of optimism in the face of uncertainty [ 3 , 30 , 59 ] provides a design tool for algorithms that trade off exploitation ( maximization of the reward ) against the need to explore state-action pairs with high epistemic uncertainty . The theoretical tool for evaluating the success of such designs is the notion of regret , which captures the loss incurred by failing to explore . Regret bounds have long been used in research on multi-armed bandits , and they have begun to become more prominent in RL as well , both in the tabular setting [ 26 , 18 , 19 , 4 , 9 , 55 ] , and in the setting of function approximation [ 28 , 57 ] . However , optimistic approaches have had limited empirical success when combined with deep neural networks in RL [ 14 ] . To be successful , these approaches need to be optimistic enough to upper bound the true value function while maintaining low estimation error [ 41 ] . This becomes challenging when using function approximation , and the result is often an uncontrolled , undesirable overestimation bias . Recently , there has been increasing evidence in support of the efficacy of adaptive algorithms [ 6 , 48 , 46 , 45 ] . An example is Agent57 [ 5 ] , the first agent to outperform the human baseline for all 57 games in the Arcade Learning Environment [ 11 ] . Agent57 adaptively switches among different exploration strategies . Our approach differs in that it aims to achieve a similar goal by actively varying the level of optimism in its value estimates . Finally , our work is also related to automated RL ( AutoRL ) , as we can consider TOP to be an example of an on-the-fly learning procedure [ 15 , 39 , 29 ] . An exciting area of future work will be to consider the interplay between the degree of optimism and model hyperparameters such as architecture and learning rate , and whether they can be adapted simultaneously . 3 Preliminaries . Reinforcement learning considers the problem of training an agent to interact with its environment so as to maximize its cumulative reward . Typically , a task and environment are cast as a Markov decision process ( MDP ) , formally defined as a tuple ( S , A , p , r , ) , where S is the state space , A is the space of possible actions , p : S ⇥A ! P ( S ) is a transition kernel , r : S ⇥A ! R is the reward function , and 2 [ 0 , 1 ) is a discounting factor . For a given policy ⇡ , the return Z⇡ = P t trt , is a random variable representing the sum of discounted rewards observed along one trajectory of states obtained from following ⇡ until some time horizon T , potentially infinite . Given a parameterization of the set of policies , { ⇡✓ : ✓ 2 ⇥ } , the goal is to update ✓ so as to maximize the expected return , or discounted cumulative reward , J ( ✓ ) = E⇡ [ P t trt ] = E [ Z⇡ ] . Actor-critic algorithms are a framework for solving this problem in which the policy ⇡ , here known as the actor , is trained to maximize expected return , while making use of a critic that evaluates the actions of the policy . Typically , the critic takes the form of a value function which predicts the expected return under the current policy , Q⇡ ( s , a ) : = E⇡ [ Zt|st = s , at = a ] . When the state space is large , Q⇡ may be parameterized by a model with parameters . The deterministic policy gradient ( DPG ) theorem [ 50 ] shows that gradient ascent on J can be performed via r✓J ( ✓ ) = E⇡ [ raQ⇡ ( s , a ) |a=⇡ ( s ) r✓⇡✓ ( s ) ] . ( 1 ) The critic is updated separately , usually via SARSA [ 51 ] , which , given a transition st , at ! rt+1 , st+1 , forms a learning signal via semi-gradient descent on the squared temporal difference ( TD ) error , 2t , where t : = yt Q⇡ ( st , at ) = rt+1 + Q⇡ ( st+1 , ⇡ ( st+1 ) ) Q⇡ ( st , at ) , ( 2 ) and where yt is the Bellman target . Rather than simply predicting the mean of the return Z⇡ under the current policy , it can be advantageous to learn a full distribution of Z⇡ given the current state and action , Z⇡ ( st , at ) [ 10 , 17 , 16 , 47 ] . In this framework , the return distribution is typically parameterized via a set of K functionals of the distribution ( e.g. , quantiles or expectiles ) which are learned via minimization of an appropriate loss function . For example , the kth quantile of the distribution at state s and associated with action a , qk ( s , a ) , can be learned via gradient descent on the Huber loss [ 25 ] of the distributional Bellman error , k = Ẑ qk ( s , a ) , for Ẑ ⇠ Z⇡ ( ·|s , a ) . While Ẑ is formally defined as a sample from the return distribution , k is typically computed in practice as K 1 PK j=1 r + qj ( s , a ) qk ( s , a ) [ 17 ] . 4 Optimism versus Pessimism . Reducing overestimation bias with pessimism It was observed by [ 53 ] that Q-learning [ 56 ] with function approximation is biased towards overestimation . Noting that this overestimation bias can introduce instability in training , [ 20 ] introduced the Twin Delayed Deep Deterministic ( TD3 ) policy gradient algorithm to correct for the bias . TD3 can be viewed as a pessimistic heuristic in which values are estimated via a SARSA-like variant of double Q-learning [ 24 ] and the Bellman target is constructed by taking the minimum of two critics : yt = rt+1 + min i2 { 1,2 } Q⇡ i ( s , ⇡✓ ( s ) + ✏ ) . ( 3 ) Here ✏ ⇠ clip ( N ( 0 , s2 ) , c , c ) is drawn from a clipped Gaussian distribution ( c is a constant ) . This added noise is used for smoothing in order to prevent the actor from overfitting to narrow peaks in the value function . Secondly , TD3 delays policy updates , updating value estimates several times between each policy gradient step . By taking the minimum of two separate critics and increasing the number of critic updates for each policy update , this approach takes a pessimistic view on the policy ’ s value in order to reduce overestimation bias . These ideas have become ubiquitous in state-of-the-art continuous control algorithms [ 7 ] , such as SAC , RAD , ( PI ) -SAC [ 21 , 31 , 33 ] . Optimism in the face of uncertainty While it is valuable to attempt to correct for overestimation of the value function , it is also important to recall that overestimation can be viewed as a form of optimism , and as such can provide a guide for exploration , a necessary ingredient in theoretical treatments of RL in terms of regret [ 27 , 26 , 4 ] . In essence , the effect of optimistic value estimation is to induce the agent to explore regions of the state space with high epistemic uncertainty , encouraging further data collection in unexplored regions . Moreover , [ 14 ] found that reducing value estimates , as done in pessimistic algorithms , can lead to pessimistic underexploration , in which actions that could lead to experience that gives the agent a better long-term reward . To address this problem , [ 14 ] introduced the Optimistic Actor-Critic ( OAC ) algorithm , which trains an exploration policy using an optimistic upper bound on the value function while constructing targets for learning using the lower bound of [ 20 ] . OAC demonstrated improved performance compared to SAC , hinting at a complex interplay between optimism and pessimism in deep RL algorithms . Trading off optimism and pessimism As we have discussed , there are arguments for both optimism and pessimism in RL . Optimism can aid exploration , but if there is significant estimation error , then a more pessimistic approach may be needed to stabilize learning . Moreover , both approaches have led to algorithms that are supported by strong empirical ev- idence . We aim to reconcile these seemingly contradictory perspectives by hypothesizing that the relative contributions of these two ingredients can vary depending on the nature of the task , with relatively simple settings revealing predominantly one aspect . As an illustrative example , we trained “ Optimistic ” and “ Pessimistic ” versions of the same deep actor-critic algorithm ( details in Section 6 ) for two different tasks and compared their performance in Figure 2 . As we can see , in the HalfCheetah task , the Optimistic agent outperforms the Pessimistic agent , while in the Hopper task , the opposite is true . This result suggests that the overall phenomenon is multi-faceted and active management of the overall optimism-pessimism trade-off is necessary . Accordingly , in the current paper we propose the use of an adaptive approach in which the degree of optimism or pessimism is adjusted dynamically during training . As a consequence of this approach , the optimal degree of optimism can vary across tasks and over the course of a single training run as the model improves . Not only does this approach reconcile the seemingly contradictory perspectives in the literature , but it also can outperform each individual framework in a wider range of tasks .

Summary: The paper considers the problem of approximate value estimation in continuous-action actor-critic architectures, raising the argument that the overestimation bias is not always (in all environments) bad. In particular, they accord to the *optimism in the face of uncertainty* principle to argue that overestimation helps in some domains due to promoting exploration, while in others the negative impacts of such bias outweigh the exploratory benefits (which is why pessimistic approaches such as TD3 have improved performance in many domains). Based on this argument the direction of the paper is a natural one: create a method that adapts the agent's degree of optimism in a single training lifetime. The authors propose such an approach where (at least) two action-value critics are trained, based on which a measure of *epistemic uncertainty* can be obtained. Based on this epistemic uncertainty, the agent's value estimation can be adjusted to be more optimistic or pessimistic. To isolate this measure from *aleatoric uncertainty*, they use distributional critics. There is only a single scalar parameter, $\beta$, that determines the degree of optimism, and the proposed method applies a multi-armed bandit formulation across episodes to optimize this parameter (over discretized values of the parameter in the range of $[-1, 0]$).