Optimistic Exploration even with a Pessimistic Initialisation

1 INTRODUCTION . In reinforcement learning ( RL ) , exploration is crucial for gathering sufficient data to infer a good control policy . As environment complexity grows , exploration becomes more challenging and simple randomisation strategies become inefficient . While most provably efficient methods for tabular RL are model-based ( Brafman and Tennenholtz , 2002 ; Strehl and Littman , 2008 ; Azar et al. , 2017 ) , in deep RL , learning models that are useful for planning is notoriously difficult and often more complex ( Hafner et al. , 2019 ) than modelfree methods . Consequently , model-free approaches have shown the best final performance on large complex tasks ( Mnih et al. , 2015 ; 2016 ; Hessel et al. , 2018 ) , especially those requiring hard exploration ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ) . Therefore , in this paper , we focus on how to devise model-free RL algorithms for efficient exploration that scale to large complex state spaces and have strong theoretical underpinnings . Despite taking inspiration from tabular algorithms , current model-free approaches to exploration in deep RL do not employ optimistic initialisation , which is crucial to provably efficient exploration in all model-free tabular algorithms . This is because deep RL algorithms do not pay special attention to the initialisation of the neural networks and instead use common initialisation schemes that yield initial Q-values around zero . In the common case of non-negative rewards , this means Q-values are initialised to their lowest possible values , i.e. , a pessimistic initialisation . While initialising a neural network optimistically would be trivial , e.g. , by setting the bias of the final layer of the network , the uncontrolled generalisation in neural networks changes this initialisation quickly . Instead , to benefit exploration , we require the Q-values for novel state-action pairs must remain high until they are explored . An empirically successful approach to exploration in deep RL , especially when reward is sparse , is intrinsic motivation ( Oudeyer and Kaplan , 2009 ) . A popular variant is based on pseudocounts ( Bellemare et al. , 2016 ) , which derive an intrinsic bonus from approximate visitation counts over states and is inspired by the tabular MBIE-EB algorithm ( Strehl and Littman , 2008 ) . However , adding a positive intrinsic bonus to the reward yields optimistic Q-values only for state-action pairs that have already been chosen sufficiently often . Incentives to explore unvisited states rely therefore on the generalisation of the neural network . Exactly how the network generalises to those novel state-action pairs is unknown , and thus it is unclear whether those estimates are optimistic when compared to nearby visited state-action pairs . Figure 1 Consider the simple example with a single state and two actions shown in Figure 1 . The left action yields +0.1 reward and the right action yields +1 reward . An agent whose Q-value estimates have been zero-initialised must at the first time step select an action randomly . As both actions are underestimated , this will increase the estimate of the chosen action . Greedy agents always pick the action with the largest Q-value estimate and will select the same action forever , failing to explore the alternative . Whether the agent learns the optimal policy or not is thus decided purely at random based on the initial Q-value estimates . This effect will only be amplified by intrinsic reward . To ensure optimism in unvisited , novel state-action pairs , we introduce Optimistic Pessimistically Initialised Q-Learning ( OPIQ ) . OPIQ does not rely on an optimistic initialisation to ensure efficient exploration , but instead augments the Q-value estimates with count-based bonuses in the following manner : Q+ ( s , a ) : = Q ( s , a ) + C ( N ( s , a ) + 1 ) M , ( 1 ) where N ( s , a ) is the number of times a state-action pair has been visited and M , C > 0 are hyperparameters . These Q+-values are then used for both action selection and during bootstrapping , unlike the above methods which only utilise Q-values during these steps . This allows OPIQ to maintain optimism when selecting actions and bootstrapping , since the Q+-values can be optimistic even when the Q-values are not . In the tabular domain , we base OPIQ on UCB-H ( Jin et al. , 2018 ) , a simple online Q-learning algorithm that uses count-based intrinsic rewards and optimistic initialisation . Instead of optimistically initialising the Q-values , we pessimistically initialise them and use Q+-values during action selection and bootstrapping . Pessimistic initialisation is used to enable a worst case analysis where all of our Q-value estimates underestimate Q∗ and is not a requirement for OPIQ . We show that these modifications retain the theoretical guarantees of UCB-H . Furthermore , our algorithm easily extends to the Deep RL setting . The primary difficulty lies in obtaining appropriate state-action counts in high-dimensional and/or continuous state spaces , which has been tackled by a variety of approaches ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ; Machado et al. , 2018a ) and is orthogonal to our contributions . We demonstrate clear performance improvements in sparse reward tasks over 1 ) a baseline DQN that just uses intrinsic motivation derived from the approximate counts , 2 ) simpler schemes that aim for an optimistic initialisation when using neural networks , and 3 ) strong exploration baselines . We show the importance of optimism during action selection for ensuring efficient exploration . Visualising the predicted Q+-values shows that they are indeed optimistic for novel state-action pairs . 2 BACKGROUND . We consider a Markov Decision Process ( MDP ) defined as a tuple ( S , A , P , R ) , where S is the state space , A is the discrete action space , P ( ·|s , a ) is the state-transition distribution , R ( ·|s , a ) is the distribution over rewards and γ ∈ [ 0 , 1 ) is the discount factor . The goal of the agent is then to maximise the expected discounted sum of rewards : E [ ∑∞ t=0 γ trt|rt ∼ R ( ·|st , at ) ] , in the discounted episodic setting . A policy π ( ·|s ) is a mapping from states to actions such that it is a valid probability distribution . Qπ ( s , a ) : = E [ ∑∞ t=0 γ trt|at ∼ π ( ·|st ) ] and Q∗ : = maxπ Qπ . Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) uses a nonlinear function approximator ( a deep neural network ) to estimate the action-value function , Q ( s , a ; θ ) ≈ Q∗ ( s , a ) , where θ are the parameters of the network . Exploration based on intrinsic rewards ( e.g. , Bellemare et al. , 2016 ) , which uses a DQN agent , additionally augments the observed rewards rt with a bonus β/ √ N ( st , at ) based on pseudo-visitation-counts N ( st , at ) . The DQN parameters θ are trained by gradient descent on the mean squared regression loss L with bootstrapped ‘ target ’ yt : L [ θ ] : = E [ ( yt︷ ︸︸ ︷ rt + β√ N ( st , at ) + γmax a′ Q ( st+1 , a ′ ; θ− ) −Q ( st , at ; θ ) ) 2∣∣∣ ( st , at , rt , st+1 ) ∼D ] . ( 2 ) The expectation is estimated with uniform samples from a replay buffer D ( Lin , 1992 ) . D stores past transitions ( st , at , rt , st+1 ) , where the state st+1 is observed after taking the action at in state st and receiving reward rt . To improve stability , DQN uses a target network , parameterised by θ− , which is periodically copied from the regular network and kept fixed for a number of iterations . 3 OPTIMISTIC PESSIMISTICALLY INITIALISED Q-LEARNING Our method Optimistic Pessimistically Initialised Q-Learning ( OPIQ ) ensures optimism in the Qvalue estimates of unvisited , novel state-action pairs in order to drive exploration . This is achieved by augmenting the Q-value estimates in the following manner : Q+ ( s , a ) : = Q ( s , a ) + C ( N ( s , a ) + 1 ) M , and using these Q+-values during action selection and bootstrapping . In this section , we motivate OPIQ , analyse it in the tabular setting , and describe a deep RL implementation . 3.1 MOTIVATIONS . Optimistic initialisation does not work with neural networks . For an optimistic initialisation to benefit exploration , the Q-values must start sufficiently high . More importantly , the values for unseen state-action pairs must remain high , until they are updated . When using a deep neural network to approximate the Q-values , we can initialise the network to output optimistic values , for example , by adjusting the final bias . However , after a small amount of training , the values for novel state-action pairs may not remain high . Furthermore , due to the generalisation of neural networks we can not know how the values for these unseen state-action pairs compare to the trained state-action pairs . Figure 2 ( left ) , which illustrates this effect for a simple regression task , shows that different initialisations can lead to dramatically different generalisations . It is therefore prohibitively difficult to use optimistic initialisation of a deep neural network to drive exploration . Instead , we augment our Q-value estimates with an optimistic bonus . Our motivation for the form of the bonus in equation 1 , C ( N ( s , a ) +1 ) M , stems from UCB-H ( Jin et al. , 2018 ) , where all tabular Q-values are initialised with H and the first update for a state-action pair completely overwrites that value because the learning rate for the update ( η1 ) is 1 . One can alternatively view these Q-values as zero-initialised with the additional term Q ( s , a ) + H · 1 { N ( s , a ) < 1 } , where N ( s , a ) is the visitation count for the state-action pair ( s , a ) . Our approach approximates the discrete indicator Algorithm 1 OPIQ algorithm Initialise Qt ( s , a ) ← 0 , N ( s , a , t ) ← 0 , ∀ ( s , a , t ) ∈ S ×A× { 1 , ... , H , H + 1 } for each episode k = 1 , ... , K do for each timestep t = 1 , ... , H do Take action at ← arg maxaQ+t ( st , a ) . Receive r ( st , at , t ) and st+1 . Increment N ( st , at , t ) . Qt ( st , at ) ← ( 1−ηN ) Qt ( st , at ) +ηN ( r ( st , at , t ) + bTN +min { H , maxa′ Q + t+1 ( st+1 , a ′ ) } ) . end end function 1 as ( N ( s , a ) + 1 ) −M for sufficiently large M . However , since gradient descent can not completely overwrite the Q-value estimate for a state-action pair after a single update , it is beneficial to have a smaller hyperparameter M that governs how quickly the optimism decays . For a worst case analysis we assume all Q-value estimates are pessimistic . In the common scenario where all rewards are nonnegative , the lowest possible return for an episode is zero . If we then zero-initialise our Q-value estimates , as is common for neural networks , we are starting with a pessimistic initialisation . As shown in Figure 2 ( left ) , we can not predict how a neural network will generalise , and thus we can not predict if the Q-value estimates for unvisited state-action pairs will be optimistic or pessimistic . We thus assume they are pessimistic in order to perform a worst case analysis . However , this is not a requirement : our method works with any initialisation and rewards . In order to then approximate an optimistic initialisation , the scaling parameter C in equation 1 can be chosen to guarantee unseen Q+-values are overestimated , for example , C : = H in the undiscounted finite-horizon tabular setting and C : = 1/ ( 1 − γ ) in the discounted episodic setting ( assuming 1 is the maximum reward obtainable at each timestep ) . However , in some environments it may be beneficial to use a smaller parameter C for faster convergence . These Q+-values are then used both during action selection and during bootstrapping . Note that in the finite horizon setting the counts N , and thus Q+ , would depend on the timestep t. Hence , we split the optimistic Q+-values into two parts : a pessimistic Q-value component and an optimistic component based solely on the counts for a state-action pair . This separates our source of optimism from the neural network function approximator , yielding Q+-values that remain high for unvisited state-action pairs , assuming a suitable counting scheme . Figure 2 ( right ) shows the effects of adding this optimistic component to a network ’ s outputs . Optimistic Q+-values provide an increased incentive to explore . By using optimistic Q+ estimates , especially during action selection and bootstrapping , the agent is incentivised to try and visit novel state-action pairs . Being optimistic during action selection in particular encourages the agent to try novel actions that have not yet been visitied . Without an optimistic estimate for novel state-action pairs the agent would have no incentive to try an action it has never taken before at a given state . Being optimistic during bootstrapping ensures the agent is incentivised to return to states in which it has not yet tried every action . This is because the maximum Q+-value will be large due to the optimism bonus . Both of these effects lead to a strong incentive to explore novel state-action pairs .

Summary: The paper proposes a method to optimistically initialize Q-values for unseen state actions for the case of Deep Q-Networks (DQNs) with high dimensional state representations, in order to step closer towards the optimistic initializations in the tabular Q-learning case which have proven guarantees of convergence. The paper shows that simple alternatives such as adding a bias to a DQN do not help as the generalization to novel states usually reduces the optimism of unvisited state-actions. Instead, a separate optimistic Q function is proposed that is an addition of two parts - a Deep Q-network which may be pessimistically initialized (in the worst case) and a term with pseudo-counts of state-actions that together form an optimistic estimate of novel state-actions. For the tabular case, the paper shows a convergence with guarantees similar to UCB-H (Jin et. al., 2018). This optimistic Q-function is used during action selection as well as bootstrapping in the n-step TD update.