$\sbf{\delta^2}$-exploration for Reinforcement Learning

1 INTRODUCTION . In Reinforcement Learning ( RL ) an agent interacts with an external environment taking sequences of actions that cause transitions between states of the environment , with the scope to maximize the sum of rewards that are gathered at each state . In a typical scenario , the RL agent does not initially posses perfect knowledge of the consequences of its actions on the environment , but has to learn that through experience . It therefore has to face the classical exploration-exploitation dilemma , i.e . deciding whether to exploit the actions that are known to maximize immediate reward or whether to explore unfamiliar actions and states in order to potentially find ways to increase future rewards . Efficient exploration requires the agent to quantify the accuracy of its current estimates of the stateaction values ( the conditional expected rewards ) , so as to trade off expected rewards and uncertainty in a principled way ( Strehl & Littman , 2008 ; O ’ Donoghue et al. , 2018 ) . Measuring the uncertainty associated with the value of each state-action is thus a key component of conventional algorithms for addressing the exploration-exploitation dilemma . These ideas have been developed into successful uncertainty-based exploration strategies such as the Upper Confidence Bound ( UCB ) algorithm , which quantifies uncertainty through confidence intervals ( Auer et al. , 2002 ) , and Thompson Sampling ( TS ) , which instead models the posterior distribution over values ( Thompson , 1933 ) . However , obtaining either of these measures of uncertainty for general sequential Reinforcement Learning problems has proven arduous because of their reliance on a measure of uncertainty around the current value predictions , which is typically infeasible to obtain for complex environments and value functions . This difficulty limits the applicability of explorations strategies based on UCB and TS for general RL problems , compromising in particular their adaptability to algorithms such as Q-learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) and DQN ( Mnih et al. , 2015 ) , which consequently typically resort to addressing exploration via inefficient heuristics like -greedy action selection . Recently , Rigotti & Zhu ( 2021 ) tackled the mentioned limitations of conventional exploration strategies in the constrained but interesting case of bandit problems . Central to that effort was the development of a novel uncertainty measure , the Sample Average Uncertainty ( SAU ) , which departs from the approach of estimating the uncertainty of expected rewards of selected actions . In contrast to what is done in UCB and TS , SAU is an uncertainty measure that only depends on the value prediction of each action . Specifically , it consists in quantifying the variance of sample averages of rewards under the assumption of known values , which can be used to simply construct an estimated variance by plugging in the estimated values . Finally , Rigotti & Zhu ( 2021 ) were able to use SAU to developed two SAU-based exploration strategies , SAU-UCB and SAU-Sampling , which were shown to display state-of-the-art performance in multi-armed bandits , contextual bandit , and deep bandits . In this work , we exploit the simplicity of SAU in Rigotti & Zhu ( 2021 ) and extend it from the bandits to the more general sequential RL setting . This extension results in δ2-exploration , a class of exploration strategies for RL which share the implementation simplicity of SAU for bandits as well as its effectiveness . Our δ2-exploration algorithms are implementable with a minimal computational overhead that is comparable to -greedy exploration , while being empirically more effective . As a practical demonstration , we apply δ2-exploration to Q-leaning ( Watkins , 1989 ) . We then show that the resulting strategies display empirically efficient exploration , which very quickly reach near optimal policies . As mentioned , in terms of computation and memory requirements , these exploration strategies do not add any additional cost compared to for instance a simple -greedy action selection strategy . Finally , we apply SAU-based exploration to Deep Q Networks ( DQN ) , and observe convincing performance gains over standard DQN in several games we tested from the Atari 2600 domain . Thanks to its efficiency , flexibility and implementation simplicity , we argue that δ2-exploration could serve as a competitive baseline and as a simple and straight-forward replacement to -greedy action selection in a wide class of RL scenarios . In the remainder of this paper , Section 2 summarizes common exploration strategies in bandits and Reinforcement Learning in order to set the stage for δ2-exploration . Section 3 establishes the notation and states the problem . Section 4.2 presents the SAU measure , and summarized SAUbased exploration strategies in the setting of multi-armed bandits as developed in Rigotti & Zhu ( 2021 ) . In Section 5 we then extend these strategies to the sequential RL setting and develop δ2exploration . Section 6 includes empirical evaluations showing substantial performance improvements over conventional exploration strategies in Q-learning and DQN . Section 7 concludes the paper . 2 RELATED WORK . The most common exploration strategy in Reinforcement Learning ( RL ) is arguably -greedy action selection , which consists in following the action with the highest estimated value with high probability , but occasionally with a small fixed probability choosing any other available action uniformly at random . Despite the clear inefficiency of not differentiating lower value actions based on their uncertainty or even their value , because of its simplicity -greedy action selection is still the exploration strategy of choice in RL . One of the most popular and efficient exploration algorithms is the Upper Confidence Bound ( UCB ) algorithm , which is very successful in the bandit setting ( Auer et al. , 2002 ; Auer , 2003 ; Rusmevichientong & Tsitsiklis , 2010 ; Abbasi-Yadkori et al. , 2011 ; Abbasi-Yadkori , 2012 ; Perchet & Rigollet , 2013 ; Slivkins , 2014 ) . This approach is based on the principle of optimism in the face of uncertainty , which elegantly addresses the exploration-exploitation trade-off by maintaining confidence intervals for action-value estimates and choosing actions optimistically within these intervals . Unfortunately , UCB is difficulty to apply to RL problems on complex domains ( Jaksch et al. , 2010 ; Wen & Van Roy , 2013 ) due to the complexity of computing its uncertainty measure . A related class of strategies to UCB are count-based methods ( Strehl & Littman , 2008 ; Bellemare et al. , 2016 ) that directly turn the counts of state-action visitations into reward bonuses to encourage the exploration of less visited ones . Despite performing near-optimally for small discrete Markov decision processes , count-based methods suffer the curse of dimensionality in high-dimensional environments ( although see Tang et al . ( 2017 ) that proposed a possible high-dimensional extension using learned hashing to discretize the state space ) . Another popular exploration strategy is Thompson Sampling ( TS ) which was introduced by Thompson ( 1933 ) . This is a Bayesian approach that follows the principle of sampling in the face of uncertainty , i.e . it samples action-values from a known posterior distribution , then selects the action with the highest sampled value . TS has been successfully applied in bandit settings ( Chapelle & Li , 2011 ; Agrawal & Goyal , 2013 ; Russo & Van Roy , 2014 ) . It has also been applied to simple Reinforcement Learning problems ( Strens , 2000 ; Osband et al. , 2013 ) . However , because calculating the posterior distributions over values is intractable in the general case , implementations of TS in even moderately sophisticated domains require the application of posterior approximation methods , like for instance in Osband et al . ( 2016 ) , which , at least in the contextual bandit setting , are known to be inefficient ( see Riquelme et al . ( 2018 ) ) . 3 PROBLEM FORMULATION . We recall the definition of a Markov Decision Process ( MDP ) as a set of states S and actionsA so that , given a current state s ∈ S and an action a ∈ A , the probability of ending up in the next state s′ ∈ S is Pss′ ( a ) , where P : S×A×S → [ 0 , 1 ] is a fixed state transition distribution with ∑ s′ Pss′ ( a ) = 1 . Moreover , the reward r is drawn from a reward distribution with E [ r|s , a , s′ ] = Rss′ ( a ) . The optimal value function Q∗ ( s , a ) solves an MDP by satisfying the following set of Bellman equations for a given discount factor γ ∈ [ 0 , 1 ) : Q∗ ( s , a ) = ∑ s′ Pss′ ( a ) [ Rss′ ( a ) + γ max a′∈A Q∗ ( s′ , a′ ) ] ∀s , a. MDPs can be solved using Q-learning ( Watkins , 1989 ) . Let Q ( s , a ) be the current estimate of the value of action a in state s , and let s′ be the state that follows s upon taking a , and r be the corresponding reward . Then Q-learning consists in updating the action-value function using a learning rate α as follows : Q ( s , a ) ← Q ( s , a ) + α [ r + γ max a′∈A Q ( s′ , a′ ) −Q ( s , a ) ] . In this update rule , Q-learning uses the max operator as an estimate the value of the next state . The term in the squared brackets δ = r + γ max a′∈A Q ( s′ , a′ ) −Q ( s , a ) ( 1 ) is known as the TD-error , because it is a generalization of the reward prediction error in the case of a Temporal-Difference learning algorithm . A last element that is crucial for the Q-learning algorithm to converge to the optimal value function is exploration . That is why Q-learning is typically implemented in combination with -greedy action selection , a simple and rather successful exploration strategy proposed in Sutton & Barto ( 2018 ) that prescribes to fix a small probability and choose an action at time t as follows : at = { a ∈ A uniformly at random , with probability ; a∗ = arg maxa∈A { Q ( st , a ) } , with probability 1− . ( 2 ) As mentioned in the previous section , -greedy exploration has the obvious inefficiency that , during an exploratory event of probability , actions are selected without differentiating on their uncertainty or even their value . But because of its simplicity -greedy action selection is typically the exploration strategy of choice in many successful RL applications , such as Mnih et al . ( 2015 ) . 4 SAMPLE AVERAGE UNCERTAINTY FOR CONTEXTUAL BANDITS . The Sample Average Uncertainty ( SAU ) is a measure of uncertainty that was recently proposed as a simple and scalable mechanism to promote exploration in the bandit setting ( Rigotti & Zhu , 2021 ) . For ease of exposition we introduce SAU from the contextual bandit perspective and notation . 4.1 CONTEXTUAL BANDITS . In contextual bandits at each step t the agent observes a context vector st ∈ X , selects an action at from a set A , after which a reward rt corresponding to the chosen action is received . The value of an action a in context st is defined as the expected reward given that a is selected , and is assumed to be a fixed but unknown function of st and a : E [ rt|at = a ] = µ ( st , a ) . The goal is to design a decisionmaking policy π : X → A that maximizes the expected reward . This goal is readily quantified in terms of minimizing expected regret , where , if action at is chosen at step t after observing context st , we say that we incur expected regret maxa′∈A { µ ( st , a′ ) } − µ ( st , at ) , which is the difference between the optimal value received by playing the optimal action and the value received following the chosen action .

Summary: The paper extends the recently introduced SAU measure from bandits to RL. The idea is to enhance the exploration of action based on an approximation of the estimation error. More precisely, the bonus is proportional to the average squared temporal difference error. This uncertainty measure is easy to compute and can be integrated into both tabular and continuous methods (e.g., Q-learning and DQN).