Efficient Exploration through Bayesian Deep Q-Networks

We propose Bayesian Deep Q-Networks ( BDQN ) , a principled and a practical Deep Reinforcement Learning ( DRL ) algorithm for Markov decision processes ( MDP ) . It combines Thompson sampling with deep-Q networks ( DQN ) . Thompson sampling ensures efficient exploration-exploitation trade-off in high dimensions . It is typically carried out through posterior sampling over the model parameters , which makes it computationally expensive . To overcome this limitation , we directly incorporate uncertainty over the value ( Q ) function . Further , we only introduce randomness in the last layer ( i.e . the output layer ) of the DQN and use independent Gaussian priors on the weights . This allows us to efficiently carry out Thompson sampling through Gaussian sampling and Bayesian Linear Regression ( BLR ) , which has fast closed-form updates . The rest of the layers of the Q network are trained through back propagation , as in a standard DQN . We apply our method to a wide range of Atari games in Arcade Learning Environments and compare BDQN to a powerful baseline : the double deep Q-network ( DDQN ) . Since BDQN carries out more efficient exploration , it is able to reach higher rewards substantially faster : in less than 5M±1M interactions for almost half of the games to reach DDQN scores while a typical run of DDQN is 50-200M . We also establish theoretical guarantees for the special case when the feature representation is d-dimensional and fixed . We show that the Bayesian regret of posterior sampling RL ( PSRL ) and frequentist regret of the optimism in the face of uncertainly ( OFU ) after N time step are upper bounded by Õ ( d √ N ) which are tight up-to logarithmic factors . To the best of our knowledge , these are the first model free theoretical guarantee for continuous state-action space MDP , beyond the tabular setting . 1 INTRODUCTION . One of the central challenges in reinforcement learning ( RL ) is to design efficient explorationexploitation trade-off that also scales to high-dimensional state and action spaces . Recently deep RL has shown good promise in being able scale to high-dimensional ( continuous ) spaces . These successes are mainly demonstrated in simulated domains where exploration is considered to be inexpensive and simple exploration strategies are deployed , e.g . ε-greedy which uniformly explores over all the actions with ε probability . Such exploration strategies inherently inefficient for complex high-dimensional environments . On the other hand , more sophisticated strategies have mostly been limited to low dimensional MDPs . For example , OFU is only practical when the domain is small enough to be represented with lookup tables for the Q-values ( Jaksch et al. , 2010 ; Brafman & Tennenholtz , 2003 ) . An alternative to optimism-under-uncertainty is Thompson Sampling ( TS ) , a general sampling and randomization approach ( in both frequentist and Bayesian settings ) ( Thompson , 1933 ) . Under the Bayesian framework , Thompson sampling maintains a prior distribution over the environmental models ( i.e . models of reward and dynamics ) , and updating the posterior based on observations collected . An action is chosen from the posterior distribution , the belief , such that it maximizes the expected return . Thompson sampling has been observed to work significantly better than optimistic approaches in many low dimensional settings such as contextual bandits ( Chapelle & Li , 2011 ) , small MDPs ( Osband et al. , 2013 ) and also has strong theoretical bounds ( Russo & Van Roy , 2014a ; b ; Agrawal & Goyal , 2012 ; Osband et al. , 2013 ; Abbasi-Yadkori & Szepesvári , 2015 ) . In the MDP setting , ( model-based ) Thompson Sampling involves sampling the parameters of reward and dynamics models , performing MDP planning using the sampled model and then computing the policy ( Strens , 2000 ; Osband et al. , 2013 ; Osband & Van Roy , 2014b ; a ) . However , the computational costs of posterior sampling and planning scales as cubic in the problem dimension , which makes it intractable for large MDPs . Therefore , some form of function approximation is required to scale Thompson Sampling to high dimensions , and this can be either of the model , the Q-value function , or the policy . To address this , Osband et al . ( 2014 ) introduced randomized least-squares value iteration ( RLSVI ) which combines linear value function approximation with Bayesian regression to directly sample the value-function weights from a distribution . The authors prove a regret bound for this approach in tabular MDPs . This has been extended to continuous spaces by Osband et al . ( 2016 ) , where deep networks are used to approximate the Q function . Through a bootstrapped-ensemble approach , several deep-Q network ( DQN ) models are trained in parallel to approximate the posterior distribution . Other works use the posterior distribution over the parameters of each node in the network and employ variational approximation ( Lipton et al. , 2016b ) or use noisy networks ( Fortunato et al. , 2017 ) . These approaches significantly increase the computation cost compared to the standard DQN . For instance , the bootstrapped-ensemble incurs a computation overhead that is linear in the number of bootstrap models . Moreover , despite principled design of the methods in the above works , they only produced modest gains over DQN in empirical studies . Contribution 1 – Design of BDQN : We introduce Bayesian Deep Q-Network ( BDQN ) , a Thompsonsampling algorithm for deep RL . It is a simple approach that extends randomized least-squares value iteration ( Osband et al. , 2014 ) to deep networks . We introduce stochasticity only in the last layer of the Q-network using independent Gaussian priors on the weights . This allows us to do efficient approximate Thompson sampling1 using Bayesian linear regression ( BLR ) , which has fast closedform updates and sampling from the resulting posterior distribution . The rest of the Q-network is trained through usual back propagation . Contribution 2 – Strong empirical results for BDQN : We test BDQN on a wide range of Atari games ( Bellemare et al. , 2013 ; Machado et al. , 2017 ) , and compare our results to a powerful baseline : Double DQN ( DDQN ) ( Van Hasselt et al. , 2016 ) a bias-reduced extension of DQN . BDQN and DDQN use the same network architecture , and follow the same target objective , and differ only in the way they select actions : DDQN uses ε-greedy exploration while BDQN performs ( approximate ) Thompson sampling . We found that BDQN is able to reach much higher cumulative rewards , and also at a higher speed , compared to DDQN on all the tested games . We also found that BDQN can be trained with much higher learning rates compared to DDQN . This is intuitive since BDQN has better exploration strategy . The cumulative reward ( score ) for BDQN at the end of training improves by a median of 300 % with a maximum of 80K % in these games . Also , BDQN has 300 % ± 40 % ( mean and standard deviation ) improvement over these games on area under the performance measure . This can be considered as a surrogate for sample complexity and regret . Indeed , no single measure of performance provides a complete picture of an algorithm , and we present detailed experiments in Section 5 . In terms of computational cost , BDQN is only slightly more expensive compared to DQN and DDQN . For the DQN in Atari games , this is the cost of inverting a 512× 512 matrix every 100,000 time steps , which is negligible . On the other hand , more sophisticated Bayesian RL techniques are significantly more expensive and have not lead to large gains over DQN and DDQN ( Osband et al. , 2016 ) . Contribution 3 – Tight Bayesian and frequentist regret upper bounds for continuous MDPs : We establish theoretical guarantees for the special case when the feature representation is fixed ( i.e . all layers except the last ) , and not learnt . We consider episodic MDPs with continuous space of states and actions such that the Q-function is a linear function of a given d-dimensional feature map . We show that when PSRL and OFUare deployed , respectively , the Bayesian regret and frequentist regret after N time step are upper bounded by Õ ( d √ N ) , which is shown to be tight up-to logarithmic factors . Since linear bandits are a special case of episodic continuous MDPs ( with horizon length 1 ) , it implies that our regret bounds are tight in the dimension d and in the number of episodes . The Bayesian bound matches the Bayesian regret bound of linear bandits ( Russo & Van Roy , 2014a ) and the frequentist bound matches the frequentist regret bound for linear bandits ( Abbasi-Yadkori 1BDQN approximates the posterior resulting in approximating Thompson sample . In the remaining , we state Thompson Sampling and its approximation as Thomson samples unless we specify . et al. , 2011 ) . To the best of our knowledge , these are the first model free theoretical guarantee for continuous MDPs beyond the tabular setting . Thus , our proposed approach has several desirable features – faster learning and better sample complexity due to targeted exploration , negligible computational overhead due to simplicity , significant improvement in experiments , and tight theoretical bounds . 2 THOMPSON SAMPLING VS ε-GREEDY AND BOLTZMANN EXPLORATION In a value approximation RL algorithm , there are different ways to manage the explorationexploitation trade-off . DQN uses a naive ε-greedy for exploration , where with ε probability it chooses a random action and with 1− ε probability it chooses the greedy action based on the estimated Q function . Note that there are only point estimates of the Q function in DQN . In contrast , our proposed Bayesian approach BDQN maintains uncertainties over the estimated Q function , and employs it to carry out Thompson Sampling based exploration . Here , we demonstrate the fundamental benefits of Thompson Sampling over ε-greedy and Boltzmann exploration strategies using simple examples . In Table 1 , we list the three strategies and their properties . ε-greedy is the simplest exploration strategy since it is uniformly random over all the non-greedy actions . Boltzmann exploration is an intermediate strategy since it uses the estimated Q function to sample the actions . However , it does not maintain uncertainties over the Q function estimation . In contrast , Thompson sampling also incorporates uncertainties over Q estimation and utilizes most information for exploration strategy . Consider the example in Figure 1 ( a ) with current estimates and uncertainties of the Q function over different actions . ε-greedy is wasteful since it assigns uniform probability to explore over 5 and 6 , which are obviously sub-optimal when the uncertainty estimates are available . In this setting , a possible remedy is Boltzmann exploration since it assigns lower probability to actions 5 and 6 but randomizes with almost the same probabilities over the remaining actions . However , Boltzmann exploration is sub-optimal in settings where there is high variances . For example if the current Q estimate is according to Figure 1 ( b ) , then Boltzmann exploration assigns almost equal probability to actions 5 and 6 , even though action 6 has much higher uncertainty and needs to be explored more . Thus , both ε-greedy and Boltzmann exploration strategies are sub-optimal since they do not maintain an uncertainty estimate over the Q function estimation . In contrast , Thompson sampling uses both estimated Q function and its uncertainty estimates to carry out the most efficient exploration .

Summary: This paper proposes a method for more efficient exploration in RL by maintaining uncertainty estimates over the learned Q-value function. It is comparable to Double DQN (DDQN) but uses its learned uncertainty estimates with Thompson sampling for exploration, rather than \epsilon-greedy. Empirically the method is significantly more sample-efficient for Atari agents than DDQN and other baselines.