Information-Directed Exploration for Deep Reinforcement Learning

1 INTRODUCTION . In Reinforcement Learning ( RL ) , an agent seeks to maximize the cumulative rewards obtained from interactions with an unknown environment . Given only knowledge based on previously observed trajectories , the agent faces the exploration-exploitation dilemma : Should the agent take actions that maximize rewards based on its current knowledge or instead investigate poorly understood states and actions to potentially improve future performance . Thus , in order to find the optimal policy the agent needs to use an appropriate exploration strategy . Popular exploration strategies , such as -greedy ( Sutton & Barto , 1998 ) , rely on random perturbations of the agent ’ s policy , which leads to undirected exploration . The theoretical RL literature offers a variety of statistically-efficient methods that are based on a measure of uncertainty in the agent ’ s model . Examples include upper confidence bound ( UCB ) ( Auer et al. , 2002 ) and Thompson sampling ( TS ) ( Thompson , 1933 ) . In recent years , these have been extended to practical exploration algorithms for large state-spaces and shown to improve performance ( Osband et al. , 2016a ; Chen et al. , 2017 ; O ’ Donoghue et al. , 2018 ; Fortunato et al. , 2018 ) . However , these methods assume that the observation noise distribution is independent of the evaluation point , while in practice heteroscedastic observation noise is omnipresent in RL . This means that the noise depends on the evaluation point , rather than being identically distributed ( homoscedastic ) . For instance , the return distribution typically depends on a sequence of interactions and , potentially , on hidden states or inherently heteroscedastic reward observations . Kirschner & Krause ( 2018 ) recently demonstrated that , even in the simpler bandit setting , classical approaches such as UCB and TS fail to efficiently account for heteroscedastic noise . In this work , we propose to use Information-Directed Sampling ( IDS ) ( Russo & Van Roy , 2014 ; Kirschner & Krause , 2018 ) for efficient exploration in RL . The IDS framework can be used to design exploration-exploitation strategies that balance the estimated instantaneous regret and the expected ∗Work done during exchange at ETH Zurich . information gain . Importantly , through the choice of an appropriate information-gain function , IDS is able to account for parametric uncertainty and heteroscedastic observation noise during exploration . As our main contribution , we propose a novel , tractable RL algorithm based on the IDS principle . We combine recent advances in distributional RL ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ) and approximate parameter uncertainty methods in order to develop both homoscedastic and heteroscedastic variants of an agent that is similar to DQN ( Mnih et al. , 2015 ) , but uses informationdirected exploration . Our evaluation on Atari 2600 games shows the importance of accounting for heteroscedastic noise and indicates that at our approach can substantially outperform alternative state-of-the-art algorithms that focus on modeling either only epistemic or only aleatoric uncertainty . To the best of our knowledge , we are the first to develop a tractable IDS algorithm for RL in large state spaces . 2 RELATED WORK . Exploration algorithms are well understood in bandits and have inspired successful extensions to RL ( Bubeck & Cesa-Bianchi , 2012 ; Lattimore & Szepesvári , 2018 ) . Many strategies rely on the ” optimism in the face of uncertainty ” ( Lai & Robbins , 1985 ) principle . These algorithms act greedily w.r.t . an augmented reward function that incorporates an exploration bonus . One prominent example is the upper confidence bound ( UCB ) algorithm ( Auer et al. , 2002 ) , which uses a bonus based on confidence intervals . A related strategy is Thompson sampling ( TS ) ( Thompson , 1933 ) , which samples actions according to their posterior probability of being optimal in a Bayesian model . This approach often provides better empirical results than optimistic strategies ( Chapelle & Li , 2011 ) . In order to extend TS to RL , one needs to maintain a distribution over Markov Decision Processes ( MDPs ) , which is difficult in general . Similar to TS , Osband et al . ( 2016b ) propose randomized linear value functions to maintain a Bayesian posterior distribution over value functions . Bootstrapped DQN ( Osband et al. , 2016a ) extends this idea to deep neural networks by using an ensemble of Qfunctions . To explore , Bootstrapped DQN randomly samples a Q-function from the ensemble and acts greedily w.r.t . the sample . Fortunato et al . ( 2018 ) and Plappert et al . ( 2018 ) investigate a similar idea and propose to adaptively perturb the parameter-space , which can also be thought of as tracking an approximate parameter posterior . O ’ Donoghue et al . ( 2018 ) propose TS in combination with an uncertainty Bellman equation , which propagates agent ’ s uncertainty in the Q-values over multiple time steps . Additionally , Chen et al . ( 2017 ) propose to use the Q-ensemble of Bootstrapped DQN to obtain approximate confidence intervals for a UCB policy . There are also multiple other ways to approximate parametric posterior in neural networks , including Neural Bayesian Linear Regression ( Snoek et al. , 2015 ; Azizzadenesheli et al. , 2018 ) , Variational Inference ( Blundell et al. , 2015 ) , Monte Carlo methods ( Neal , 1995 ; Mandt et al. , 2016 ; Welling & Teh , 2011 ) , and Bayesian Dropout ( Gal & Ghahramani , 2016 ) . For an empirical comparison of these , we refer the reader to Riquelme et al . ( 2018 ) . A shortcoming of all approaches mentioned above is that , while they consider parametric uncertainty , they do not account for heteroscedastic noise during exploration . In contrast , distributional RL algorithms , such as Categorical DQN ( C51 ) ( Bellemare et al. , 2017 ) and Quantile Regression DQN ( QR-DQN ) ( Dabney et al. , 2018b ) , approximate the distribution over the Q-values directly . However , both methods do not take advantage of the return distribution for exploration and use -greedy exploration . Implicit Quantile Networks ( IQN ) ( Dabney et al. , 2018a ) instead use a risksensitive policy based on a return distribution learned via quantile regression and outperform both C51 and QR-DQN on Atari-57 . Similarly , Moerland et al . ( 2018 ) and Dilokthanakul & Shanahan ( 2018 ) act optimistically w.r.t . the return distribution in deterministic MDPs . However , these approaches to not consider parametric uncertainty . Return and parametric uncertainty have previously been combined for exploration by Tang & Agrawal ( 2018 ) and Moerland et al . ( 2017 ) . Both methods account for parametric uncertainty by sampling parameters that define a distribution over Q-values . The former then act greedily with respect to the expectation of this distribution , while the latter additionally samples a return for each action and then acts greedily with respect to it . However , like Thompson sampling , these approaches do not appropriately exploit the heteroscedastic nature of the return . In particular , noisier actions are more likely to be chosen , which can slow down learning . Our method is based on Information-Directed Sampling ( IDS ) , which can explicitly account for parametric uncertainty and heteroscedasticity in the return distribution . IDS has been primarily studied in the bandit setting ( Russo & Van Roy , 2014 ; Kirschner & Krause , 2018 ) . Zanette & Sarkar ( 2017 ) extend it to finite MDPs , but their approach remains impractical for large state spaces , since it requires to find the optimal policies for a set of MDPs at the beginning of each episode . 3 BACKGROUND . We model the agent-environment interaction with a MDP ( S , A , R , P , γ ) , where S and A are the state and action spaces , R ( s , a ) is the stochastic reward function , P ( s′|s , a ) is the probability of transitioning from state s to state s′ after taking action a , and γ ∈ [ 0 , 1 ) is the discount factor . A policy π ( ·|s ) ∈ P ( A ) maps a state s ∈ S to a distribution over actions . For a fixed policy π , the discounted return of action a in state s is a random variable Zπ ( s , a ) = ∑∞ t=0 γ tR ( st , at ) , with initial state s = s0 and action a = a0 and transition probabilities st ∼ P ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) . The return distribution Z statisfied the Bellman equation , Zπ ( s , a ) D = R ( s , a ) + γZπ ( s′ , a′ ) , ( 1 ) where D= denotes distributional equality . If we take the expectation of ( 1 ) , the usual Bellman equation ( Bellman , 1957 ) for the Q-function , Qπ ( s , a ) = E [ Zπ ( s , a ) ] , follows as Qπ ( s , a ) = E [ R ( s , a ) ] + γEP , π [ Qπ ( s′ , a′ ) ] . ( 2 ) The objective is to find an optimal policy π∗ that maximizes the expected total discounted return E [ Zπ ( s , a ) ] = Qπ ( s , a ) for all s ∈ S , a ∈ A . 3.1 UNCERTAINTY IN REINFORCEMENT LEARNING . To find such an optimal policy , the majority of RL algorithms use a point estimate of the Q-function , Q ( s , a ) . However , such methods can be inefficient , because they can be overconfident about the performance of suboptimal actions if the optimal ones have not been evaluated before . A natural solution for more efficient exploration is to use uncertainty information . In this context , there are two source of uncertainty . Parametric ( epistemic ) uncertainty is a result of ambiguity over the class of models that explain that data seen so far , while intrinsic ( aleatoric ) uncertainty is caused by stochasticity in the environment or policy , and is captured by the distribution over returns ( Moerland et al. , 2017 ) . Osband et al . ( 2016a ) estimate parametric uncertainty with a Bootstrapped DQN . They maintain an ensemble ofK Q-functions , { Qk } Kk=1 , which is represented by a multi-headed deep neural network . To train the network , the standard bootstrap method ( Efron , 1979 ; Hastie et al. , 2001 ) constructs K different datasets by sampling with replacement from the global data pool . Instead , Osband et al . ( 2016a ) trains all network heads on the exact same data and diversifies the Q-ensemble via two other mechanisms . First , each headQk ( s , a ; θ ) is trained on its own independent target headQk ( s , a ; θ− ) , which is periodically updated ( Mnih et al. , 2015 ) . Further , each head is randomly initialized , which , combined with the nonlinear parameterization and the independently targets , provides sufficient diversification . Intrinsic uncertainty is captured by the return distribution Zπ . While Q-learning ( Watkins , 1989 ) aims to estimate the expected discounted return Qπ ( s , a ) = E [ Zπ ( s , a ) ] , distributional RL approximates the random return Zπ ( s , a ) directly . As in standard Q-learning ( Watkins , 1989 ) , one can define a distributional Bellman optimality operator based on ( 1 ) , T Z ( s , a ) D : = R ( s , a ) + γZ ( s′ , arg max a′∈A E [ Z ( s′ , a′ ) ] ) . ( 3 ) To estimate the distribution of Z , we use the approach of C51 ( Bellemare et al. , 2017 ) in the following . It parameterizes the return as a categorical distribution over a set of equidistant atoms in a fixed interval [ Vmin , Vmax ] . The atom probabilities are parameterized by a softmax distribution over the outputs of a parametric model . Since the parameterization Zθ and the Bellman update T Zθ have disjoint supports , the algorithm requires an additional step Φ that projects the shifted support of T Zθ onto [ Vmin , Vmax ] . Then it minimizes the Kullback-Leibler divergence DKL ( ΦT Zθ||Zθ ) .

Summary: The authors propose a way of extending Information-Directed Sampling (IDS) to reinforcement learning. The proposed approach uses Bootstrapped DQN to estimate parametric uncertainty in Q-values, and distributional RL to estimate intrinsic uncertainty in the return. The two types of uncertainty are combined to obtain a simple exploration strategy based on IDS. The approach outperforms a number of strong baselines on a subset of 12 Atari 2600 games.