Information-Directed Exploration for Deep Reinforcement Learning

1 INTRODUCTION . In Reinforcement Learning ( RL ) , an agent seeks to maximize the cumulative rewards obtained from interactions with an unknown environment . Given only knowledge based on previously observed trajectories , the agent faces the exploration-exploitation dilemma : Should the agent take actions that maximize rewards based on its current knowledge or instead investigate poorly understood states and actions to potentially improve future performance . Thus , in order to find the optimal policy the agent needs to use an appropriate exploration strategy . Popular exploration strategies , such as -greedy ( Sutton & Barto , 1998 ) , rely on random perturbations of the agent ’ s policy , which leads to undirected exploration . The theoretical RL literature offers a variety of statistically-efficient methods that are based on a measure of uncertainty in the agent ’ s model . Examples include upper confidence bound ( UCB ) ( Auer et al. , 2002 ) and Thompson sampling ( TS ) ( Thompson , 1933 ) . In recent years , these have been extended to practical exploration algorithms for large state-spaces and shown to improve performance ( Osband et al. , 2016a ; Chen et al. , 2017 ; O ’ Donoghue et al. , 2018 ; Fortunato et al. , 2018 ) . However , these methods assume that the observation noise distribution is independent of the evaluation point , while in practice heteroscedastic observation noise is omnipresent in RL . This means that the noise depends on the evaluation point , rather than being identically distributed ( homoscedastic ) . For instance , the return distribution typically depends on a sequence of interactions and , potentially , on hidden states or inherently heteroscedastic reward observations . Kirschner & Krause ( 2018 ) recently demonstrated that , even in the simpler bandit setting , classical approaches such as UCB and TS fail to efficiently account for heteroscedastic noise . In this work , we propose to use Information-Directed Sampling ( IDS ) ( Russo & Van Roy , 2014 ; Kirschner & Krause , 2018 ) for efficient exploration in RL . The IDS framework can be used to design exploration-exploitation strategies that balance the estimated instantaneous regret and the expected ∗Work done during exchange at ETH Zurich . information gain . Importantly , through the choice of an appropriate information-gain function , IDS is able to account for parametric uncertainty and heteroscedastic observation noise during exploration . As our main contribution , we propose a novel , tractable RL algorithm based on the IDS principle . We combine recent advances in distributional RL ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ) and approximate parameter uncertainty methods in order to develop both homoscedastic and heteroscedastic variants of an agent that is similar to DQN ( Mnih et al. , 2015 ) , but uses informationdirected exploration . Our evaluation on Atari 2600 games shows the importance of accounting for heteroscedastic noise and indicates that at our approach can substantially outperform alternative state-of-the-art algorithms that focus on modeling either only epistemic or only aleatoric uncertainty . To the best of our knowledge , we are the first to develop a tractable IDS algorithm for RL in large state spaces . 2 RELATED WORK . Exploration algorithms are well understood in bandits and have inspired successful extensions to RL ( Bubeck & Cesa-Bianchi , 2012 ; Lattimore & Szepesvári , 2018 ) . Many strategies rely on the ” optimism in the face of uncertainty ” ( Lai & Robbins , 1985 ) principle . These algorithms act greedily w.r.t . an augmented reward function that incorporates an exploration bonus . One prominent example is the upper confidence bound ( UCB ) algorithm ( Auer et al. , 2002 ) , which uses a bonus based on confidence intervals . A related strategy is Thompson sampling ( TS ) ( Thompson , 1933 ) , which samples actions according to their posterior probability of being optimal in a Bayesian model . This approach often provides better empirical results than optimistic strategies ( Chapelle & Li , 2011 ) . In order to extend TS to RL , one needs to maintain a distribution over Markov Decision Processes ( MDPs ) , which is difficult in general . Similar to TS , Osband et al . ( 2016b ) propose randomized linear value functions to maintain a Bayesian posterior distribution over value functions . Bootstrapped DQN ( Osband et al. , 2016a ) extends this idea to deep neural networks by using an ensemble of Qfunctions . To explore , Bootstrapped DQN randomly samples a Q-function from the ensemble and acts greedily w.r.t . the sample . Fortunato et al . ( 2018 ) and Plappert et al . ( 2018 ) investigate a similar idea and propose to adaptively perturb the parameter-space , which can also be thought of as tracking an approximate parameter posterior . O ’ Donoghue et al . ( 2018 ) propose TS in combination with an uncertainty Bellman equation , which propagates agent ’ s uncertainty in the Q-values over multiple time steps . Additionally , Chen et al . ( 2017 ) propose to use the Q-ensemble of Bootstrapped DQN to obtain approximate confidence intervals for a UCB policy . There are also multiple other ways to approximate parametric posterior in neural networks , including Neural Bayesian Linear Regression ( Snoek et al. , 2015 ; Azizzadenesheli et al. , 2018 ) , Variational Inference ( Blundell et al. , 2015 ) , Monte Carlo methods ( Neal , 1995 ; Mandt et al. , 2016 ; Welling & Teh , 2011 ) , and Bayesian Dropout ( Gal & Ghahramani , 2016 ) . For an empirical comparison of these , we refer the reader to Riquelme et al . ( 2018 ) . A shortcoming of all approaches mentioned above is that , while they consider parametric uncertainty , they do not account for heteroscedastic noise during exploration . In contrast , distributional RL algorithms , such as Categorical DQN ( C51 ) ( Bellemare et al. , 2017 ) and Quantile Regression DQN ( QR-DQN ) ( Dabney et al. , 2018b ) , approximate the distribution over the Q-values directly . However , both methods do not take advantage of the return distribution for exploration and use -greedy exploration . Implicit Quantile Networks ( IQN ) ( Dabney et al. , 2018a ) instead use a risksensitive policy based on a return distribution learned via quantile regression and outperform both C51 and QR-DQN on Atari-57 . Similarly , Moerland et al . ( 2018 ) and Dilokthanakul & Shanahan ( 2018 ) act optimistically w.r.t . the return distribution in deterministic MDPs . However , these approaches to not consider parametric uncertainty . Return and parametric uncertainty have previously been combined for exploration by Tang & Agrawal ( 2018 ) and Moerland et al . ( 2017 ) . Both methods account for parametric uncertainty by sampling parameters that define a distribution over Q-values . The former then act greedily with respect to the expectation of this distribution , while the latter additionally samples a return for each action and then acts greedily with respect to it . However , like Thompson sampling , these approaches do not appropriately exploit the heteroscedastic nature of the return . In particular , noisier actions are more likely to be chosen , which can slow down learning . Our method is based on Information-Directed Sampling ( IDS ) , which can explicitly account for parametric uncertainty and heteroscedasticity in the return distribution . IDS has been primarily studied in the bandit setting ( Russo & Van Roy , 2014 ; Kirschner & Krause , 2018 ) . Zanette & Sarkar ( 2017 ) extend it to finite MDPs , but their approach remains impractical for large state spaces , since it requires to find the optimal policies for a set of MDPs at the beginning of each episode . 3 BACKGROUND . We model the agent-environment interaction with a MDP ( S , A , R , P , γ ) , where S and A are the state and action spaces , R ( s , a ) is the stochastic reward function , P ( s′|s , a ) is the probability of transitioning from state s to state s′ after taking action a , and γ ∈ [ 0 , 1 ) is the discount factor . A policy π ( ·|s ) ∈ P ( A ) maps a state s ∈ S to a distribution over actions . For a fixed policy π , the discounted return of action a in state s is a random variable Zπ ( s , a ) = ∑∞ t=0 γ tR ( st , at ) , with initial state s = s0 and action a = a0 and transition probabilities st ∼ P ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) . The return distribution Z statisfied the Bellman equation , Zπ ( s , a ) D = R ( s , a ) + γZπ ( s′ , a′ ) , ( 1 ) where D= denotes distributional equality . If we take the expectation of ( 1 ) , the usual Bellman equation ( Bellman , 1957 ) for the Q-function , Qπ ( s , a ) = E [ Zπ ( s , a ) ] , follows as Qπ ( s , a ) = E [ R ( s , a ) ] + γEP , π [ Qπ ( s′ , a′ ) ] . ( 2 ) The objective is to find an optimal policy π∗ that maximizes the expected total discounted return E [ Zπ ( s , a ) ] = Qπ ( s , a ) for all s ∈ S , a ∈ A . 3.1 UNCERTAINTY IN REINFORCEMENT LEARNING . To find such an optimal policy , the majority of RL algorithms use a point estimate of the Q-function , Q ( s , a ) . However , such methods can be inefficient , because they can be overconfident about the performance of suboptimal actions if the optimal ones have not been evaluated before . A natural solution for more efficient exploration is to use uncertainty information . In this context , there are two source of uncertainty . Parametric ( epistemic ) uncertainty is a result of ambiguity over the class of models that explain that data seen so far , while intrinsic ( aleatoric ) uncertainty is caused by stochasticity in the environment or policy , and is captured by the distribution over returns ( Moerland et al. , 2017 ) . Osband et al . ( 2016a ) estimate parametric uncertainty with a Bootstrapped DQN . They maintain an ensemble ofK Q-functions , { Qk } Kk=1 , which is represented by a multi-headed deep neural network . To train the network , the standard bootstrap method ( Efron , 1979 ; Hastie et al. , 2001 ) constructs K different datasets by sampling with replacement from the global data pool . Instead , Osband et al . ( 2016a ) trains all network heads on the exact same data and diversifies the Q-ensemble via two other mechanisms . First , each headQk ( s , a ; θ ) is trained on its own independent target headQk ( s , a ; θ− ) , which is periodically updated ( Mnih et al. , 2015 ) . Further , each head is randomly initialized , which , combined with the nonlinear parameterization and the independently targets , provides sufficient diversification . Intrinsic uncertainty is captured by the return distribution Zπ . While Q-learning ( Watkins , 1989 ) aims to estimate the expected discounted return Qπ ( s , a ) = E [ Zπ ( s , a ) ] , distributional RL approximates the random return Zπ ( s , a ) directly . As in standard Q-learning ( Watkins , 1989 ) , one can define a distributional Bellman optimality operator based on ( 1 ) , T Z ( s , a ) D : = R ( s , a ) + γZ ( s′ , arg max a′∈A E [ Z ( s′ , a′ ) ] ) . ( 3 ) To estimate the distribution of Z , we use the approach of C51 ( Bellemare et al. , 2017 ) in the following . It parameterizes the return as a categorical distribution over a set of equidistant atoms in a fixed interval [ Vmin , Vmax ] . The atom probabilities are parameterized by a softmax distribution over the outputs of a parametric model . Since the parameterization Zθ and the Bellman update T Zθ have disjoint supports , the algorithm requires an additional step Φ that projects the shifted support of T Zθ onto [ Vmin , Vmax ] . Then it minimizes the Kullback-Leibler divergence DKL ( ΦT Zθ||Zθ ) .

Summary: This paper investigates sophistical exploration approaches for reinforcement learning. Motivated by the fact that most of bandit algorithms do not handle heteroscedasticity of noise, the authors built on Information Direct Sampling and on Distributional Reinforcement Learning to propose a new exploration algorithm family. Two versions of the exploration strategy are evaluated against the state-of-the-art on Atari games: DQN-IDS for homoscedatic noise and C51-IDS for heteroscedastic noise.