Temporal Difference Uncertainties as a Signal for Exploration

An effective approach to exploration in reinforcement learning is to rely on an agent ’ s uncertainty over the optimal policy , which can yield near-optimal exploration strategies in tabular settings . However , in non-tabular settings that involve function approximators , obtaining accurate uncertainty estimates is almost as challenging as the exploration problem itself . In this paper , we highlight that value estimates are easily biased and temporally inconsistent . In light of this , we propose a novel method for estimating uncertainty over the value function that relies on inducing a distribution over temporal difference errors . This exploration signal controls for state-action transitions so as to isolate uncertainty in value that is due to uncertainty over the agent ’ s parameters . Because our measure of uncertainty conditions on state-action transitions , we can not act on this measure directly . Instead , we incorporate it as an intrinsic reward and treat exploration as a separate learning problem , induced by the agent ’ s temporal difference uncertainties . We introduce a distinct exploration policy that learns to collect data with high estimated uncertainty , which gives rise to a “ curriculum ” that smoothly changes throughout learning and vanishes in the limit of perfect value estimates . We evaluate our method on hardexploration tasks , including Deep Sea and Atari 2600 environments and find that our proposed form of exploration facilitates efficient exploration . 1 INTRODUCTION . Striking the right balance between exploration and exploitation is fundamental to the reinforcement learning problem . A common approach is to derive exploration from the policy being learned . Dithering strategies , such as -greedy exploration , render a reward-maximising policy stochastic around its reward maximising behaviour ( Williams & Peng , 1991 ) . Other methods encourage higher entropy in the policy ( Ziebart et al. , 2008 ) , introduce an intrinsic reward ( Singh et al. , 2005 ) , or drive exploration by sampling from the agent ’ s belief over the MDP ( Strens , 2000 ) . While greedy or entropy-maximising policies can not facilitate temporally extended exploration ( Osband et al. , 2013 ; 2016a ) , the efficacy of intrinsic rewards depends crucially on how they relate to the extrinsic reward that comes from the environment ( Burda et al. , 2018a ) . Typically , intrinsic rewards for exploration provide a bonus for visiting novel states ( e.g Bellemare et al. , 2016 ) or visiting states where the agent can not predict future transitions ( e.g Pathak et al. , 2017 ; Burda et al. , 2018a ) . Such approaches can facilitate learning an optimal policy , but they can also fail entirely in large environments as they prioritise novelty over rewards ( Burda et al. , 2018b ) . Methods based on the agent ’ s uncertainty over the optimal policy explicitly trade off exploration and exploitation ( Kearns & Singh , 2002 ) . Posterior Sampling for Reinforcement Learning ( PSRL ; Strens , 2000 ; Osband et al. , 2013 ) is one such approach , which models a distribution over Markov Decision Processes ( MDPs ) . While PSRL is near-optimal in tabular settings ( Osband et al. , 2013 ; 2016b ) , it can not be easily scaled to complex problems that require function approximators . Prior work has attempted to overcome this by instead directly estimating the agent ’ s uncertainty over the policy ’ s value function ( Osband et al. , 2016a ; Moerland et al. , 2017 ; Osband et al. , 2019 ; O ’ Donoghue et al. , 2018 ; Janz et al. , 2019 ) . While these approaches can scale posterior sampling to complex problems and nonlinear function approximators , estimating uncertainty over value functions introduces issues that can cause a bias in the posterior distribution ( Janz et al. , 2019 ) . In response to these challenges , we introduce Temporal Difference Uncertainties ( TDU ) , which derives an intrinsic reward from the agent ’ s uncertainty over the value function . Concretely , TDU relies on the Bootstrapped DQN ( Osband et al. , 2016a ) and separates exploration and reward-maximising behaviour into two separate policies that bootstrap from a shared replay buffer . This separation allows us to derive an exploration signal for the exploratory policy from estimates of uncertainty of the reward-maximising policy . Thus , TDU encourages exploration to collect data with high model uncertainty over reward-maximising behaviour , which is made possible by treating exploration as a separate learning problem . In contrast to prior works that directly estimate value function uncertainty , we estimate uncertainty over temporal difference ( TD ) errors . By conditioning on observed stateaction transitions , TDU controls for environment uncertainty and provides an exploration signal only insofar as there is model uncertainty . We demonstrate that TDU can facilitate efficient exploration in challenging exploration problems such as Deep Sea and Montezuma ’ s Revenge . 2 ESTIMATING VALUE FUNCTION UNCERTAINTY IS HARD . We begin by highlighting that estimating uncertainty over the value function can suffer from bias that is very hard to overcome with typical approaches ( see also Janz et al. , 2019 ) . Our analysis shows that biased estimates arise because uncertainty estimates require an integration over unknown future state visitations . This requires tremendous model capacity and is in general infeasible . Our results show that we can not escape a bias in general , but we can take steps to mitigate it by conditioning on an observed trajectory . Doing so removes some uncertainty over future state-visitations and we show in Section 3 that it can result in a substantially smaller bias . We consider a Markov Decision Process ( S , A , P , R , γ ) for some given state space ( S ) , action space ( A ) , transition dynamics ( P ) , reward function ( R ) and discount factor ( γ ) . For a given ( deterministic ) policy π : S 7→ A , the action value function is defined as the expected cumulative reward under the policy starting from state s with action a : Qπ ( s , a ) : = Eπ [ ∞∑ t=0 γtrt+1 ∣∣∣∣∣ s0 = s , a0 = a ] = Er∼R ( s , a ) s′∼P ( s , a ) [ r + γQπ ( s ′ , π ( s′ ) ) ] , ( 1 ) where t index time and the expectation Eπ is with respect to realised rewards r sampled under the policy π ; the right-hand side characterises Q recursively under the Bellman equation . The actionvalue function Qπ is estimated under a function approximator Qθ parameterised by θ . Uncertainty over Qπ is expressed by placing a distribution over the parameters of the function approximator , p ( θ ) . We overload notation slightly and write p ( θ ) to denote the probability density function pθ over a random variable θ . Further , we denote by θ ∼ p ( θ ) a random sample θ from the distribution defined by pθ . Methods that rely on posterior sampling under function approximators assume that the induced distribution , p ( Qθ ) , is an accurate estimate of the agent ’ s uncertainty over its value function , p ( Qπ ) , so that sampling Qθ ∼ p ( Qθ ) is approximately equivalent to sampling from Qπ ∼ p ( Qπ ) . For this to hold , the moments of p ( Qθ ) at each state-action pair ( s , a ) must correspond to the expected moments in future states . In particular , moments of p ( Qπ ) must satisfy a Bellman Equation akin to Eq . 1 ( O ’ Donoghue et al. , 2018 ) . We focus on the mean ( E ) and variance ( V ) : Eθ [ Qθ ( s , a ) ] = Eθ [ Er , s′ [ r + γQθ ( s′ , π ( s′ ) ) ] ] , ( 2 ) Vθ [ Qθ ( s , a ) ] = Vθ [ Er , s′ [ r + γQθ ( s′ , π ( s′ ) ) ] ] . ( 3 ) If Eθ [ Qθ ] and Vθ [ Qθ ] fail to satisfy these conditions , the estimates of E [ Qπ ] and V [ Qπ ] are biased , causing a bias in exploration under posterior sampling from p ( Qθ ) . Formally , the agent ’ s uncertainty over p ( Q ) implies uncertainty over the MDP ( Strens , 2000 ) . Given a belief over the MDP , i.e. , a distribution p ( M ) , we can associate each M ∼ p ( M ) with a distinct value function QMπ . Lemma 1 below shows that , for p ( θ ) to be interpreted as representing some p ( M ) by push-forward to p ( Qθ ) , the induced moments must match under the Bellman Equation . Lemma 1 . If Eθ [ Qθ ] and Vθ [ Qθ ] fail to satisfy Eqs . 2 and 3 , respectively , they are biased estimators of EM [ QMπ ] and VM [ QMπ ] for any choice of p ( M ) . All proofs are deferred to Appendix B. Lemma 1 highlights why estimating uncertainty over value functions is so challenging ; while the left-hand sides of Eqs . 2 and 3 are stochastic in θ only , the right-hand sides depend on marginalising over the MDP . This requires the function approximator to generalise to unseen future trajectories . Lemma 1 is therefore a statement about scale ; the harder it is to generalise , the more likely we are to observe a bias—even in deterministic environments . This requirement of “ strong generalisation ” poses a particular problem for neural networks that tend to interpolate over the training data ( e.g . Li et al. , 2020 ; Liu et al. , 2020 ; Belkin et al. , 2019 ) , but the issue is more general . In particular , we show that factorising the posterior p ( θ ) will typically cause estimation bias for all but tabular MDPs . This is problematic because it is often computationally infeasible to maintain a full posterior ; previous work either maintains a full posterior over the final layer of the function approximator ( Osband et al. , 2016a ; O ’ Donoghue et al. , 2018 ; Janz et al. , 2019 ) or maintains a diagonal posterior over all parameters ( Fortunato et al. , 2018 ; Plappert et al. , 2018 ) of the neural network . Either method limits how expressive the function approximator can be with respect to future states , thereby causing an estimation bias . To establish this formally , letQθ : = w◦φϑ , where θ = ( w1 , . . . , wn , ϑ1 , . . . , ϑv ) , withw ∈ Rn a linear projection and φ : S ×A → Rn a feature extractor with parameters ϑ ∈ Rv . Proposition 1 . If the number of state-action pairs where Eθ [ Qθ ( s , a ) ] 6= Eθ [ Qθ ( s′ , a′ ) ] is greater than n , where w ∈ Rn , then Eθ [ Qθ ] and Vθ [ Qθ ] are biased estimators of EM [ QMπ ] and VM [ QMπ ] for any choice of p ( M ) . This result is a consequence of the feature extractor ψ mapping into a co-domain that is larger than the space spanned by w ; a bias results from having more unique state-action representations ψ ( s , a ) than degrees of freedom in w. The implication is that function approximators under factorised posteriors can not generalise uncertainty estimates across states ( a similar observation in tabular settings was made by Janz et al. , 2019 ) —they can only produce temporally consistent uncertainty estimates if they have the capacity to memorise point-wise uncertainty estimates for each ( s , a ) , which defeats the purpose of a function approximator . This is a statement about the structure of p ( θ ) and holds for any estimation method . Thus , common approaches to uncertainty estimation with neural networks generally fail to provide unbiased uncertainty estimates over the value function in non-trivial MDPs . Proposition 1 shows that to accurately capture value function uncertainty , we need a full posterior over parameters , which is often infeasible . It also underscores that the main issue is the dependence on future state visitation . This motivates Temporal Difference Uncertainties as an estimate of uncertainty conditioned on observed state-action transitions .

Summary: This paper proposes to use an intrinsic reward based on uncertainties calculated from temporal difference errors. The approach, called Temporal Difference Uncertainties (TDU), estimates the variance of td errors across multiple (bootstrapped) parameters, for a given state, action, next state and reward, where variability is due only to variance in parameters. The other addition is to learn a separate set of action-values that use this intrinsic reward, from the bootstrap set. Actions are then taken by randomly sampling an action-value function from the combined set.