Successor Uncertainties: exploration and uncertainty in temporal difference learning

1 Introduction . We consider a sequential decision making problem in which an agent interacts with an unknown environment , modelled as a Markov Decision Process ( MDP ) . The agent ’ s goal is to learn a policy that maximises the expected cumulative reward while keeping the total number of interactions with the environment low . To do so , the agent has to balance exploration of the environment and exploitation of the information it has already acquired . Because the agent learns sequentially through interaction with a potentially stochastic environment , the consequences of each action are not necessarily known ahead of time and this complicates striking the optimal trade-off between exploration and exploitation . A particular strategy to address this issue is maintaining a probabilistic model that reflects agent ’ s current knowledge , or lack thereof , about the future effects of each action for any given state . Vitally , the model must incorporate uncertainty about not only the immediate but also the downstream effects of agent ’ s actions , and reflect the varying degree of uncertainty associated with different sequences of states encountered and actions taken by the agent . We will therefore focus on two concepts we believe to be crucial for effective exploration under uncertainty : propagation of uncertainty and incorporation of dependencies . We will say that a probabilistic model of the Q function propagates uncertainty whenever the degree of uncertainty over the Q value for each state-action pair is determined by the cumulative uncertainty of all future state-actions the agent may encounter . Notice that for any joint distribution of the Q function that propagates uncertainty , the model obtained by taking the product of its marginal distributions over Q values also propagates uncertainty . In other words , propagation of uncertainty does not guarantee that any dependencies between individual Q values , like the one implied by the Bellman equation , are captured by the model . To this end , we will say that a model incorporates dependencies if the dependency structure of individual Q values obeys the Bellman equation . Further details are provided in section 3 . We illustrate importance of propagation of uncertainties and incorporation of dependencies in figure 1 . The agent is faced with a sequence of binary up/down choices and receives a positive reward if and only if it executes a particularly long sequence of uninterrupted up actions . In such a scenario , purely random exploration is extremely ineffective . Similarly , exploration which considers just immediate consequences of actions also performs poorly — shortsightedly reducing one-step uncertainty leads to exploration equivalent in expectation to that of a uniform exploration policy ( more on this in section 6 ) . In contrast , a model which propagates uncertainty and incorporates dependencies can be used to guide the agent into under-explored areas irrespective of their distance from the current state , resulting in more effective exploration of the environment . Whilst simple , this MDP captures the key features of classic exploration benchmarks like Montezuma ’ s Revenge ( Bellemare et al. , 2013 ) , where exploration is hindered by sparse reward and low connectivity of the state-space . The main contributions of our work are summarised below : 1 . Identification of propagation of uncertainty and incorporation of uncertainties as desirable attributes of models used for probabilistic exploration ; 2 . Introduction of Successor Uncertainties , a probabilistic model which both propa- gates uncertainty and incorporates dependencies ; 3 . Empirical investigation demonstrating how lack of either propagation of uncertainty or incorporation of dependencies can lead to inefficient exploration on classic exploration benchmarks ( Dearden et al. , 1998 ) . Experiments on the Atari benchmark suite ( Bellemare et al. , 2013 ) confirming that Successor Uncertainties can be scaled to work in complex domains . 2 Background . Before proceeding , we briefly review model-free reinforcement learning and Bayesian linear regression . These are key concepts we shall use throughout the remainder of our paper . Markov Decision Process We consider a finite MDP with a horizon H ∈ N , defined on a probability space ( Ω , F , P ) . An MDP is a tuple ( S , A , T ) , where S is a finite state space , A a finite action space , and T : S × A → P ( S × R ) a transition probability kernel with R ⊂ R a bounded reward set , and P ( · ) the set of probability measures on given space . For each time step t ∈ N , the agent selects an action At by sampling from a distribution specified by the policy πt : S → P ( A ) for the current state St , and receives a new state and reward ( St+1 , Rt+1 ) ∼ T ( St , At ) . This gives rise to a random process ( St , At ) t≥0 , Markov with respect to its natural filtration ( Ft ) t≥0 , and a sequence of random variables ( Rt ) t≥1 . The task of solving an MDP is that of finding an optimal policy π = ( πt ) 0≤t≤H that maximises the expected return Jπ = E [ ∑H τ=0 γ τRτ+1 ] where γ ∈ [ 0 , 1 ) is a discount factor . Model-free reinforcement learning All exploration methods we consider repose on a model-free approach to solving MDPs . These approaches invariably use estimates of state-action values and the corresponding Q function , Qπt = E [ ∑H τ=t γ τ−tRτ+1 | Ft ] , to guide exploration . Because computation of Qπt is usually intractable , it is estimated for each ( s , a ) ∈ S × A using a ( stationary ) parametric model Q̂π : S ×A → R. This model is trained to obey the Bellman equation , Qπt = E [ Rt+1 | Ft ] + γE [ Qπt+1 | Ft ] , by minimising a monotonically increasing function of the temporal difference error ` td : ( s , a , s′ , r , π ) 7→ |Q̂π ( s , a ) − r − γEa′∼π ( s′ ) [ Q̂π ( s′ , a′ ) ] | on the data collected through exploration . The result of this procedure is an estimate of the optimal policy defined by greedy behaviour with respect to Q̂πt , that is π ? : s 7→ |A|−1δA with A = { a : Q̂π ( s , a ) = maxa′∈A Q̂π ( s , a′ ) } and |A| its cardinality . Bayesian linear regression The goal of linear regression is to fit a model of the form r = 〈φ , w〉 using a data set D = { ( φi , ri ) } ti=0 . We will treat estimation of w in a fully Bayesian manner , assuming a multivariate normal prior Pw = N ( 0 , Σp ) for w and a Gaussian likelihood Pr|φ , w = N ( 〈φ , w〉 , β ) . By application of the Bayes ’ rule , the posterior over the weights given D is then Pw|D = N ( µw , Σw ) . Here Σw = ( Σp + β−1 ∑t i=0 φiφ T i ) −1 and µw = Σw ( ∑t i=0 riφi ) . Both µw and Σw can be computed in an online fashion . 3 Probabilistic exploration in reinforcement learning . One of the most popular approaches to probabilistic exploration is posterior sampling ( Dearden et al. , 1998 ; Strens , 2000 ) , also known as Thompson sampling ( Thompson , 1933 ) . The key feature of this method is that a distribution PQ̂π over Q̂π is maintained instead of a point estimate . Exploration then proceeds by iterating the following steps : ( i ) sample Q̂π ∼ PQ̂π ; ( ii ) explore the environment using the greedy policy with respect to the sampled Q̂π ; ( iii ) update PQ̂π based on the collected data . Effectiveness of posterior sampling is determined by properties of PQ̂π . We will focus on two properties that we believe to be of vital importance . 1 . Propagation of uncertainty : the marginal distribution over Q̂π ( s , a ) must reflect the total downstream uncertainty about all Q̂π ( s′ , a′ ) that may be encountered when taking action a in state s and then following policy π. Propagation of uncertainty ensures that uncertainty at any step is connected to expected uncertainties at subsequent time-steps ( O ’ Donoghue et al. , 2018 ) . It is sufficient for approaches based on marginal statistics , for example upper confidence bound exploration ( Auer et al. , 2002 ) , but does not guarantee coherence of individual samples . 2 . Incorporation of dependencies : the set of random variables { Q̂π ( s , a ) } s∈S , a∈A must ( almost surely ) obey dependencies specified by the Bellman equation . Posterior sampling uses greedy policy with respect to a single sample from PQ̂π . It is thus crucial that the value at any time-step is connected to the expected value at subsequent time-steps for every such sample , not only in expectation . If a sample which does not satisfy the Bellman equation is used , individual sampled Q values may carry conflicting information at each time-step . Importantly for our discussion later , a model PQ̂π propagates uncertainty if it incorporates dependencies but the converse is not necessarily true . 4 Uncertainty propagation through Successor Features . We present a new method , Successsor Uncertainties , which employs a probabilistic model PQ̂π that does not assume a factorisation of the posterior over the state-action space . The model both propagates uncertainties and incorporates dependencies , and thus leads to temporally correlated exploration when combined with posterior sampling . 4.1 The Successor Uncertainty model . Suppose that an embedding φ : S × A → Rd+ exists , d ∈ N , such that ||φ||2 = 1 and E [ Rt+1 | Ft ] = 〈φt , w〉 for some w in Rd and φt = φ ( St , At ) . Then Qπt = E [ H∑ τ=t γτ−tRτ+1 | Ft ] = E [ H∑ τ=t γτ−t〈φτ , w〉 | Ft ] = 〈 Eπ [ H∑ τ=t γτ−tφτ | Ft ] , w 〉 , where the second equality follows from the tower property of conditional expectation and the third is a simple application of the dominated convergence theorem . We define ψπt = E [ ∑H τ=t γ τ−tφτ | Ft ] which allows us to express the Q function succinctly in terms of this quantity as Qπt = 〈ψπt , w〉 . The random vectors ψπt are known as successor features in the literature ( Dayan , 1993 ; Barreto et al. , 2017 ) , and correspond to the ( discounted ) expected future occurrence of each feature when following a given policy π . We develop the Successor Uncertainties model for Qπt , denoted by Q̂πSF , by modelling the quantities w and ψπt . We use a stationary estimator of the successor features ψ̂π which will be obtained by noting that ψπt = φt + γ E [ ψπt+1 | Ft ] , meaning any standard temporal difference approach applies . We perform Bayesian linear regression to infer a distribution over w using a Gaussian prior N ( 0 , αI ) and likelihood N ( 〈φ , w〉 , β ) . This makes the computation of the posterior over w given D , Pw|D = N ( µw , Σw ) , with D the set of all so far observed states and rewards , analytically tractable . Finally , stacking all successor feature estimates into a matrix Ψ̂π = [ ψ̂π ( s , a ) ] > s∈S , a∈A , the implied distribution over Q̂πSF can be shown to be Law ( Q̂πSF ) = N ( Ψ̂πµw , Ψ̂πΣw ( Ψ̂π ) > ) , highlighting the dependence between individual Q̂πSF ( s , a ) introduced by our construction . A limitation of Successor Uncertainties is that only the rewards are explicitly treated in a probabilistic way , whereas a point estimate is used for the transition model in construction of ψ̂π . This issue may be partially alleviated following an argument analogous to the one made by O ’ Donoghue et al . ( 2018 ) . Specifically , if we are in a tabular setting and φ ( s , a ) are one-hot encoded representations of individual state-actions , the posterior variance Σw will be diagonal with β ( nsa + βα ) −1 for its non-zero entries , where nsa the number of times the tuple ( s , a ) has been observed . Combined with the above derived Law ( Q̂πSF ) , the marginal variance over Q̂πSF ( s , a ) will be proportional to βn−1sa which will decrease at the same rate as if we had incorporated a Dirichlet model over the transitions ( O ’ Donoghue et al. , 2018 ) . Thus , if α = β and β is treated as a hyperparameter ( as in our experiments ) , setting β sufficiently high can adjust our estimate for the uncertainty about the transition model . As in ( O ’ Donoghue et al. , 2018 ) , a similar argument can be made in terms of pseudocounts ( Bellemare et al. , 2016 ) for linear Q function models . Finally , our derivations show that ψπ needs to be relearnt for each policy π which would generally require a significant amount of computational resources . To address this issue , we instead learn ψπ̄ ? where π̄ ? = ∫ π ? wdPw|D is the average over policies π ? w ; here π ? w is the greedy policy with respect to the particular Q function sample determined by w. This approximation reduces coherence of the dependencies within each Q function sample , but allows the algorithm to scale to large applications .

Summary: The paper proposes an exploration approach (either based on posterior sampling or optimism) based on successor features representation. A high probability ellipsoid confidence set (defined by the Gram matrix \Sigma_t) is estimated based on linear regression (using some features \phi) to the immediate reward function. Now for any policy \pi, a confidence interval for the Q-value of any policy \pi can be derived by application of the \psi transformation, where \psi = expected sum of discounted future \phi under \pi. The algorithm selects action by posterior sampling (or UCB) in the \psi-space.