Hyperbolic Discounting and Learning Over Multiple Horizons

1 INTRODUCTION . The standard treatment of the reinforcement learning ( RL ) problem is the Markov Decision Process ( MDP ) which includes a discount factor 0 ≤ γ ≤ 1 that exponentially reduces the present value of future rewards ( Bellman , 1957 ; Sutton & Barto , 1998 ) . A reward rt received in t-time steps is devalued to γtrt , a discounted utility model introduced by Samuelson ( 1937 ) . This establishes a timepreference for rewards realized sooner rather than later . The decision to exponentially discount future rewards by γ leads to value functions that satisfy theoretical convergence properties ( Bertsekas , 1995 ) . The magnitude of γ also plays a role in stabilizing learning dynamics of RL algorithms ( Prokhorov & Wunsch , 1997 ; Bertsekas & Tsitsiklis , 1996 ) and has recently been treated as a hyperparameter of the optimization ( OpenAI , 2018 ; Xu et al. , 2018 ) . However , both the magnitude and the functional form of this discounting function establish priors over the solutions learned . The magnitude of γ chosen establishes an effective horizon for the agent of 1/ ( 1− γ ) , far beyond which rewards are neglected ( Kearns & Singh , 2002 ) . This effectively imposes a time-scale of the environment , which may not be accurate . Further , the exponential discounting of future rewards is consistent with a prior belief that there is a known constant per-time-step hazard rate ( Sozou , 1998 ) or probability of dying of 1− γ ( Lattimore & Hutter , 2011 ) . Additionally , discounting future values exponentially and according to a single discount factor γ does not harmonize with the measured value preferences in humans1 and animals ( Mazur , 1985 ; 1997 ; Ainslie , 1992 ; Green & Myerson , 2004 ; Maia , 2009 ) . A wealth of empirical evidence has been amassed that humans , monkeys , rats and pigeons instead discount future returns hyperbolically , where dk ( t ) = 11+kt , for some positive k > 0 ( Ainslie , 1975 ; 1992 ; Mazur , 1985 ; 1997 ; Frederick et al. , 2002 ; Green et al. , 1981 ; Green & Myerson , 2004 ) . This discrepancy between the time-preferences of animals from the exponential discounted measure of value might be presumed irrational . But Sozou ( 1998 ) showed that hyperbolic time-preferences is mathematically consistent with the agent maintaining some uncertainty over the prior belief of the hazard rate in the environment . Hazard rate h ( t ) measures the per-time-step risk the agent incurs as it acts in the environment due to a potential early death . Precisely , if s ( t ) is the probability that the 1Time-preference reversals are one implication . Consider two hypothetical choices : ( 1 ) a stranger offers $ 1M now or $ 1.1M dollars tomorrow ( 2 ) a stranger instead offers $ 1M in 99 days versus $ 1.1M in 100 days . agent is alive at time t then the hazard rate is h ( t ) = − ddt lns ( t ) . We consider the case where there is a fixed , but potentially unknown hazard rate h ( t ) = λ ≥ 0 . The prior belief of the hazard rate p ( λ ) implies a specific discount function Sozou ( 1998 ) . Under this formalism , the canonical case in RL of discounting future rewards according to d ( t ) = γt is consistent with the belief that there exists a single hazard rate λ = e−γ known with certainty . Further details are available in Appendix A . Common RL environments are also characterized by risk , but often in a narrower sense . In deterministic environments like the original Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) stochasticity is often introduced through techniques like no-ops ( Mnih et al. , 2015 ) and sticky actions ( Machado et al. , 2018 ) where the action execution is noisy . Physics simulators may have noise and the randomness of the policy itself induces risk . But even with these stochastic injections the risk to reward emerges in a more restricted sense . In Section 2 we show that a prior distribution reflecting the uncertainty over the hazard rate , has an associated discount function in the sense that an MDP with either this hazard distribution or the discount function , has the same value function for all policies . This equivalence implies that learning policies with a discount function can be interpreted as making them robust to the associated hazard distribution . Thus , discounting serves as a tool to ensure that policies deployed in the real world perform well even under risks they were not trained under . We propose an algorithm that approximates hyperbolic discounting while building on successful Qlearning ( Watkins & Dayan , 1992 ) tools and their associated theoretical guarantees . We show learning many Q-values , each discounting exponentially with a different discount factor γ , can be aggregated to approximate hyperbolic ( and other non-exponential ) discount factors . We demonstrate the efficacy of our approximation scheme in our proposed Pathworld environment which is characterized both by an uncertain per-time-step risk to the agent . Conceptually , Pathworld emulates a foraging environment where an agent must balance easily realizable , small meals versus more distant , fruitful meals . We then consider higher-dimensional deep RL agents in the ALE , where we measure the benefits of hyperbolic discounting . This approximation mirrors the work of Kurth-Nelson & Redish ( 2009 ) ; Redish & Kurth-Nelson ( 2010 ) which empirically demonstrates that modeling a finite set of µAgents simultaneously can approximate hyperbolic discounting function . Our method then generalizes to other non-hyperbolic discount functions and uses deep neural networks to model the different Q-values from a shared representation . Surprisingly and in addition to enabling new non-exponential discounting schemes , we observe that learning a set of Q-values is beneficial as an auxiliary task ( Jaderberg et al. , 2016 ) . Adding this multi-horizon auxiliary task often improves over a state-of-the-art baseline , Rainbow ( Hessel et al. , 2018 ) in the ALE ( Bellemare et al. , 2013 ) . This work questions the RL paradigm of learning policies through a single discount function which exponentially discounts future rewards through the following contributions : 1 . Hazardous MDPs . We formulate MDPs with hazard present and demonstrate an equivalence between undiscounted values learned under hazards and ( potentially nonexponentially ) discounted values without hazard . 2 . Hyperbolic ( and other non-exponential ) -agent . A practical approach for training an agent which discounts future rewards by a hyperbolic ( or other non-exponential ) discount function and acts according to this . 3 . Multi-horizon auxiliary task . A demonstration of multi-horizon learning over many γ simultaneously as an effective auxiliary task . 2 HAZARD IN MDPS . To study MDPs with hazard distributions and general discount functions we introduce two modifications . The hazardous MDP now is defined by the tuple < S , A , R , P , H , d > . In standard form , the state space S and the action space A may be discrete or continuous . The learner observes samples from the environment transition probability P ( st+1|st , at ) for going from st ∈ S to st+1 ∈ S given at ∈ A . We will consider the case where P is a sub-stochastic transition function , which defines an episodic MDP . The environment emits a bounded reward r : S × A → [ rmin , rmax ] on each transition . In this work we consider non-infinite episodic MDPs . The first difference is that at the beginning of each episode , a hazard λ ∈ [ 0 , ∞ ) is sampled from the hazard distributionH . This is equivalent to sampling a continuing probability γ = e−λ . During the episode , the hazard modified transition function will be Pλ , in that Pλ ( s′|s , a ) = e−λP ( s′|s , a ) . The second difference is that we now consider a general discount function d ( t ) . This differs from the standard approach of exponential discounting in RL with γ according to d ( t ) = γt , which is a special case . This setting makes a close connection to partially observable Markov Decision Process ( POMDP ) ( Kaelbling et al. , 1998 ) where one might consider λ as an unobserved variable . However , the classic POMDP definition contains an explicit discount function γ as part of its definition which does not appear here . A policy π : S → A is a mapping from states to actions . The state action value function QH , dπ ( s , a ) is the expected discounted rewards after taking action a in state s and then following policy π until termination . QH , dπ ( s , a ) = EλEπ , Pλ [ ∞∑ t=0 d ( t ) R ( st , at ) |s0 = s , a0 = a ] ( 1 ) where λ ∼ H and Eπ , Pλ implies that st+1 ∼ Pλ ( ·|st , at ) and at ∼ π ( ·|st ) . 2.1 EQUIVALENCE BETWEEN HAZARD AND DISCOUNTING . In the hazardous MDP setting we observe the same connections between hazard and discount functions delineated in Appendix A . This expresses an equivalence between the value function of an MDP with a discount and MDP with a hazard distribution . For example , there exists an equivalence between the exponential discount function d ( t ) = γt to the undiscounted case where the agent is subject to a ( 1− γ ) per time-step of dying ( Lattimore & Hutter , 2011 ) . The typical Q-value ( left side of Equation 2 ) is when the agent acts in an environment without hazard λ = 0 or H = δ ( 0 ) and discounts future rewards according to d ( t ) = γt = e−λt which we denote as Qδ ( 0 ) , γ t π ( s , a ) . The alternative Q-value ( right side of Equation 2 ) is when the agent acts under hazard rate λ = − ln γ but does not discount future rewards which we denote as Q δ ( − ln γ ) ,1 π ( s , a ) . Qδ ( 0 ) , γ t π ( s , a ) = Q δ ( − ln γ ) ,1 π ( s , a ) ∀ π , s , a . ( 2 ) where δ ( x ) denotes the Dirac delta distribution at x . This follows from Pλ ( s′|s , a ) = e−λP ( s′|s , a ) Eπ , P [ ∞∑ t=0 γtR ( st , at ) |s0 = s , a0 = a ] = Eπ , P [ ∞∑ t=0 e−λtR ( st , at ) |s0 = s , a0 = a ] = Eπ , Pλ [ ∞∑ t=0 R ( st , at ) |s0 = s , a0 = a ] We also show a similar equivalence between hyperbolic discounting and the specific hazard distribution pk ( λ ) = 1k exp ( −λ/k ) , where again , λ ∈ [ 0 , ∞ ) in Appendix E. Qδ ( 0 ) , Γkπ ( s , a ) = Q pk,1 π ( s , a ) For notational brevity later in the paper , we will omit the explicit hazard distributionH-superscript if the environment is not hazardous . This formulation builds upon Sozou ( 1998 ) ’ s relate of hazard rate and discount functions and shows that this holds for generalized Q-values in reinforcement learning .

Summary: The paper investigates hyperbolic discounting as a more biologically plausible alternative to exponential discounting in reinforcement learning. First, it formulates a notion of hazard in MDPs as constant exponential discounting and shows that hyperbolic discounting is consistent with uncertainty over the hazard rate. The paper then shows how value functions learned with exponential discounting can be used to approximate value functions with other forms of discounting. Specifically, the paper shows in section 4 how exponentially-discounted value functions can be used to approximate hyperbolically discounted value functions. The paper then presents experiments on a small MDP and Atari 2600 games, showing that learning discounted action values with many different discount rates as an auxiliary task improves performance on most Atari games.