Hyperbolic Discounting and Learning Over Multiple Horizons
Authors: William Fedus
Carles Gelada
Yoshua Bengio
Marc G. Bellemare
Hugo Larochelle
Venue:  (2020)
Keywords: Deep learning
reinforcement learning
discounting
hyperbolic discounting
auxiliary tasks

Abstract: Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process.  The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences.  Here we extend earlier work of Kurth-Nelson and Redish and propose an efficient deep reinforcement learning agent that acts via hyperbolic discounting and other non-exponential discount mechanisms. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL.  Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over state-of-the-art methods.

Under review as a conference paper at ICLR 2020
HYPERBOLIC DISCOUNTING AND LEARNING OVER MULTIPLE HORIZONS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) typically defines a discount factor (r) as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bell- man equation. However, evidence from psychology. economics and neuroscience suggests that humans and animals instead have hyperbolic time- ume-preferences ( (ItR) for 0). Here we extend earlier work of Kurth-Nelson and Redish and propose efficient deep reinforcement learning agent that acts via hyperbolic discounting and other non-exponential discount mechanisms. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar learning techniques in RL. Additionally, and independent hyperbolic discounting, we make a surprising discovery that simultaneously learn- ing value functions over multiple time-horizons is an effective auxiliary task which often improves over sfate-of-the-art methods.
INTRODUCTION
The standard treatment of the reinforcement learning (RL) problem is the Markov Decision Process (MDP) which includes a discount factor 0 < y < that exponentially reduces the present value of future rewards (Bellman) 1957, Sutton & Barto 1998). A reward r, received 1-time steps is devalued to 7'r,, discounted utility model introduced by Samuelson (1977). This establishes a time- preference for rewards realized sooner rather than later. The decision to exponentially discount future rewards by 7 leads to value functions that satisfy theoretical convergence properties (Bertsekas) 1995). The magnitude of y also plays a role in stabilizing learning dynamics of RL algorithms (Prokhorov & Wunsch 1997 Bertsekas & Tsitsiklis 1996) and has recently been treated a hyperparameter of the optimization OpenAA 2018; et al. (2018).
However, both the magnitude and the functional form of this discounting function establish priors over the solutions learned. The magnitude of y chosen establishes an effective horizon for the agent of 1/(1 (), far beyond which rewards are neglected (Kearns & Singh 2002). This effectively imposes a time-scale of the environment, which may not be accurate. Further, the exponential discounting of future rewards is consistent with prior belief that there is known constant per-time-step hazard rate (Sozou 1998) probability of dying of - (Lattimore & Hutter 2011
Additionally, discounting future values exponentially and according single discount factor does not harmonize with the measured value preferences in humans" and animals (Mazur 1985
1997. Ainslie 1992, Green & Myerson 2004 Maia 2009). wealth of empirical evidence has
been amassed that humans, monkeys, rats and pigeons instead discount future returns hyperbolically.
where (t)
T for some positive & > 0 (Ainslic 1975 1992 Mazur 1985 1997 Frederick
al. 2002 Green et al. 1981 Green & Myerson (2004).
This discrepancy between the time- teeeeerefeeeccss of animals from the exponential discounted measure of value might be presumed irrational. [Stooo (1998) showed that hyperbolic emee-rreerenees is mathematically consistent with the agent maintaining some uncertainty over the prior belief of the hazard rate in the environment. Hazard rate h(t) measures the per-time-step risk the agent incurs as it acts in the environment due to potential early death. Precisely, if s()) is the probability that the
'Time-preffe preference reversals are one implication. Consider two hypothetical choices: (1) stranger offers $1M now $1.1M dollars tomorrow (2) a stranger instcad offers S1M in 99 days versus $1.1M 1M in 100 days.

---

Under review as a conference paper at ICLR 2020
agent is alive at time t then the hazard rate is h(t) d Ins)... We consider the case where there is
de
a fixed, but potentially unknown hazard rate h(t) = À 2 0. The prior belief of the hazard rate p()) implics specific discount function Sozou (1998). Under this formalism, the canonical case in RL of discounting future rewards according to d() 7f is consistent with the belief that there exists a
single hazard rate À = e known with certainty. Further details are available in Appendix A
Common RL environments are also character- ized by risk, but often in a narrower sense. In deterministic environments like the original Ar- cade Learning Environment (ALE) (Bellemare it al.) 2013) stochasticity is often introduced through techniques like no-ops (Mnih et al. 2015) and sticky actions (Machado et al. 2018)
Hyperbolic versus Exponential Discounting
10
discount function hyperbolic (--1.8)
Discount Rate
where the action execution is noisy. Physies sim- ulators may have noise and the randomness of the policy itself induces risk. But even with these stochastic injections the risk to reward emerges in a more restricted sense. In Section 2 we show that a prior distribution reflecting the uncertainty over the hazard rate, has an as- sociated discount function in the sense that an MDP with either this hazard distribution or the discount function, has the same value function for all policies. This equivalence implies that learning policies with a discount function can be interpreted as making them robust to the as- sociated hazard distribution. Thus, discounting serves tool to ensure that policies deployed in the real world perform well even under risks they were not trained under.
0.4
0.2
0.0
200
400
660
800
1009
Time delay
Figure I: Hyperbolic versus exponential discount- ing. Humans and animals often exhibit hyperbolic discounts (blue curve) which have shallower dis- count declines for large horizons. In contrast, RL agents often optimize exponential discounts (or- ange curve) which drop at constant rate regard- less of how distant the return.
We propose an algorithm that approximates hyperbolic discounting while building on successful Q- learning (Watkins & Dayan 1992) tools and their associated theoretical guarantees. We show learning many Q-values, each discounting exponentially with a different discount factor y, can be aggregated approximate hyperbolic (and other non-exponential) discount factors. We demonstrate the efficacy of our approximation scheme in our proposed Pathworld environment which is characterized both by an uncertain ver-time-step risk the agent. Conceptually, Pathworld emulates a foraging environment where an agent must balance easily realizable, small meals versus more distant, fruitful meals. We then consider gigher-dimensional deep RL. agents in the ALE, where we measure the benefits of hyperbolic discounting. This approximation mirrors the work of Kurth-Nelson & Redish (2009); Redish& & Kurth-Nelson (2010) which empirically demonstrates that modeling a finite set of jAAgents simultaneously can approximate hyperbolic discounting function. Our method then generalizes to other non-hyperbolic discount functions and uses deep neural networks to model the different Q-values from shared representation.
Surprisingly and in addition to enabling new non-exponential discounting schemes, we observe that learning a set of Q-values is beneficial as an auxiliary task (Jaderberg et al. 2016). Adding this multi-horizon auxiliary task often improves over a state-of-the-ard baseline, Rainbow (Hessel) et al. (2018) in the ALE (Bellemare al. 2013). This work questions the RL paradigm of learning policies through single discount function which exponentially discounts future rewards through the following contributions:
1. Hazardous MDPs. We formulate MDPs with hazard present and demonstrate an equivalence between undiscounted values learned under hazards and (potentially non- exponentially) discounted values without hazard.
2. Hyperbolic (and other non-exponential--geem A practical approach for training an agent which discounts future rewards by a hyperbolic (or other non-exponential) discount function and acts according to this.
3. Multi-horizon auxiliary task. A demonstration of multi-horizon learning over many ? simultaneously effective auxiliary task.
2

---

Under review as a conference paper at ICLR 2020
2 HAZARD IN MDPS
To study MDPs with hazard distributions and general discount functions we introduce two modifica- tions. The hazardous MDP now defined by the tuple < S,A, R. P. H. >. standard form, the state space S and the action space A may be discrete or continuous. The learner observes samples from the environment transition probability P( [s,, at) for going from 8,
8t+1 given
az A. We will consider the case where P is a sub-stochastic transition function, which defines an episodic MDP. The environment emits a bounded reward r : A [Tmin; Tmar] on each transition. In this work consider non- non-infinite episodic MDPs.
The first difference that at the beginning of each episode, hazard X E [0, oo) is sampled from the hazard distribution H. This is equivalent sampling a continuing probability 7 e During the episode, the hazard modified transition function will be Px. in that Px(s'|s, |s, a) = 6 P>s''' a). The second difference is that we consider general discount function d(t). This differs from the standard approach of exponential discounting in RL with 7 according to d(t) = 7', which is a special case. This setting makes close connection partially observable Markov Decision Process (POMDP) (Kaelbling et al., 1998) where one might consider À as an unobserved variable. However, the classic POMDP definition contains an explicit discount function y as part of its definition which does not appear here.
policy क S + A a mapping from states to actions. The state action value function Q7 is the expected discounted rewards after taking action in state s and then following policy until termination.
(s, a)
as, = E, Σ d(t, = s, 5, = =a
(I)
where x ~ H and E, implies that $141 (x at) and ~ ((1stt).
2.1 EQUIVALENCE BETWEEN HAZARD AND DISCOUNTING
In the hazardous MDP setting we observe the same connections between hazard and discount functions delineated in Appendix A This expresses an equivalence between the value function of an MDP with discount and MDP with a hazard distribution.
For example. there exists an equivalence between the exponential discount function d(t) = 7 to the undiscounted case where the agent is subject to a (1 1 7) per time-step of dying (Lattimore & Hutter 2011). The typical Q-value (left side of Equation 2 is when the agent acts in an environment without hazard X = 0 or H = 8(0) and discounts future rewards according to dd()) = 7 e -
which we denote as (00)) (s, a). The alternative Q-value (right side of Equation 2) is when the In y but does not discount future rewards which we denote as
agent acts under hazard rate Q In (s, a).
00 (s, a) Q (on--mnn).l in a) V s,
(2)
where (xx) denotes the Dirac delta distribution x. This follows from Px(s'(s, = =e
80
E P
y4 (((s,, 8, ag = a E,T.
XI R(s),a)))s0 8,00 =
En
Σ
We also show similar equivalence between hyperbolic discounting and the specific hazard distribu- tion Pke (A) = [exp(->/k). where again, À E [0, 2o) in Appendix
g (s. Qu.. (s.
For notational brevity later in the paper, will omit the explicit hazard distribution H- H-superseript if the environment is not hazardous. This formulation builds upon Sozou (1998)'s relate of hazard rate and discount functions and shows that this holds for generalized Q-values in reinforcement learning.
3

---

Under review as a conference paper at ICLR 2020
3 COMPUTING NON-EXPONENTIAL Q- D-VLLU
We now show how one can re-purpose exponentially-discoomee Q-values to compute hyperbolic (and other--non-expoeentaa) discounted Q-values. The central challenge with using non-exponential discount strategies that most RL algorithms use some form of TD learning (Sutton 1988). This family of algorithms exploits the Bellman equation (Bellman 1958) which, when using exponential discounting, relates the value function one state with the value the following state.
D (s,a) a) = Ex, p( + (n
(3)
where expectation E denotes sampling a TT-(s), P(-|s, a), and a' (-1ss'). Being able to reuse TD methods without being constrained to exponential discounting is thus an important challenge. We propose here scheme deduce hyperbolic as well as other con-exponentialy discounted Q-values when our discount function has particular form.
Lemma 3.1. Let (x, a) be the state action value function under exponential discounting in hazardous MDP < A, R, P. H, H > and let HR. (s, a) refer to the value function in the same
MDP except for new discounting S, S, R. P. H. d If there exists a function that
[0, 1] - R such
d(t) = -[- w(7))d
(4)
which we will refer to the exponential weighting condition. then
(s, S(" (s,
(5)
Proof. Applying the condition on d,
20
H. -EXET.PA
Σ
= 0, =a a
(6)
ExEm, w())
Σ R(sssa)))ss 0.00
d7
= //' ((17)
(8)
The exchange in the above proof is valid if Σ R((s, at) < oo. The exponential weighting condition is satisfied for hyperbolic discounting and other discounting that might want to consider (see Appendix|B [F] for examples). As an example, the hyperbolic discount can be expressed as the integral of function (((,,)) for 1) in Equation
1
1lkktt dy
k
1 kt
- w(>))' yields the desired hyperbolic discount factor (t) 1+k This integral can be derived by Sozou's Laplace transform
This equationn tells an integral over a function ((),t)= = -/- =
of the hazard rate prior H. = p(A) in Equation 18 and then applying our change of variables 7 = e ~ relating RL discount factors to hazard rates. The computation of hyperbolic and other discount
functions is demonstrated detail in Appendix
This prescription gives tool to produce general forms of con-exponnttially discounted Q-values using our familiar exponentially discounted Q-values traditionally learned in RL (Sutton 1988 Sutton & Barto. 1998).
4

---

Under review as a conference paper at ICLR 2020
4 APPROXIMATING HYPERBOLIC Q- V-LUUS
Section S describes an equivalence between ahperbllialllyddscounnee Q-values and integrals of ssponentially-discuuntie Q-values, however, the method required evaluating infinite set of value functions. We therefore present practical approach to approximate discounting I'(t) I4Ki using finite set of functions learned via standard Q-learning (Watkins & Dayan 1992p. To avoid estimating an infinite number of Q2-values of Q Qaavalues to consider, each with their own 7.. We use a gracticallymmineed approach to choose G that emphasizes evaluating larger values of y rather than uniformly choosing points and empirically performs well as seen in Section
introduce free hyperparameter (n-,) which is the total number
g [70, 71.-. 7m,
(10)
Our approach is described in Appendix G Each Q% computes the discounted sum of returns according that specific discount factor Q% (s, a) E (7:)
8, do a]. We previously proposed two equivalent approaches for computing hyperbolic Q-values, but for simplicity we consider the one presented in Lemma 3.1 The set of Q-values permits us to estimate the integral through Riemann
sum (Equation I1) which is described in further detail in Appendix
(s) = =/0 w(7))2(s,add)
(11)
Σ (141 w(T) (s,a) MEG
(12)
where we estimate the integral through a lower bound. We consolidate this entire process Figure 11 where we show the full process of rewriting the hyperbolic discount rate, hyperbolically-disoom Q-value, the approximation and the instantiated agent. This approach is similar to that of Kurth Nelson & Redish 2009) where each μAgent models specific discount factor yy However, this differs in that our final agent computes a weighted average over each Q-value rather than a sampling operation of each agent based on y-distribution.
HYPERBOLIC RESULTS
5.1 WHEN TO DISCOUNT HYPERBOLICALLY?
The benefits of hyperbolic discounting will be greatest under two conditions: uncertain hazard and non-trivial intertemporal decisions. The first condition can arise under a unobserved hazard-rate variable X drawn independently at the beginning of each episode from H = p(A). The second condition emerges with choice between smaller nearby rewards versus larger distant rewards In the absence of both properties we would not expect any advantage to discounting hyperbolically. To see why, if there is single-true hazard rate Asnv than an optimal 7" = e Aan exists and future rewards should be discounted exponentially according to it. Further, if there is a single path through the environment with perfect alignment of short- and long-term objectives, all discounting schemes yield the same optimal policy.
5.2 PATHWORLD EXPERIMENTS
We note two sources for discounting rewards in the future: time delay and survival probability (Section 2). In Pathworld we train to maximize hyperbolically discounted returns (E, [((t) dz)) under no hazard (Ht- = 5(²- 0)) but then evaluate the undiscounted returns d((t) = 1.0 t with the paths subject hazard H - Texp(-A/k))
Through this procedure, we are able to train an agent that is robust to hazards in the environment. The agent makes one decision in Pathworld (Figure 2 which of the N paths investigate. Once a path is chosen, the agent continues until it reaches the end or until it dies. This is similar to multi-armed bandit, with each action subject dynamic risk. The paths vary quadratically in length with the index d(i)= ,2 but the rewards increase linearly with the path index r()) i. This presents
A trivial intertemporal decision is one between small distant rewards versus large close rewards. For example, the choice between $100 now versus $10 tomorrow.
5

---

Under review as a conference paper at ICLR 2020
=
9
a. p
r=2,=4
r= 1,d=1
The Pathworld. Each state (white circle) indicates the accompanying reward r and the distance from the starting state d. From the start state, the agent makes single action: which which path to follow to the end. Longer paths have a larger rewards the end, but the agent incurs a higher risk on a longer path.
Figure 2:
non-trivial decision for the agent. At deployment, unobserved hazard X H. is drawn and the agent subject to a per-time-step risk dying (1 ^-^). This environment differs from the adjusting-delay procedure presented by Mazur (1987) and then later modified by Kurth-Nelson & Redish (2009). Rather then determining time- snee-prfferences through criable-timing of rewards, we determine time-preferences through risk the reward.
4.5
Hyperbolic vs. Exponential Value Est.
Discount function
MSE
beareccalva hypertol: value
hyperbolic value
0.002
Expected value El
garme
0=9975
0.566
7=0.95
1.461
2.5
2.0
7=0,9
2.253
2.5
7=0.99
2.288
3.0
0=0.75
2.809
20
12 14
Path index
Figure 3: In each episode of Pathworld an unobserved
Table 1:
The average mean squared error (MSE) over each of the paths in Figure showing that our approximation scheme well-approximates the true value-
hazard p(A) is drawn and the agent is subject to
total risk of the reward not being realized of (1
e e-Ayd(a) where d((a) is the path length. When the agent's s
hazard prior matches the true hazard distribution, the
value estimate agrees well with the theoretical value. Exponentially discounted values fail approximate the true value (Table I..
profile.
Figure validates that our approach well-apprrximmtes the true hyperbolic value of each path when the hazard prior matches the true distribution. Agents that discount exponentially according to a single y (the typical case RL) incorrectly value the paths. We examine further the failure of exponential discounting in this hazardous setting. For this environment, the true hazard parameter in the prior was & 0.05 (i.e. X 20exp(-X/00
Therefore, deployment, the agent must deal with dynamic levels of risk and faces non-trivial decision of which path to follow. Even if we tune an agent's ss = 0.975 such that chooses the correct arg-max path, still fails to capture the functional form (Figure 3) and it achieves high error over all paths (Table []. If the arg-mas action was not available or the agent was proposed to evaluate non non-trivial intertemporal decisions, it would act sub-optimally. In Appendix B we consider additional experiments where the agent's 'ss prior over hazard more realistically does not exactly match the environment true hazard rate and demonstrate the benefit of appropriate priors.

---

Under review as a conference paper at ICLR 2020
5.3
ATARI 2600 EXPERIMENTS
With our approach validated in Pathworld, we now move to the high-dimensional environment of Atari 2600, specifically, ALE. We use the Rainbow variant from Dopamine (Castro al. 2018 which implements three of the six considered improvements from the original paper: distributional RL. predicting n-step returns and prioritized replay buffers. The agent (Figure 4 maintains shared representation h((s) of state, but computes Q-value logits for each of the N % via Q4 _ (8, a) Wch(s) where W; and b, are the learnable parameters of the affine transformation for that head. ReL.U-nonliiearity is used within the body of the network (Nair & Hinton 2010).
Lita d)
Our model predicts muliiple hortzons
FC 512
convolational
saiide
represcatation.
com 4 ande
stide
Modd Gosla
Figure Multi-horize model predicts Q-values for n, separate discount functions thereby modeling different effective horizons. Each Q-value is a lightweight computation, an affine transformation off a shared representation. By modeling over multiple time-horizons, we now have the option to construct policies that act according to particular value weighted combination.
Hyperparameter details are provided in Appendix and when applicable, they default to the standard Dopamine values. We find strong performance improvements of the hyperbolic agent built on Rainbow
blue bars) on random subset of Atari 2600 games in Figure
6 MULTI-HORIZON AUXILIARY TASK RESULTS
To dissect the Hyper-Rainbow improvements, recognize that two properties from the base Rainbow agent have changed:
Behavior policy, μ.1 The agent acts according to hyperbolic Q-values computed by our approximation described in Section 4
2. Learn over multiple horizons. The agent simultaneously learns Q-values over many rather than Q-value for single ?
this subset of 19 games, Hyper-Rainbow improves upon games and in some cases, by large margins. But we seek here more complete understanding of the underlying driver of this improvement ALE through an ablation study.
The second modification can be regarded as introducing auxiliary task (Jaderberg al. 2016). Therefore, to attribute the performance of each properly we construct a Rainbow agent augmented with the multi-horizon auxiliary task (referred to still act according to the original policy. That is, Maiti-Rainnow acts to maximize expected rewards discounted by a fixed Jaction but now learns over multiple horizons shown in Figure 4
and shown in orange) but have it
7

---

Under review as a conference paper at ICLR 2020
124
Hyper-Rainbow Multi-Rainboww
227
Percent
Qasta"
Zanaen
Alco
Wim Nowooooooo
Quart
Ensur
Borkoeee
Porg
Ganeol
Attuu
Morbauma
Ga Game Name
Figure 5: We compare the Hyper-Rainbow (in blue) agent versus the Multi-Rainbow (orange) agent on a random subset of 19 games from ALE (3 seeds each). For each game, the percentage performance improvement for each algorithm against Rainbow is recorded. There is no significant difference whether the agent acts according to hyperbolically-discoome discounted (Multi-Rainoowy Q-values suggesting the performance improvement in ALE emerges from the multi-horizon auxiliary task.
(Hyper-Rainbow) or exponentially-
We find that the effectiveness of this as a stand-alone auxiliary task. This is not entirely unexpected given the rather special-case of hazard exhibited in ALE through sticky-actions (Machado et al., 2018).
agent performs nearly as well on these games, suggesting the
We examine further and investigate the performance of this auxiliary task across the full Arcade I Learning Environment (Bellemare et al. 2017) using the recommended evaluation by (Machado ct al. 2018). Doing so we find strong empirical benefits of the multi hottihhoozzn auxiliary task over the satte-of-theaar Rainbow agent as shown in Figure
Auxiliary Task Improvement for Rainhow Agent
10"
Same Name
Figure 6: Performance improvement over Rainbow using the morti-horrizon auxiliary task in Atari Learning Environment (3 seeds each).
6.1 ANALYSIS AND ABLATION STUDIES
To understand the interplay of the multi-horizon auxiliary task with other improvements in deep RL, we test random subset of 10 Atari 2600 games against improvements in Rainbow (Hessel ct al. 2018). On this set of games we measure a consistent improvement with multi-horizon C51 in 9 out of the 10 games over the base C51 agent (Bellemare et al. 2017) in Figure 7
Figure 7 indicates that the current implementation of Multi-Rain Ralti-RRanbow does not generally build successfully on the prioritized replay buffer. On the subset of ten games considered, we find that four out of ten games (Pong, Venture, Gravitar and Zaxxon) are negatively impacted despite (Hessel et al. 2018) finding it to be of considerable benefit and specifically beneficial in three out of these
8

---

Under review as a conference paper at ICLR 2020
10t
Mult-C51 improvement over C51
107
MuRtiC51 + nistop over Multi-C51
10
300
Detent
101
Uphbom
Sceemal
Tang
(a) Multi-CS1
Multi-C51 + n-step
Mult-C51 prioritzed over Multi-C51
Multt-Rainbow over Multi-C5]
10"
101
100 100
20th
xo1
dokcown
Seatuee
Peg
7h00n
Come Nama
Canshhune
(c) Multi-C51 priority
(d) Multi-Rainbow (=Multi-C51 n-
step priority)
Figure 7: Measuring the Rainbow improvements top of the Multi-C51 baseline subset of 10 games in the Arcade Learning Environment (3 seeds each). On this subset, we find that the multi- horizon auxiliary task interfaces well with n-step methods (top right) but poorly with prioritized replay buffer (bottom left).
four games (Venture was not considered). The current prioritization scheme simply averaged the temporal-diffe -enporal-difference errors over all Q-values establish priority. Alternative prioritization schemes are resulted in comparable performance indicating this is an open issue (Appendix J].
RELATED WORK
Hyperbolic discounting in economics. Hyperbolic discounting is well-studied in the field of eco- nomics (Sozou 1998 Dasgupta & Maskin 2005). Dasgupta and Maskin (2005) proposes softer interpretation than Sozou 1998) (which produces a per-time-step of death via the hazard rate) and demonstrates that uncertainty over the riming of rewards also give rise to hyperbolic discount- ing and preference reversals, a hallmark of hyperbolic discounting. Hyperbolic discounting was initially presumed not lend itself to TD>based solutions (Daw & Touretzky 2000) but the field has evolved on this point. Maia (2009) proposes solution directions that find models that discount perssihpeerbolcallly even though each learns with exponential discounting (Loewenstein (996) but reaffirms the difficulty. Finally, Alexander and Brown (2010) proposes hyperbolically discounted temporal difference (HDTD) learning by making connections hazard.
Behavior RL and hyperbolic discounting in neuroscience. TD-learning has long been used for modeling behavioral reinforcement learning (Montague et al., 1996;; Schultz et al. 1997] /Stttnn & Barto 1998). TD-learning computes the error as the difference between the expected value and actual value Sutton & Barto 1998, Daw 2003) where the error signal emerges from unexpected rewards.
However, these computations traditionally rely exponential discounting as part of the estimate of the value which disagrees with empirical evidence humans and animals (Strotz 1955 Mazur
1985 1997 Ainslie 1975 1992). Hyperbolic discounting has been proposed as alternative to
exponential discounting though it has been debated as an accurate model (Kacelnik) 1997] Frederick et al. 2002). Naive modifications TD-learning discount hyperbolically present issues since the simple forms are inconsistent (Daw Touretzky 20000) Redish & Kurth-Nelson (010)) RL models
have been proposed to explain behavioral effects of humans and animals (Fu & Anderson
2006

---

Under review as a conference paper at ICLR 2020
Rangel et al. 2008) but Saurth-Nelson & Redish (2009) demonstrated that distributed exponential discount factors can directly model hyperbolic discounting. This work proposes the μAgent, an agent that models the value function with specific discount factor y. When the distributed set of pAgent's votes on the action, this was shown to approximate hyperbolic discounting well in the adjusting-dela assay experiments (Mazur 1987). Using the hazard formulation established in [Sozou (1998), demonstrate how to extend this to other non-hyperbolic discount functions and demonstrate the efficacy of using a deep neural network model the different Q-values from shared representation.
Towards more flexible discounting in reinforcement learning. RL researchers have recently
adopted more flexible versions beyond fixed discount factor (Feinberg & Shwartz 1994. Sutton 1995 Sutton et al. White
White (017). Optimal policies are studied in Feinberg & Shwartz (1944)
where two value functions with different discount factors are used. Introducing the discount factor as an argument to be queried for a set of timescales is considered in both Horde (Sutton et al. 2011)
and y-nets (Sherstan et al,, 2018). Reinke et al. (2017) proposes the Average Reward Independent Gamma Ensemble framework which imitates the average return estimator. Lattimore and Hutter (20TT) generalizes the original discounting model through discount functions that vary with the age of preferences in hyperbolic discounting. The need to increase training stability via effective horizon was addressed in |François- Lavee,, Fonteneau, and Ernst (2015) who proposed dynamic strategies for the discount factor y. Meta-learning approaches to deal with the discount factor have been proposed in Xu, van Hasselt, and Silver (2018). Finally, Pitiis| (2019) characterizes rational decision making in sequential processes, formalizing a process that admits a state-action dependent discount rates. Operating over multiple time scales has a long history in RL. Sutton (1955) generalizes the work of Singh (1992) and Dayan and Hinton (1993) to formalize a multi-time scale TD learning model theory. Previous work has been explored on solving MDPs with multiple reward functions and multiple discount factors though these relied on separate transition models (Feinberg & Shwartz 1999; Dolgov & Durfee, (005). Edwards, Littman, and Isbell (2015) considers decomposing a reward function into separate components each with its own discount factor. In our work, continue to model the same rewards, but now model the value over different horizons. Recent work in difficult exploration games demonstrates the efficacy of two different discount factors (Burda al.] 2018) one for intrinsic rewards and one for extrinsic rewards. Finally, and concurrent with this work, Romoff et al. (2019) proposes the TD(A)- -algoorgt which breaks a value function into a series of value functions with smaller discount factors.
the agent, expressing
Auxiliary tasks in reinforcement learning. Finally, auxiliary tasks have been successfully employed and found be of considerable benefit in RL. /Suddarth and Kergosien (1990) used auxiliary tasks to facilitate representation learning. Building upon this, work in RL has consistently demonstrated benefits of auxiliary tasks augment the low-information coming from the environment through extrinsic rewards (Lample & Chaplot 2017: Mirowski et al. 2016 Jaderberg et al. 2016; Veeriah
ct al. 2018 Sutton et al. (011)
DISCUSSION AND FUTURE WORK
This work builds on a body of work that questions one of the basic premises of RL: one should maximize the exponentially discounted returns via single discount factor. By learning over multiple horizons simultaneously, we have broadened the scope of our learning algorithms. Through this we have shown that we can enable acting according to new discounting schemes and that learning multiple horizons a powerful stand-alone auxiliary task. Our method well-approximates hyperbolic discounting and performs better in hazardous MDP distributions. This may be viewed as part of an algorithmic toolkit to model alternative discount functions.
However, this work still does not fully capture more general aspects of risk since the hazard rate may be a function of time. Further, hazard may not be an intrinsic property of the environment but joint property of both the policy and the environment. If an agent purses a policy leading to dangerous state distributions then it will naturally be subject to higher hazards and vice-versa this creates a complicated circular dependency. We would therefore expect an interplay between time- time-preference and policy. This is not simple to deal with but recent work proposing state-action dependent discounting (Pitis. 2019) may provide formalism for more general preefreeee schemes.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
George Ainslie. Specious reward: a behavioral theory of impulsiveness and impulse control. Psycho- logical bulletin, 82(4))463, 1975.
George Ainslie. Picoeconomics:
The strategic interaction of successive motivational states within the person. Cambridge University Press, 1992.
William H Alexander and Joshua W Brown. Hyperbolically discounted temporal difference learning. Neural computation, 22(6):1511-1227, 2010.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ-
ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. arXiv preprint arXiv: 2017.
Richard Bellman. markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679-684, 1957.
Richard Bellman. On a routing problem. Quarterly of applied mathematics, 16(1):87-90, 1958.
Dimitri Bertsekas. Neuro-dynamic programming: an overview. 1995.
Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming, volume Athena Scientific Belmont, MA, 1996.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810.128%. 2018.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A research framework for deep reinforcement learning. CoRR, abs/1812.061 10, 2018
URL http: /tat::///xxivi org/ abs/ 812 06110
Partha Dasgupta and Eric Maskin. Uncertainty and hyperbolic discounting. American Economic Review, 9594):120-1129, 2005.
Nathaniel D Daw. Reinforcement learning models of the dopamine system and their behavioral implications. PhD thesis, Carnegie Mellon University, 2003.
Nathaniel D Daw and David S Touretzky. Behavioral considerations suggest an average reward td model of the dopamine system. Neurocomputing, 32:679-684, 2000.
Peter Dayan and Geoffrey E Hinton. Feudal reinforeemnt learninnng In Advances in neural information processing systems, pp. 271-278, 1993.
Dmitri Dolgov and Edmund Durfee. Stationary deterministic policies for constrained mdps with multiple rewards, costs, and discount factors. Ann Arbor, 1001:48109, 2005.
Ashley Edwards, Michael L Littman, and Charles Isbell. Expressing tasks robustly via multiple discount factors. 2015.
Eugene A Feinberg and Adam Shwartz. Markov decision models with weighted discounted criteria. Mathematics of Operations Research, 19(1):152-168, 1994.
Eugene A Feinberg and Adam Shwartz. Constrained dynamic programming with two discount factors: Applications and an algorithm. IEEE Transactions on Automatic Control, 44(3)6628-631, 1999.
Vincent François-Lavee Raphael Fonteneau, and Damien Ernst. How to discount deep reinforcement learning: Towards new dynamic strategies. arXiv preprint arXiv:1512.02011, 2015.
Shane Frederick, George Loewenstein, and Ted O O'donoghue. Time discounting and time preference: A critical review. Journal of economic literature, 40(2):351-401, 2002.

---

Under review as a conference paper at ICLR 2020
Wai-Tat Fu and John R Anderson. From recurrent choice skill learning A reinforcerenntleamingg model. Journal of experimental psychology: General, 135, 2006.
Leonard Green and Joel Myerson. A discounting framework for choice with delayed and probabilistic rewards. Psychological bulletin, 130(5):769, 2004.
Leonard Green, Ewin B Fisher, Steven Perlow, and Lisa Sherman. Preference reversal and self control: Choice as a function of reward amount and delay. Behaviour Analysis Letters, 1981.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 161.:06611.0397,, 2016.
Alex Kacelnik. Normative and descriptive models of decision making: time discounting and risk sensitivity. Characterizing human psychological adaptations, 208:51-66, 1997.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)999-34, 1998.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning polynomial time. Machine learning, 49(2-3):209-225 232, 2002.
Zeb Kurth-Nelson and A David Redish. Temporal-differrncee reinforcement learning with distributed representations. PLoS One, 4(10)=e7362, 2009.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. 2017.
Tor Lattimore and Mareus Hutter. Time consistent discounting. In International Conference on Algorithmic Learning Theory, pp. 383-397. Springer, 2011.
George Loewenstein. Out of control: Visceral influences behavior. Organizational behavior and human decision processes, 65(3):272-292, 1996.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
Tiago Maia. Reinforcement learning, conditioning, and the brain: Successes and challenges.
Cognitive, Affective, & Behavioral Neuroscience, 9():3433-36 2009.
James E Mazur. Probability and delay of reinforcement as factors in discrete-trial choice. Journal of the Experimental Analysis of Behavior, 43(3):341-351, 1985.
James E Mazur. An adjusting procedure for studying delayed reinforcement. 1987.
James E Mazur. Choice, delay, probability, and conditioned reinforcement. Animal Learning & Behavior, 25(2):131-147, 1997.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning navigate in complex environments. arXiv preprint arXiv:1611.03673 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51auure,1887555 518(7540):529, 2015.
Read Montague, Peter Dayan, and Terrence Sejnowski. A framework for mesencephalic dopamine systems based predictive hebbian learning. Journal of neuroscience, 16(5):1936-194) 1996.
12

---

Under review as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
OpenAl. Openai five.
2018.
Silviu Pitis. Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. AAAI Press, 2019.
Danil Prokhorov and Donald Wunsch. Adaptive critic designs. IEEE transactions on Neural Networks, 8()):977-1007, 1997.
Antonio Rangel, Colin Camerer, and P Read Montague. A framework for studying the neurobiology of value-based decision making. Nature reviews neuroscience, 9(7):545, 2008.
David Redish and Zeb Kurth-Nelson. Neural models of temporal discounting. 2010.
Chris Reinke, Eiji Uchibe, and Kenji Doya. Average reward optimization with multiple discounting reinforcement learners. In International Conference Neural Information Processing, pp. 789- 800. Springer, 2017.
Joshua Romoff, Peter Henderson, Ahmed Touati, Yann Ollivier, Emma Brunskill, and Joelle Pincau. Separating value functions across time-scales. arXiv preprint 1rXiv:1902.013 2019.
Paul A Samuelson. A note on measurement of utility. The review of economic studies, 4(2):155-1611
1937.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, (75(5506):1593-1111 1997.
Craig Sherstan, James MacGlashan, and Patrick M. Pilarski. Generalizing value estimation over timescal. In FAIM Workshop on Prediction and Generative Modeling in Reinforcement Learning, 2018.
Satinder P Singh. Scaling reinforcement learning algorithms by learning variable temporal resolution models. In Machine Learning Proceedings 1992, pp. 406-415. Elsevier, 1992.
Peter Sozou. On hyperbolic discounting and uncertain hazard rates. Proceedings of the Royal Society of London B: Biological Sciences, 265(1409):2015-22) 2020, 1998.
Robert Henry Strotz. Myopia and inconsistency in dynamic utility maximization. The Review of Economic Studies, 23(3):165-180, 1955.
Steven Suddarth and YL Kergosien. Rule-injection hints as a means of improving network performance and learning time. In Neural Networks, pp. 120-129. Springer, 1990.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 44, 1988.
Richard S Sutton. Td models: Modeling the world at a mixture of time scales. In Machine Learning Proceedings 1995, pp. 531-539. Elsevier, 1995.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 1998.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Vystems-Volume' Multiagent Systems- Volume 2, pp. 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
Vivck Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning, arXiv preprint arXiv:1806.00600 2018.
Christopher JCH Watkins and Peter Dayan. Q-leariing. Machine learning, 8(3-4):279-29 1992.
13

---

Under review as a conference paper at ICLR 2020
Martha White. Unifying task specification in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3742-3750. JMLR. org. 2017.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. arXiv preprint arXiv: 180.00000, 2018.
14

---

Under review as a conference paper at ICLR 2020
A
SozoU (1998): BELIEF OF RISK IMPLIES A DISCOUNT FUNCTION
Sozou 1998) formalizes time preferences in which future rewards are discounted based on the probability that the agent will not survive to collect them due to encountered risk hazard. Definition A.1. Survival s(t) is the probability of the agent surviving until time t.
s (t) = P(agent is alive|at at time t)
(13)
future reward ry is less valuable presently if the agent unlikely survive collect the agent is risk-neutral, the present value of future reward received at time-/ should be discounted by the probability that the agent will survive until time / to collect it, {
v (rr)) = s()rr
(14)
Consequently, if the agent is certain to survive, s(t) then the reward not discounted per Equation 14 From this it is then convenient to define the hazard rate. Definition A.2. Hazard rate h(tt is the negative rate of change of the log-survivee at time
k(t) = dt
Ins(t)
(15)
or equivalently expressed as h(t) de(t) dt a()) Therefore the environment is considered hazardous
at time if the log survival decreasing sharply.
Sozou 1998) demonstrates that the prior belief of the risk in the environment implies specific discounting function. When the risk occurs at a known constant rate than the agent should discount future rewards exponentially. However, when the agent holds uncertainty over the hazard rate then hyperbolic and alternative discounting rates arise.
A.1 KNOWN HAZARD IMPLIES EXPONENTIAL DISCOUNT
We recover the familiar exponential discount function in RL based a prior assumption that the
environment has known constant hazard. Consider a known hazard rate of h(t) = X 2 0. Definition
A.2 sets a first order differential equation À Ins(t)) de(() 1
The solution for the survival
*(8)
rate is s(t) e which can be related to the RL discount factor y
(1) e ~
(16)
This interprets 7 as the per-time-step probability of the episode continuing. This also allows connect the hazard rate À E [0, oo] to the discount factor E [0,1).
(17)
oc, then the corresponding discount factor becomes increasingly myopic 7 - O. Conversely, as the environment hazard vanishes, À - O, the corresponding agent becomes increasingly far-sighted 7 + In RL we commonly choose single ? which is consistent with the prior belief that there exists known constant hazard rate X -In(7). We now relax the assumption that the agent holds this strong prior that it exactly knows the true hazard rate. From Bayesian perspective, a looser prior allows for some uncertainty in the underlying hazard rate of the environment which we will see the following section.
As the hazard increases
A.2 UNCERTAIN HAZARD IMPLIES NON-EXPONENTIAL DISCOUNT
We may not always be so confident of the true risk in the environment and instead reflect this underlying uncertainty in the hazard rate through a hazard prior p(A). Our survival rate then computed by weighting specific exponential survival rates defined by a given À over prior p())
st))
p(A)e dd
(18)
A=0
Note the difference in RL. where future rewards are discounted by time-delay the value v(r.) y're-
15

---

Under review as a conference paper at ICLR 2020
Sozou (1998) shows that under an exponential prior of hazard p()) eex((->/k) the expected
survival rate for the agent is hyperbolic
(())
(t())
(19)
1 kt
We denote the hyperbolic discount by (t) to make the connection to in reinforcement learning explicit. Surther,/Szzon (1998) shows that different priors over hazard correspond to different discount functions. We reproduce two figures in Figure 8 showing the correspondence between different hazard rate priors and the resultant discount functions. The common approach in RL is to maintain delta-hazard (black line) which leads to exponential discounting of future rewards. Different priors lead to non-exponential discount functions.
Hazard Prior z(A)
Discount Function
1.4
prior
prior discount function
exponential prior
exponential prior >1/
1.2
uniform prior - --
1.0
gamma prior
discount
gamma prior 1/11
0.6
delta prior --- --
0.6
0.4
0.4
0.2
Figure We reproduce two figures from Sozou (1998). There correspondence between hazard rate priors and the resulting discount function. In RI., we typically discount future rewards exponentially which is consistent with a Dirac delta prior (black line) on the hazard rate indicating no uncertainty of hazard rate. However, this is special case and priors with uncertainty over the hazard rate imply new discount functions. All priors have the same mean hazard rate Elp(A)] =
B ADDITIONAL PATHWORLD EXPERIMENTS
In Figure we consider the case that the agent still holds an exponential prior but has the wrong coefficient k and in Figure [ [0 we consider the case where the agent still holds an exponential prior but the true hazard is actually drawn from uniform distribution with the same mean.
Through these two validating experiments, we demonstrate the robustness of estimating hyperbolic discounted Q-values in the case when the environment presents dynamic levels of risk and the agent faces non-trivial decisions. Hyperbolic discounting is preferable to exponential discounting even when the agent's 's prior does not precisely match the true environment hazard rate distribution, by coefficient (Figure 9 or by functional form (Figure 10
16

---

Under review as a conference paper at ICLR 2020
Value for Diff. Hyperbolic Coefficients (8)
Ville edlivak theoreral-aa0
Discount function
MSE
Expected value Elria]]
K--6 Jeet
k=0.05
0.002
k=0.1
0.493
2.0
k=0.025
0.814
1.5
k=0.2
1.281
3.0
0.5
0.0
Path
Figure Case when the hazard coefficient does not match that environment hazard. Here the true haz- = 0.05, but we compute values
Table 2:
The average mean squared error (MSE) over each of paths in Figure As the prior is further away from the true value of k = 0.05, the error increases. However, notice that the errors for large factor-of-2 changes sult in generally lower errors than the agent had considered only single exponential discount factor in Table
ard coefficient for hyperbolic agents with mismatched priors in range = [0.025, 0.05, 0.1, 0.2]. Predictably, the mismatched priors result in higher prediction error of value but performs more reliably than exponential discounting, re- sulting in cumulative lower error. Numerical results in Table
re-
Value Under uniform Hazard
4.5
ediintt
4.0
Discount function
MSE
Expected value [(r(o]]
3.5
garmme-0.577 0.575
panme-6..5 55
hyperbolic value
0.235
? 0.975
0.266
7 0.95
0.470
7 0.99
4.029
2.0
0.5
0.0
10
12
Path index
Figure 10: If the hazard rate now drawn according to a uniform distribution (with the same mean as before) the original hyperbolic discount matches the functional form better than exponential discounting. Numerical results in Table
Table 3:
The average squared error (MSE) over each of the paths in Figure TT] when the underlying hazard is drawn ac- cording uniform distribution. We find that hyperbolic discount- ing results is more robust to haz- ards drawn from uniform distri- bution than exponential discount- ing.
17

---

Under review as a conference paper at ICLR 2020
C ALTERNATIVE APPROACH TO HYPERBOLIC Q- VALUES
C.1 COMPUTING HYPERBOLIC Q- VALUES
Let's start with the case where we would like estimate the value function where rewards are discounted hyperbolically instead of the common exponential scheme. We refer to the hyperbolic Q-values as Q below in Equation 21
Q (s, 0) ET
(((1) + ( 02) +... 8,
(20)
E Σ I(((t)((s),a))) 8,a
(21)
We may relate the hyperbolic Q. -value to the values learned through standard Q-learning. To do so, notice that the hyperbolic discount r can be expressed as the integral of certain function f(7
for y = [0, 1) in Equation 22
(t)
(22)
y=
kt
The integral over this specific function f(v, 1) = Tht yields the desired hyperbolic discount factor (t) by considering an infinite set of exponential discount factors 7 over domain y E [0, 1).
Recognize that the integrand 7kt is the standard exponential discount factor which suggests connection standard Q-learning (Watkins & Dayan 1992). This suggests that if we could consider an infinite set of then we can combine them to yield hyperbolic discounts for the corresponding time-step t. We build on this idea of modeling many y throughout this work.
We employ Equation 22 and return to the task of computing hyperbolic Q-values Q
(= a) =E
(()))) 8,
(23)
R Rlon
= Ex
(24)
Σ R(ss aels Ellst dy
Ex
(25)
0=
(26)
where (t) has been replaced on the first line by (tt and the exchange is valid if Ee k < oc. This shows that we can compute the Qa-velue according to hyperbolic discount factor by considering an infinite set of Q7 -vauees computed through standard Q-learning. Examining further, each 7 E j0, 1) results in TD-errors learned for new >xx For values of k < 1, which extends the horizon of the hyperbolic discounting, this would result in larger 7.
'Hyperbolic Q-values generally be infinite for bounded rewards. We consider non-infinite episodic MDPs only.
18

---

Under review as a conference paper at ICLR 2020
D VISUAL SUMMARY OF APPROACH
We summarize our approach for estimating non-exponential discounted Q-values here.
Hyperbolically-diccoutte Q-values expressed weighting over exponentiall- -disoouted Q-values using same weights w(y):
1. A hyperbolic discount function
r
1+ kr
O)(s, f. w(y) aa(s, dy
can be expressed as weighting over exponential discount functions y!
r = f('
w(y) dy
Where @ (s. is simultaneously learned for each
exponential discount /: G
with weights wky) = 11-1
(see Table l).
(s, a)
C (r,))
(r,
3. The integral in box can be approximated with Riemann sum over the discrete intervals:
= 170.71 ... nv]
at 1011 1d w(x) ()
Learning simultaneous exponential Q-values effective auxiliary tasks.
Figure 11: Summary of our approach to approximating hyperbolic (and other non-exponential) Q-values via weighted sum of exponentialy--dicounned Q-vaulues.
19

---

Under review as a conference paper at ICLR 2020
E EQUIVALENCE OF HYPERBOLIC DISCOUNTING AND EXPONENTIAL HAZARD
Following Section A we also show a similar equivalence between hyperbolic discounting and the specific hazard distribution (A( =exp(-)/k), where again, À [0, oo)
(k = Br
(((t) ae]ls0 s, 0, =
x
Ph (2)e (2)e--Me Mad) Rise, ax)1 - 0,
= Er
Pk (A)
Σ
M a[ = e
dd
A=0
= (Ampx()BBT.RR
(((stt0))(88 8, 8,00 a
= Ex Dn
R(s,, a. az){so = 0,
= Q (8)
Where the first step uses Equation 19 This equivalence implies that discount factors can be used to learn policies that are robust to hazards.
F ALTERNATIVE DISCOUNT FUNCTIONS
We expand upon three special cases to see how functions f(r.t) = w(y) may be related to different discount functions d(t).
We summarize in Table 4 how particular hazard prior p(A) can be computed via integrating over specific weightings w(7) and the corresponding discount function.
H p(A)
d(1)
w(7)
Dirac Delta Prior
8(1- - k)
c-k((= (7k)")
28(-1n9 In ( k)
7
Exponential Prior
# -
I I+ki
1 1/K-11
if E [0,
1 00
if E (e-
Uniform Prior
Je 0,
1 let
e
otherwise
otherwise
Table 4: Different hazard priors H = p(2) can be alternatively expressed through weighting ex- ponential discount functions 7 by w(7). This table matches different hazard distributions their associated discounting function and the weighting function per Lemma 3.1 The typical case in RL Dirac Delta Prior over hazard rate 8(A k). We only show this in detail for completeness; one would not follow such convoluted path to arrive back at an exponential discount but this approach holds for richer priors. The derivations can be found in the Appendix
Three cases:
1. Delta hazard prior: p(A) = 5(A k)
2. Exponential hazard prior: p()) - -A/k
3. Uniform hazard prior: p(A) for À [0
20

---

Under review as a conference paper at ICLR 2020
For the three cases begin with the Laplace transform the prior p()) - p(A)e dd and then chnage the variables according the relation between 7 e Equation 17
F.1 DELTA HAZARD PRIOR
A delta prior p\) = 8(A k)) on the hazard rate is consistent with exponential discounting.
S p(A)e dd
/ 5(2 --ee At dd
A=0
where 8(A -k) is Dirac delta function defined over variable À with value k. The change of variable e (equivalently À = - In 7) yields differentials dA = -1 and the limits X = 0 + = 1
and A - 7 Additionally, the hazard rate value À le is equivalent to the ?
d(t)
p(A)E Md -ddx
x=0
1 6(- lnn - k))²" (-20h)
1=1
5(- 8(-nn 16))t-1d7
0=
kt
7k
where we define a "/k = e to make the connection to standard RL discounting explicit. Additionally and reiterating, the of single discount factor, in this case YYk, is equivalent to the prior that single hazard exists in the environment.
F.2 EXPONENTIAL HAZARD PRIOR
Again, the change of variable y - yields differentials dA -- dy and the limits À + and À -
0 dA
o(-1n (-1er)
p(A)e
p(
A=0
9=1
(1-ln))" 1-
0=
where p(-) the prior. With the exponential prior p(A) = e -\/k) and by substituting -Iny verify Equation
exp(Inn/kk)' dr / exp(In>/ 1/1gg" dr
7. 1/k+t-1
1/k+1
+
1
21

---

Under review as a conference paper at ICLR 2020
F.3 UNIFORM HAZARD PRIOR
Finally if we hold uniform prior over hazard, for X E [0, k] then Sozou (998) shows the Laplace transform yields
d(t) =
p(A)e p(A)e-dras ->
0
yp
k:
k
_ At
kt
A=D
(1
-kt)
kt
Use the same change of variables to relate this to y. The bounds of the integral become À = 0 -
y = and + -
d(t)
g-- dr
1
kt 9=0-
kl (1
1
kt)
which recovers the discounting scheme.
G DETERMINING THE Y INTERVAL
We provide further detail for which y we choose to model and motivation why. We choose a Ymux which is the largest y to learn through Bellman updates. If we using the hyperbolic coefficient in Equation 19 and we are approximating the integral with nyy our "/max would be
Ymax (1 -1--0 b*
(27)
However, allowing Ymax - get arbitrarily close may result in learning instabilities Bertsekas 1995). Therefore we compute an exponentiation base of exp(In(1 1/k //..) which bounds our Tmax a known stable value. This induces an approximation error which is described more in Appendix H
H APPROXIMATION ERRORS
Instead evaluating the upper bound of Equation at we evaluate at Yyux which yields T /(1+/(t)))) Our approximation induces an error in the approximation of the hyperbolic discount.
This approximation error in the Riemann sum increases as the Ymux decreases as evidenced by Figure 12 When the maximum value of Tmax 1 then the approximation becomes more accurate as supported in Table up to small random errors.
I ESTIMATING HYPERBOLIC COEFFICIENTS
As discussed, we can estimate the hyperbolic discount in two different ways. We illustrate the resulting estimates here and resulting approximations. We use lower-bound Riemann sums in both cases for simplicity but more sophisticated integral estimates exist.
As noted earlier, considered two different integrals for computed the hyperbolic coefficients. Under the form derived by the Laplace transform, the integrals are sharply peaked as - 1. The difference in integrals visually apparent comparing in Figure 13
22

---

Under review as a conference paper at ICLR 2020
Path Value Estimate Across Max-Gammas
2.5
Expected value E[ria]]
2.0
Discount function
MSE
max-y=0.999
0.002
1.5
max m
0.003
1.0
max-r=0.99
0.233
max->=0.95
1.638
0.5
max-y=0.9
2.281
0.0
0
6
8
12
14
Path index
Figure 12: By instead evaluating our integral up to Ymma rather than to I, we induce approximation error which increases with Numerical results Table
Table
The average mean squared error (MSE) over each of the paths Figure
D=0,10.
Discount
9.
92 0.4
1.7
Discount for k- 0,10, t-5
Discount for 1..0.00.14
0.-
03
02
7
1.,
3. 0.8
0.3
1.2
Discount for bwO 10, 2=
Discount for low 10, t-
0.5-
38
03
02
64
0.6
7
(a) Our approach.
(b) Alternative approach.
Figure 13: Comparison of hyperbolic coefficient integral estimation between the two approaches. (a) We approximate the integral of the function n via a lower estimate rectangles at specific y-values. The sum of these rectangles approximates the hyperbolic discounting scheme 1/(1 kt) for time
(b) Alternative form for approximating hyperbolic coefficients which is sharply peaked 7 1 which led to larger errors in estimation under our initial techniques.
23

---

Under review as a conference paper at ICLR 2020
J PERFORMANCE OF DIFFERENT REPLAY BUFFER PRIORITIZATION SCHEME
As found through our ablation study in Figure 7 the Multi-Rainbow auxiliary task interacted poorly with the prioritized replay buffer when the TD-errors were averaged evenly across all heads. As an alternative scheme, we considered prioritizing according to the largest 7, which is also the 7 defining the Q-values by which the agent acts.
The (preliminary) results of this new prioritization scheme is in Figure 14
Multi-Rainbow Improvement over Rainbow (prioritize-largesst)
loekyy
Spacoinoddri
Phocnix
Pitall
Brewkzut
Andweti
Yo rsRevvnnge nge
Gopher
Crevilar
Name thaaGame
StanSunnce
Tennis
padde
Kungfulamsie ber
MrsP
TimeP lot
BontieZone
Solanis
Game Name
Benzerk
Attenels
Tutariéham
srall
Haro
BeamRider
Venture
Ensturo
BevatorAction
EmeryClimber
UphDtwwn
Bochn
Kangarbo
Robobank
Doutie Dunk
JoumeyEaccae
Corsival
Domonkttackk
Anseril
Trosibite
AiRoald
jamesbond
Mardaeumellaeengee
10t -100100 1ol
Percent Improvement (log)
Figure 14: The (preliminary) performance improvement over Rainbow using the multi-horizon auxiliary task in Atari Learning Environment when we instead prioritize according to the TD-errors computed from the largest 7 (3 seeds each).
To this point, there is evidence that prioritizing according to the TD-errors generated by the largest gamma is better strategy than averaging.
TThese runs have been computed over approximately 100 out of 200 iterations and will be updated for the final version.
24

---

Under review as a conference paper at ICLR 2020
K HYPERPARAMETERS
For all our experiments in DQN Mnih et al. (0015), C51 Bellemare et al. (2017) 7) and Rainbow
Hessel et al. (2018), benchmark against the baselines by Castro al. 2018) and the
default hyperparameters for each of the respective algorithmss. That is, our Multi-agent uses the same optimization, learning rates, and hyperparameters it's base class.
Hyperparameter
Value
Runner wtterrsstcckyacytticieoe
St cky act oons prob 0.25 200
Runner. num, iterations
Runner.training steps
250000
Rumner.evaluationn.ee steps Runner.max_ Runneer.nakksstep__ee__p_.peee
125000
27000
1000000
32
Raiinboxkgent..nmm_e.
51
Rainbowkgent.
Radiboowggent.uuattt..oooo..
3
htw.bowwggt..mb.p.pppeeeeeeeeeee Ra
20000
4
8000
0.01
RainbovAgget.eepssloo.eeaaa bainbookgennteessiion...e period
0.001
250000
Rainbowkgent..eeplyy...tttt Reinborkgeet..f_ddvvice Neinmoow/gent..guiiiizrr
prioritized'
1 /gpp:
etf. E.E.traiin.Admmpeiiii ()
tf. tf.train. : Leapbeiiniie.r..reeennee_een cr.
0.0000625
0.00115
ByperRaaluboolggett.mubeee ofgaamaaagaaa
1C
HyperRainbowkgent.co gamnaaaaaaaaaaaaaaaaaaaaaaa max
0.99
ByperRaiinoowkgent.oo htt.ttt.pppppp_eppeeeeeeeepee HyperRai www.aaaatttttttttnnngggee..e..
0.01
ggeat_gamma
Table 6: Configurations for the Multi-C51 and Multi-Rainbow used with Dopamine Castro et al.
(2018).
L AUXILIARY TASK RESULTS
Final results of the multi-horizon auxiliary task on Rainbow (aulti-Rainbow) in Table 7
25

---

Under review as a conference paper at ICLR 2020
Game Name
DQN
CS1
Rainbow
Multi-Rainbow
AirRaid
8190.3
9191.2
16941.2
12659.5
Alien
2666.0
2611.4
3858.9
3917.2
Amidar
1306.0
1488.2
2805.7
2477.0
Assault
1661.6
2079.0
3815.9
3415.1
Asterix
3772.5
15289.5
19789.2
24385.6
Asteroids
844.7
1241.5
1524.1
1654.5
Atlantis
935784.0
894862.0
890592.0
923276.7
BankHeist
723.5
863.4
1209.0
1132.0
Battležone
20508.5
28323.2
42911.1
38827.1
BeamRider
6326.4
6070.6
7026.7
7610.9
Berzerk
590.3
538.3
864.0
879.1
Bowling
49.8
68.8
62.9
Boxing
83.3
83.5
98.8
99.3
Breakout
146.6
254.1
123.9
162.5
Carnival
4967.9
4917.1
5211.8
5072.2
Centipede
3419.9
8068.9
6878.0
6946.6
ChopperConmand
3084.5
6230.4
13415.1
13942.9
CrazyClimber
113992.2
146072.3
151454.9
160161.0
DemonAttack
7229.2
8485.1
19738.0
14780.9
DeubleDunk
-4.5
2.7
22.6
21.9
ElevatorAction
2434.3
73416.0
81958.0
85633.3
Enduro
895.0
1652.9
2290.1
2337.5
FishingDerby
12.4
16.6
44.5
45,1
Freeway
26.3
33.8
33.8
33.8
Frostbite
1609.6
4522.8
8988.5
7929.7
Gopher
6685.8
8301.1
11749.6
13664.6
Gravitar
339.1
709.8
1293.0
1638.7
Hero
17548.5
34117.8
47545.4
50141.8
IceHockey
-5.0
-3.3
2.6
6.3
Jamesbond
618.3
816.5
1263.8
773.4
JourneyEscape
-2604.2
-1759.1
-818.1
-1002.9
Kangaroo
13118.1
9419.7
13794.0
13930.6
Krull
6558.0
7232.3
6292.5
6645.7
KungFuMaster
26161.2
27089.5
30169.6
31635.2
MontezumaRovenge
2.6
1087.5
501.3
800.3
MsPacman
3664.0
3986.2
4254.2
4707.3
NameThisGame
7808.1
12934.0
9658.9
11045.9
Phoenix
5893.4
6577.3
8979.0
23720.3
Pitfall
-11.8
0.0
Pong
17.4
19.7
20.3
20,6
Pooyan
3800.8
3771.2
6347.7
4670,0
PrivateEye
2051.8
19868.5
21591.4
888.9
Obert
11011.4
11616.6
19733.2
20817.4
Riverraid
12502.4
13780.4
21624.2
21421.2
RoadRunner
40903.3
49039.8
56527.4
55613.0
Robotank
62.5
64.7
67.9
67.2
Seaquest
2512.4
38242.7
11791.5
64985.0
Skiing
-13114,9
11996..7
11792.9
-1600..3
5olaris
2062.7
2788.0
3061.9
3139.9
SpaceInvaders
1976.0
4781.9
4927.9
8802.1
StarCunner
47174.3
35812.4
58630.5
72943.2
Tennis
-0.0
22.2
0.0
0.0
TimePilot
3862.5
8562.7
12486.1
14421.7
Tutankham
141.1
253.1
255.6
264.9
UpNDewn
10977.6
9844.8
42572.5
50862.3
Venture 88.0
1430.7
1612.4
1639.9
VideoPinball
222710.4
594468.5
651413.1
650701.1
Wi wirffofworr
3150.8
3633.8
8992.3
9318.9
YarsRevenge
25372.0
12534.2
47183.8
49929.4
Zaxxon
5199.9
7509.8
15906.2
21921.3
Table 7: Multi- Multi-Rainbow agent returns versus the DQN, C51 and Rainbow agents of Dopamine Castro et al. (201)).
26


Reviews:
[{"review_id": "SJgBEDMhtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studied the discounting factor of RL under the uncertain hazard and non-trivial intertemporal decisions, by proposing a hyperbolic discounting strategy. The authors show the equivalence between the hyperbolic discount function and the temporal difference learning techniques in RL. Experimental results show the performance of the proposed method by comparing it with state-of-the-art. In general,  the text is well written and easy to follow. However,  I am not familiar with the context of hyperbolic computation.  I do not know any related works or what to expect from the results. I could not find anything wrong with this paper, but also do not have any intelligent questions to ask. Therefore, I am not sure about the technical contribution of this paper to the area. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1lYtMykcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that hyperbolic and other non-exponential discounting mechanisms have been more utilized by humans and animals for value preferences than exponential discounting as widely used in RL literature. The authors claim that hyperbolic discounting mechanisms are especially preferred in the setting of maintaining uncertainty over the prior belief of the hazard rate in the environment and propose an efficient approximation of the Q function with hyperbolic and other non-exponential discounting mechanisms as a weighted sum of Q-functions with the standard exponential discounting factor. The paper shows empirical evidence that hyperbolic discounting function can more accurately estimate the value in a vanilla Pathworld environment and also demonstrate that the approximated multi-horizon Q functions can improve performance on ALE, which is largely attributed to learning over multi-horizons as an auxiliary task.\n\nOverall, this paper is an extension of the prior work Kurth-Nelson & Redish (2009). It seems to me that the difference between this paper and Kurth-Nelson & Redish (2009) is that in this paper, the approximated Q-value with hyperbolic discounting function is a weighted sum over each Q-values using exponential discounting factor gamma, while in Kurth-Nelson & Redish (2009), the Q-value is estimated by sampling one Q-value based on the distribution of the gamma. Another difference as claimed in the paper is that the authors also present an approximation of other non-hyperbolic discounting functions. Those extensions seem a bit incremental and are not well supported by the experimental results in the paper. The authors didn’t compare their method to Kurth-Nelson & Redish (2009) in both Pathworld and Atari 2600, which seems insufficient to demonstrate the point that the extension is useful. Moreover, the authors didn’t conduct experiments on non-hyperbolic discounting functions, which makes the claim that approximating non-hyperbolic discounting functions is one of the paper’s contributions unsubstantiated empirically.\n\nIn Section 6, the experiments in Atari 2600 show that Hyper-Rainbow and Multi-Rainbow have similar performance, and the authors claim that such results imply that learning over multi-horizons as an auxiliary task is more useful than choosing different discounting schemes. This seems to contradict the whole point of the paper that the hyperbolic discounting mechanism is biologically more reasonable than exponential discounting functions. The authors then claim that learning Q-values with multiple discounting factors is also one of the paper’s contributions, which makes the paper look like a list of tricks to get RL to work in the multi-task setting rather than a unifying framework of the hyperbolic discounting scheme. I think the authors might need to pick a deep RL domain similar to Pathworld, i.e. an environment with risk increasing as the length of the trajectory increases, but in high-dimensional state space and continuous action space, to better demonstrate the effectiveness of the method.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e2NwJ4jB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates hyperbolic discounting as a more biologically plausible alternative to exponential discounting in reinforcement learning. First, it formulates a notion of hazard in MDPs as constant exponential discounting and shows that hyperbolic discounting is consistent with uncertainty over the hazard rate. The paper then shows how value functions learned with exponential discounting can be used to approximate value functions with other forms of discounting. Specifically, the paper shows in section 4 how exponentially-discounted value functions can be used to approximate hyperbolically discounted value functions. The paper then presents experiments on a small MDP and Atari 2600 games, showing that learning discounted action values with many different discount rates as an auxiliary task improves performance on most Atari games.\n\nOverall, I very slightly tend towards accepting this paper for publication. While the idea of hyperbolic discounting didn't seem to pay off in terms of performance on the Atari 2600 games, the idea still has some merit as being more biologically plausible than exponential discounting, and may perform better on a more suitable environment. In addition, the discovery that learning many action-value functions with different exponential discounting rates as auxiliary tasks improves performance is quite interesting.\n\nNotes:\n- The footnote on the first page seems to just trail off. It would be better to explicitly state what the takeaway is. I'm guessing humans and animals in general would prefer \\$1M now in the first scenario, but \\$1.1M in the second scenario?\n- The idea of 1/(1-gamma) being an \"effective horizon\" seems questionable. For gamma=0.9 the \"effective horizon\" (aka expected number of timesteps before termination) would be 1/(1-.9)=10 timesteps. However, 0.9^(10)=~0.3486 means ~34% of the probability density is still to the right of that number, so calling 1/(1-gamma) an \"effective horizon\" seems to violate what we normally mean by \"horizon\".\n- A single hazard rate for the entire environment seems counterintuitive. Some states in life are definitely more hazardous than others, which suggests state-dependent discounting might be an interesting idea to explore.\n- Many people in the field think we shouldn't even be doing discounting at all (section 10.4 of Reinforcement Learning by Sutton and Barto), only total reward for episodic problems and average reward for continuing problems.\n- Pathworld experiments seem like they don't show anything useful; the hyperbolic discounting matches the true hazard behaviour of the environment (i.e., the hazard rate is drawn from a distribution) so it performs better than exponential discounting. However, if the environment had a fixed hazard rate then exponential discounting would perform better.\n- Using only 3 random seeds does not seem like enough for the experiments. Also there is no measure of standard error or statistical significance on the graphs.\n- Introducing a hyperparameter that increases computation and memory does not really seem like it's solving the problem, and certainly not \"efficiently\" as claimed in the abstract.\n- How was the value of the new hyperparameter set for the experiments? Were several values tried? More importantly, how can a good value for the new hyperparameter be chosen without prior knowledge of the environment?\n- Atari 2600 games don't really seem like a great environment to showcase the benefits of hyperbolic discounting.\n- It would be interesting to see a comparison with existing auxiliary task approaches, but due to space constraints should probably be left for future work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was some support for the ideas presented in this paper, it was on the borderline, and ultimately did not make the cut for publication at ICLR.

Concerns were raised as to the significance of the contribution, beyond that of past work.
Decision: Reject