Reinforcement Learning with Non-Exponential Discounting
Authors: Matthias Schultheis
Constantin A. Rothkopf
Heinz Koeppl
Venue: NeurIPS 2022 Accept (2022)
Keywords: discounting
reinforcement learning
optimal control
continuous time
human decision making
inverse reinforcement learning

Abstract: Commonly in reinforcement learning (RL), rewards are discounted over time using an exponential function to model time preference, thereby bounding the expected long-term reward. In contrast, in economics and psychology, it has been shown that humans often adopt a hyperbolic discounting scheme, which is optimal when a specific task termination time distribution is assumed. In this work, we propose a theory for continuous-time model-based reinforcement learning generalized to arbitrary discount functions. This formulation covers the case in which there is a non-exponential random termination time. We derive a Hamilton–Jacobi–Bellman (HJB) equation characterizing the optimal policy and describe how it can be solved using a collocation method, which uses deep learning for function approximation. Further, we show how the inverse RL problem can be approached, in which one tries to recover properties of the discount function given decision data. We validate the applicability of our proposed approach on two simulated problems. Our approach opens the way for the analysis of human discounting in sequential decision-making tasks.

Reinforcement Learning with Non-Exponential Discounting
Matthias Schultheis
Constantin A. Rothkopf
Centre for Cognitive Science
Centre for Cognitive Science
Technische Universität Darmstadt
Technische Universität Darmstadt
constantin.rothkope tu-darmstadt.de
Heinz Koeppl
Centre for Cognitive Science
Technische Universität Darmstadt heinz.koeppl @ tu darmstadt..e
Abstract
Commonly in reinforcement learning (RL), rewards are discounted over time using an exponential function to model time preference, thereby bounding the expected long-term reward. In contrast, in economics and psychology. it has been shown that humans often adopt hyperbolic discounting scheme, which is optimal when specific task termination time distribution is assumed. In this work, we propose a theory for continuous time model-based reinforcement learning generalized to arbitrary discount functions. This formulation covers the case in which there is non-exponential random termination time. We derive a Hamitton-Jccoi--Bae (HJB) equation characterizing the optimal policy and describe how it can be solved using a collocation method, which uses deep learning for function approximation. Further, show how the inverse RL problem can be approached, in which one tries to recover properties of the discount function given decision data. We validate the applicability of proposed approach on two simulated problems. Our approach opens the way for the analysis of human discounting in sequential decision decision-making making tasks.
Introduction
An often observed phenomenon in humans and animals is that they prefer rewards rather sooner than later [1]. It comes with no surprise that an animal searching for food aims to find as soon as possible and employees working in company prefer to be paid after each month instead of after each year. In behavioral experiments, it was ohserved that people are even willing to pay a price to receive rewards earlier [2]. It can be concluded that rewards become of less value in the future, phenomenon which is known as discounting [2].
Modeling and experimentally inferring discounting functions, which describe how values are dis- counted over time, already have a long history in economics and psychology, where many different functional forms have been proposed [3, 4, 5]. While from economic perspective, fixed interest rate seems reasonable, leading to an exponential discount function, human behavior oftentimes better described by hyperbolic curve [6]. The reason for this is that human decisions typically not consistent regarding shifting rewards in time, a phenomenon named preference reversal: We might find that subject prefers smaller reward the same day over larger reward one day after. However, when given the choice between the same smaller reward in 365 days, compared to the same
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

larger reward in 366 days, the subject is more likely to be willing to wait one day more for the larger reward.
While some literature has branded the observed discounting behavior as not being rational [7], there has been increasing amount of work identifying circumstances under which this behavior is indeed optimal [8. 9, 10]. One widely-adopted theory rationalizing hyperbolic discounting is to assume a constant risk for the reward become unavailable but with the risk being uncertain [11]. Under this condition, one should adapt the preference over time, as with time the expected risk decreases.
Discounting is also widely applied in the field of reinforcement learning (RL) and optimal control [12, 13, 14]. First, when modeling infinite horizon time objectives, a discount function is needed to make the expected long-term reward objective well-defined, as otherwise, it would become infinite. Second, for autonomous agents, it also makes sense to model a preference for earlier rewards in order to find shortest paths save time and energy. Third, the discount function can be interpreted the probability of termination inducing a specific end-time distribution [15, 12].
Despite these obvious connections, discounting models of psychology and reinforcement learning have remained mostly independent with few exceptions [16. 17]. Generalizing reinforcement learning to broader range of discount functions would enable solutions for applications in which the end time follows a specific distribution. On the other hand, methods for determining optimal decisions in sequential decision- pecision-making tasks with general end-time distributions would provide tools which can help to explain human decision decision-making making under uncertainty.
In this work, we present a theory for model-based reinforcement learning in continuous time based on non-exponential discount functions. First, we investigate the conditions under which the objective of maximizing the long-term reward formulated with hyperbolic discounting is well-defined. Second, we derive a Haniton--acoobiBellns describe how to solve it to obtain the optimal policy. Third, we provide an approach to tackle the inverse reinforcement learning (IRL) problem, to estimate parameters the discount function given decision data. Finally, we show the applicability of our proposed method on two simulated problems.
(HJB)-type equation for a general discount function and
2 Related work
Optimal control in continuous time and space has a long history with many classical works [18, 19, 20, 21]. Continuous-tinn reinforcement learning formulations have been developed [14] and various solution methods have been proposed [22, 23]. Solution approaches that solve the HJB equation directly include linearization techniques [24, 25], path integral formulations [26, 27, 12], and collocation based methods [28). In recent years, it has become increasingly popular to use neural networks for function approximation to solve the HJB equation [29, 30, 31, 32, 33].
Non-exponential discounting was considered in literature in behavioral economies, psychology, and neuroscience [4, 34. 2]. For humans and animals, preference reversal behavior has been described [35, 36, 5] and different functional forms for the discount function have been proposed [3, 37]. In most works, only non-sequential decision-
tasks have been studied, i.c., situations with single decision for each independent trial. In this line of work, the method of limits has been used to elicit discount functions based on binary decisions [38, 39, 40] or adjustment method has been applied [41]. Some methods have been proposed that specifically aim to efficiently estimate the discount function for binary choice responses [42, 43]. There have been few exceptions for which decision trials were actually dependent but could still be considered independently for the analysis [44, 45). Further, there has been research aiming find rational explanations for the encountered discounting behavior [8, 9, 10), e.g., by assuming uncertainty about a constant hazard [11].
Decision processes with non-exponential time distributions have been studied in the field of semi- Markov Decision Processes (SMDPs) [46, 47, 48, 49], where transition times between discrete states can follow arbitrary distributions. MDPs with quasi-hyperbolli discounting, for which all future rewards are additionally discounted by a constant factor, have been addressed in [50, 51, 52]. They can be used to model preference reversal but are limited to the specific form assumed. Stochastic processes with quasi-hyperbon management [53, 54, 55, 56].
discounting have been considered in financial economics for portfolio
More closely related to our work, Fedus et al. [17] presented an approximate method for solving MDPs with hyperbolic discounting. They approached the problem by solving the corresponding
2

---

proonentiallyddiscoone problem for many different discount factors and combining the results. Part of the approximation, however, that the value function and policy are static, failing to model preference reversals over time. Another line of work considers MDPs with discount factors that are coupled to the value function to imitate hyperbolic discounting behavior [16, 57]. As time is only considered indirectly through the magnitude of the value function, these approaches cannot be used for finding the solution to a given specific discount function or eliciting the discount function from data.
Finally, inverse reinforcement learning approaches have been mainly used learn reward functions given data [58, 59, 60, 61]. These methods were also applied to learn properties of human behavior [62, 63, 64, 65]. Other inverse approaches have focused learning dynamics models [66] or learning rules [67] from sequential decision maainn data.
Background
3.1 Survival analysis
In survival analysis [68], one is interested the duration until events occur. In its classical form one considers single event, for which the duration can be described as continuous random variable with cumulative distribution function F(I) P(T s 1) and probability density function f(t), where E R denotes the elapsed time. In survival analysis literature, F(t is known as the failure function and the survival function is defined as
5(t) = F(t) = P(T > t). The survival function monotonically decreasing and has the properties S(0) and lim S(t) O. By
the conditioning rule, one finds P(T > t T > to) S(1 ((to)) if t > to. The hazard rate is P(t
defined as at)) = lim A7
S <1+ Af f S T), yielding the relations
S(t - 5(t + At)
St)
and S(1) = exp (-((act)))
(1)) = lim
(1)
Ar-0 At
S(t)
S(t)
For constant hazard rate ((t) = A, one finds S(t) exp(- -X)), which can be shown to be the unique memory-less survival function [68], i.e., P(T t + At
P(T Af) for all
t, At E R,
3.2 Discounting and preference reversal
We consider the setting in which subject collects a single reward and shows form of time preference, i.e., rewards are desired rather sooner than later. We model the value of a reward r E IR as a function L. IR T -> R with I.(r, S(t) r, where S({) is the discount function decreasing with time regard S(t) as the survival function (Section 3.1), with the interpretation that the reward becomes unavailable with hazard rate (l) [11].
t R5- With the convention that ((t) and lim, S(t)
When assuming a constant hazard rate att) = X, we say that the subject discounts exponentially, as S(t) exp(-Xte By the memory-less property, have that if the subject prefers reward ri after t over 72 after f2, she or he would remain consistent with the clection if we presented the choice again later in time. On the other hand, if we assume constant but unknown hazard rate À with belief' p(A) = Gamma(X: X; ag. Bo), we obtain hyperbolic form for the expected survival function:
S(t: 0o, Bo) /. exp((>t))(()) dp
3o + 1)
The posterior belief over later point in time be derived using Bayes rule and is given by p(A (t) = Gamma(A, ao. 8o + t). The expected hazard rate a(t) is given by the posterior mean,
(()) / Ap(A
00
30 +t
For discount functions other than the exponential, such as the hyperbolic discount function in Eq. (2), the hazard rate varies over time and preferences among options may change.
3

---

3.3 Optimal control
In stochastic optimal control [69]. we consider a Markovian system with continuous state x(t) e R" evolving according to the stochastic differential equation (SDE) dX((t - f()(t), u(t) dt G(X(t),, (t)) (t) where t E IR denotes time, f: X x xu + 2' the drift function, G X' x U + X' R the dispersion matrix, and W I m--immeniionll Brownian motion. The goal of optimal control is to determine the control inputs u(t) E u given the current state x at time t, in order to maximize the expected long term discounted reward with reward function R X The solution is characterized by the optimal value function,
R.
[.," exp(-A((= - ())
(x) = max E (000
dr x(t) =
(4)
where maximization is carried out over all trajectories U[t,c)) == (u(r)]re(ttoo) The quantity
exp(-At) is the discount factor, which ensures convergence of the integral and models a preference for earlier rewards. À can be interpreted in terms of survival analysis as the hazard rate for termination (cf. Section 3.1). According to the principle of optimality, the stochastic Gamilton-JacobiBellle (HJB) equation, given by
XV (x)
max uel
R(x, u) + (x) f(x. u) + tr (x)(x, u)G(x, u)r}
provides a condition for the optimal value function. Here, partial derivatives are denoted by the index notation, i.c., Vx denotes the partial derivative of V w.r.t. x and Vxx the respective Hessian. Note that the optimal value function depends on time only indirectly through the state x due to the Markov property and the memory-less discount function. This dependence also applies to the optimal policy T X + Ut given by the maximizer of the right-hand side of the HJB equation, i.c..
1 R(x, u) VØ (x)) f(x, u) + 2 VV (x)
π (x) arg max uell
4 Reinforcement learning with general discount function
We consider a system as in stochastic optimal control (cf. Section 3.3) with continuous state space IR" and finite set of controls U. Instead of exponential discount function, allow for general survival functions (t) based on a time-dependent hazard rate o<t). This setting generalizes the discounting formulation of Section 3.2 sequential decisions. As will see later, the resulting value function will become time-dependent in contrast to Section 3.3 and we can further allow for time-dependent dynamics and reward without increasing complexity. We therefore assume for the state evolution the SDE (t) = f(X(t), u(t), t) dt + G(X(t), u(f), t) dW(() with f X x U x I + X' and G X' x x T + X' x R The objective function measuring the total expected discounted reward is given by
J (o) EE [[o ((T))((()))) u(T), d()))
(5)
with time- dependent reward function R X 4 B + R. Analogous Section 3.3, we define the expected reward-to-ge as the value function
[,,0 ((r,00)
S(T) S(t)
(*(xxt) t) = max E
u(7,) X(t)= =
(6)
where S(r)/S(t) is the probability of survival until time т, conditioned on the fact that one already has survived until time (ef. Section 3.1). In contrast to Eq. (4), the value function becomes time-dependent through the general survival function and also the optimal policy depends on time.
4.1 Technical requirements of model and discount function
First, the stochastic process defined in Section 4 must have strongly unique solution, which is the case if f and G grow most linearly in X and are Lipschitz continuous in the same variable [70]. Further, the optimal value function is only in rare cases smooth enough to be solution in the

---

"classical" sense. Instead, one considers viscosity solution [20], which satisfies the HJB equation in an appropriate generalized sense. A sufficient condition for the existence of such solution is that and G are continuous with bounded continuous first derivatives w.r.f. X and I, and bounded. R and S need be continuous and grow at most polynomially X and in an absolute sense. More details on the existence and uniqueness of viscosity solutions can be found in [20]. A third requirement is that the integral in the definition the value function Eq. (5) converges. For the case hyperbolic discount function as in Eq. (2), we find the following theorem:
Theorem 1. Consider the hyperbolic discount function in Eq. (2). If reward function R(x, u, t) is bounded above for all (x, u. E X x U B.. and oxg 1, the value function defined in equation If R(x, u, t) is bounded below for all (x, u, t) E X x U x B.,. and ao 1. the value function not well-defined as becomes infinite.
Eq. (6) is
Proof. See Appendix A.
In the examples considered later, we will assume a bounded reward function and a hyperbolic discount function with ogp > 1, for which Eq. (5) and Eq. (6) are well-defined.
4.2 HJB equation for general discount function
In the following, we give a brief overview of the derivation of the HJB equation for a general discount function. A more detailed derivation is provided in Appendix B. First, we split the integral in Eq. (6)
into two terms such that we obtain recursive formulation of the value function:
S(T) ((X(())))))))))) (T), u(T), S(L)
(x(x))) = max E U[G.IAI]
S(L + At) S(t)
(X + At), At) X(t) =
For the second term in the expectation, we apply Taylor expansion and Itō's formula [69] and obtain
A ('(X(t At), + Ar) = (X(()))) I. H+/
() (T))
H+A
((X))
V=((());7)dr dr
(+ 2 tr
[12 (X()), G(X()) G(X(T) dr + o(At).
Plugging this result into the equation above, dividing both sides by At. and taking the limit At + as well as calculating the expectation w.r.t. W(I) leads to the desired HJB equation
a(I)V"(x, = max u
R(x, u, Vx () (x, f(x, u,
+
(Vix (x, t) G(x, u, u, G(x,
where (t) can be recognized to be the hazard rate corresponding to the survival function S(t). We define the r.h.s. of the HJB equation in Eq. (7) without the maximization w.r.t. the action as
Qx u, = R(x, u. t) + V (x, t) (x.
+ 2 tr [V t) (xx, u. t) G(x, u, +)
so that the optimal policy is given by n" (x, 1) = arg max, Q(x, u, 1). Later on, we will consider hyperbolic discounting, for which the hazard rate att is given by Eq. (3), i.c. a(t) an/(30 t).
5

---

Algorithm 1: Computation of the optimal value function and policy for non-exp. discounting
Result: Optimal value function / (x, t), optimal policy T (x,t)
Input: Parameters 0 of the discount function, system model, number of iterations K Initialize parameters vi of neural network for modeling VV (x, t); for k 0 to K do
Sample batch of states and time points (I....
Push (x, ())= through the network to obtain V (1(1....N33
Use back proppagaatt to compute V. 1/
Evaluate EVV,", x, in Eq. (8) and determine maximizing action ú; for i ,,...,
Use back-propagation to compute the gradient of Σ, E(V, &2, w.r.t ;); Update & using the gradient:
end
return / (x, Th (x. arg max, Q(x, t,
4.3
Solving the HJB equation
In order to solve the HJB equation in Eq. (7), which is PDE, we apply a collocation- based method [28, 29, 31]. To do so, we first reformulate the HJB equation
E(V,x,t) = ax(t) + max R(x, t) + Vi(x t) + (x(,t) u,x,
(8)
+ 2 tr (Vxx t) u,
u) G(x u, t) =
and use a function approximator 1 (x, t) for V (xx The parameters &0 of the approximator can be determined by sampling random states x; and time points & and minimizing Σ, E'V", 42 w.r.t. th. Calculating the derivatives Vx t), V." (x, t), (x, t) and differentiating the objective function w.r.t. 1/ is straightforward via automatic differentiation when choosing a neural network as function approximator [71]. J. As & is not bounded, we need to choose a reparametrization of t which maps all é to the interval [0, 1) before feeding the values into the network. More details about the implemented parametrization and application of the collocation method are provided in Appendix D. The complete algorithm learn the value function and policy is provided in Algorithm 1.
4.4 Inverse reinforcement learning for inferring the discount function
When analyzing human behavior, one might be interested learning the underlying discount function that led to a set of observed choices. In contrast to standard inverse reinforcement learning settings, where the goal is to learn the underlying reward function R, here we assume that the reward function is given and the discount function S(t) is unknown and needs to be inferred. To learn the function, we assume a parametric form with parameters O.
As given data, we assume the states and time points at which the subject switches from one action to another one, i.e.,
fx, u .u + describing that in state x time t the action is switched t The observed decision maker is assumed to the optimal policy Txx () arg max, Q(x, u, z). Shortly before and after switching time I, we have Q(x(L - At), u 1- - A1) > Q(x(t At), ut At) and Q(x(t + At), +t At) < Q(x(t At), u t Åt). respectively, indicating that before t action u is preferred over ut and vice versa afterwards. By letting At + O, one finds (xx(t), u f) = Q(x(t), ut, .).. A sensible objective for the inverse problem is therefore minimize Σ, F (xi, u, uf li), defined as the squared difference of both Q values, i.e.,
F(x, ut .t)
[Q(x, u ,t) Q(x, u .t)²²
R(x, ut, (t), R(x, u .t) Vx(xx (f(x,ut,0) () f(x, ,()
(9)
1 tr (Vxx(x,t) (G(x, ut G(x, - - ,t) ,t)²)
+ 2

---

Algorithm 2: Computation of the gradient of /' w.r.t. 0 for inferring the discount function
Result: Gradient of F (Eq. (9)) w.r.t. 0 Eq. (10)
Input: Parameters 0 of the discount function, system model, number of iterations K Determine / (x, u, t) using Algorithm
Initialize parameters & of neural network for modeling Vgg (x, (); for k 0 to K do
Sample batch of states and time points (x, (
Push (x, f)i= through the networks to obtain V/
(i and V.D
Use mack-propagationn to compute (Vo") Ixi, (Vo") )e.. (V Xxxi
Evaluate E; HVV. V f,) f in Eq. (11) for I...., N:
Use back-propagation to compute the gradient of Σ, H(V,, Vol & &.) w.r.t. o;
Update using the gradient: end
return dF/d0 by evaluating Eq. (10) using Vi0 (x,
w.r.t. the parameters O of the discount function. The optimization needs to take Eq. (8) constraint into account to ensure that the HJB equation fulfilled. Note that Eq. (9) depends indirectly on the parameters 0 through the definition of the value function. However, the objective function F could also directly depend on 0, we consider the general case the following. In principle, the state at which the switching occurs also depends on 9. Nevertheless, will assume that the parameters have only a minor influence on the states of switching, so that terms including the variation of x w.r.t. can be neglected.
Gradient computation
For the minimization, is useful to determine the gradient of the objective function F w.r.t 6, which can be formulated using the chain rule for total derivatives
dF
Fe Fv + FVz (Vx) + Fvize (Vxx)ø
d0
= Fo F.VeVV Fv, (Vo)x Pyze (Vo)xx
(10)
By switching the order of the partial derivatives in the last step, have obtained expression for the gradient depending on Ve. The partial derivative Ve measures the influence of the discounting parameters the value function and is not straightforward to evaluate. We can get PDE for this quantity by observing that the constraint E(V", x, t) in Eq. (8) zero everywhere for the optimal value function V and therefore its derivative also needs to be zero. This method is known as the forward sensitivity method [72] and provides an additional PDE for the desired gradient Vo:
dE d0
0
F. + Vo + Fy (Vi) + Evy, (Vx) Flvex (Vxx)o
(11) To solve the PDE and obtain Ve. we can use the same procedure as in Section 4.3. To obtain the gradient, first has to solve the HJB equation Eq. (7) to obtain V then Eq. (11) to determine V Afterwards, Eq. (10) gives the gradient update the parameters of the discount function. The quantities Ve. (Vo)x. x- (Vo) for the derivative be computed via automatic differentiation as in Section 4.3. The complete algorithm for computing the gradient is presented in Algorithm
Eo + Vo + Ev, (Vo), + Ev, (Vo) Eve. (Vo)xx H(V vo,
5 Experiments
We tested our derived method on two simulated problems with a random termination time following a hyperbolic survival function. First, we solved the HJB equation in Eq. (7) using the collocation method (Algorithm 1) and computed the optimal policy. Then, for applying the inverse reinforcement learning method, we evaluated the objective function in Eq. (9) and computed gradients for different parameter sets 0
[ao, 86] of the hyperbolic discount function in Eq. (2) using Algorithm 2. For
7

---

A Leerned value function
B Learned function (o=0.5)
C Leamndd policy
Simulation
spend Invept
Invest
0.5
2.5
5.0
10.0
12.5
E Learned value function (exp.)
Learned (exp, b-0.5)
Gradients parameter pairs
values for parameter pairs
xpend
Insull
0.4
Figure| 1: Results for the investment problem. A Learned value function for account balances and time points for hyperbolic discounting. B Learned Q function for both actions over time (median and quantiles for 10 runs). C Learned policy showing preference reversals for all account balances except for b 1. D Simulation with account balance (top), interest rate (middle), and action (bottom) over time. E Learned value function for an exponential discount function, which is constant over time. G Learned Q function for an exponential discount function. G Gradients obtained for parameter pairs [oo. Su] of the discount function given simulated data on a 11 X 11 grid. The parameters used to generate the simulated trajectories are indicated by red cross. H Values of the objective function F (Eq. (9)) for parameter pairs on a 11 x 11 grid when applying the IRL method.
generating data, we randomly sampled starting states and determined the time points at which subjects would switch their action. Afterward, the determined time points were distorted by Gaussian noise.
All proposed methods were implemented in Python using the PyTorch framework [71] and available online!. The used hyperparameters are listed in Appendix F. As tasks, we considered an investment problem and a problem controlling point on line. In the following we provide brief overview of the considered problems, more details can be found in Appendix E.
5.1 Experimental tasks
In the investment problem, we model a subject having to decide whether to invest her or his income in the bank account leading future interests (rewards) spend the money for immediate reward. We model the state as the current balance of the bank account as well as the current interest rate. When the money is spent (action spend), the subject receives rewards with rate of 0.1 but the balance of the account remains unchanged. When the income is invested (action invest), the balance of the bank account increases with rate of 0.1 but there is additional reward. In both cases the subject receives rewards through interests, being proportional to the current balance on the bank account. We assume that the interest rate varies over time following Gaussian diffusion model. To keep the state bounded, we model a maximum balance for the account.
In the line problem, the task is to control point along line. The state consists of the current position of the point. Possible actions are left and right, which move the point to the respective direction, as well as stay, which does not move the point at all. When moving the point, there is Gaussian diffusion on the position and one has to pay a small action cost (negative reward). We model dependenn reward modeling high reward for distant states on one side and a low reward for close states on the other side. This task can be seen as continuous extension of the classical discounting tasks. A more detailed description of the functional form is provided in Appendix E.
hntpps:/ggit.wwth htt/bcssnooexxp--ee
8

---

Learned value function
B Learned Q function (x=d)
c Learned polkcy
D Simulation
sday rigtt
right
2
Inf
-1
E Learned vallue function (exp.)
F Leerned fun. (ex).
G Gradients for parameter pairs
F valuee for parameteer paire
000
0.
left
20-
stay
ri14
3
Figure 2: Results for the line problem. A Learned value function for hyperbolic discount function. Learned Q function for each action over time (median and quantiles for 10 runs) C Learned policy showing preference reversals. D Simulation showing the state (top) and action (bottom) over time. E Learned value function for an exponential discount function. G Learned Q function for an exponential discount function. G Gradients obtained for parameter pairs (oo, 8o] given simulated data. The parameter used to generate the simulated trajectories is indicated by a red cross. H Values of the objective function of the IRL methods for parameter pairs on 15 15 grid.
15 x 15 grid
5.2 Results
The proposed method in Algorithm I produces plausible value functions for the considered problems. Figure A shows the learned value function for the investment problem. The values are increasing with account balance (b), as expected rewards through interests become higher. Further in time, the value also increases, as the hazard rate can be assumed to be lower and one is expected to collect rewards for longer time. Figure B and C shows the learned Q function and policy, respectively. While it is advantageous to spend the income in the beginning, preference reversal occurs when the risk of termination is assumed be relatively low, and investing becomes more attractive. The simulation in Fig. D also reflects this behavior. For comparison, when assuming exponential discounting (Fig. E and F), the value and Q function do not show preference reversal and remain constant over time. With regard to the inverse approach, Fig. G shows the computed gradients of the IRL objective function F in Eq. (9) for different parameter pairs. One can observe that the gradients mostly point in the direction of the parameters used for simulation, indicated by red cross. Figure H shows the evaluated IRL objective function F for the parameter pairs, correctly displaying low values in the area close to the true parameters.
For the line problem, the computed value function is depicted in Fig. 2 A. Also in this task, the learned value increases over time and is high in areas close to highly rewarding states. Figure 2 B and C shows the learned Q function and policy respectively. In the beginning when the risk could be potentially high, if not close enough to the large reward, is best to move right to collect the smaller sooner reward. With time when the risk of termination decreases, it becomes increasingly advantageous to move left to collect the larger later reward. In between, there is a time span, during which it is best to stay: The moving cost is too high for moving right and shortly turning afterward, but the risk is not low enough for being worth it to move left to the higher reward. The simulation in Fig. 2 D presents a sampled trajectory, showing that the subject first moves right until the maximum smaller sooner reward at x 0.5 is reached and stays there for some time. After the risk has decreased over time, the simulated subject finally moves right to collect the larger later reward her or his preferences have reversed. As for the investment problem, the value and Q function (Fig. 2 E and F) do not model any time-varying behavior. With regard to the IRL. problem, the gradients and function values of F of Eq. (9) in Fig. 2 G and H also lead closely to parameter values used for the simulations.

---

6 Conclusion
In this work, we have proposed method for reinforcement learning with non-exponential discount functions. The approach can be used to solve decision-making problems with an arbitrary end-time distribution and to model human discounting behavior. First, we have shown the conditions for which the problem is well-defined when using a hyperbolic discount function. Then, we derived a HJB-type equation providing conditions for the optimal time-dependent value function. We presented how the obtained PDE can be approximately solved using collocation method, leading to the optimal policy. Further, we introduced an approach for the inverse problem, in which the discount function needs to be inferred given behavioral data. The application of our methods on two simulated problems led to plausible solutions, opening the way for further applications such as the use in human experiments.
Limitations and future work In our proposed methods, we assume finite action space. While this assumption applies to many behavioral experiments, it can be a limitation for the application to classical optimal control. To extend the method to continuous control, one has to determine how the maximization problem in the HJB equation solved. For strictly convex action costs, there have been approaches proposed that efficiently solve the maximization in the HJB equation [31], under some conditions even in closed-form [29]. Further, for Lipschitz-continun possible to approximate the control dynamics to avoid solving the optimization problem [73]. Another requirement of our method is that the model needs to be known. Model-free approaches, such as TD- error learning for general discount functions could be explored, in line with work such as [16]. Further, when approaching problems with large state spaces, the proposed collocation method is likely to converge slowly. For these cases, adapted collocation methods [30] or advantage updating approaches [23] could be considered instead. To apply our method to more advanced problems in financial engineering, such as modeling stocks with discontinuous returns [74], one has consider general jump-diffusion processes. While an extension to these models is straightforward from a theoretical point, we here left it out to keep the focus on the handling of the discount function. Regarding our inverse reinforcement learning approach, we have assumed the states and time points of the action switches to be given. While this assumption is reasonable for many human behavior experiments, it might be interesting to learn discount functions for given discretized trajectories instead. Also, incorporating an extended timing model of human decisions [75] instead of fixed-varin variance Gaussian diffusion would be an interesting extension.
controls, it is
In the future, we are planning apply our proposed methods in human experiments to get new insights to human discounting behavior. Characterizing individual human subjects by analyzing their behavior comes with the risk to be used with negative social impact. This matter can be counteracted by collecting only anonymized data for the application of our method.
Acknowledgments and Disclosure of Funding
We thank Bastian Alt for insightful discussions and for providing feedback on the first draft. This work was supported by the high-performance computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt and the project "Whitehox" funded by the Priority Program LOEWE of the Hessian Ministry of Higher Education, Science, Research and Art.
References
[1] Ted O' O'Donoghue and Matthew Rabin. The cconomics of immediate gratification. Journal of Behavioral Decision Making, 13(2):233-250, 2000. 2000.
[2] Shane Frederick, George Loewenstein, and Ted O'donoghue. Time discounting and time preference: A critical review. Journal of Economic Literature, 40(2):351-401, 2002.
[3] Todd L McKerchar, Leonard Green, Joel Myerson, T Stephen Pickford, Jade C Hill, and Steven C Stout, A comparison of four models of delay discounting in humans. Behavioural Processes, 81(2):256-2 2295666-2,,,-2 2009.
[4] Robert Henry Strotz. Myopia and inconsistency in dynamic utility maximization. The Review of Economic Studies, 23(3):165-180) 1955.
[5] Richard Thaler. Some empirical evidence on dynamic inconsistency. Economics Letters, 8(3):201-209 1981.
10

---

[6] James E Mazur. An adjusting procedure for studying delayed reinforcement. Quantitative Anulyves of Behavior, 5:55-73, 1987.
[7] Irving Fisher. Theory of interest: as determined by impatience spend income and opportunity to invest it. Augustusm Kelly Publishers, 1930.
[8] Partha Dasgupta and Eric Maskin. Uncertainty and hyperbolic discounting. American Economic Review, 95(4):1290 1299, 2005.
[9] Taiki Takahashi. Loss of self-control in intertemporal choice may be attributable to logarithmic time- perception. Medical Hypotheses, 65(4):691-695 2005.
[10] Debajyoti Ray and Peter Bossaerts. Positive temporal dependence of the biological clock implies hyperbolic discounting. Frontiers in Neuroxcience, 5:2, 2011.
[11] Peter Sozou. On hyperbolic discounting and uncertain hazard rates. Proceedings of the Royal Society of London. Series B: Biological Sciences, 265(1409) 1998.
[12] Richard S Sutton and Andrew G Barto. Reinforcement learning: An infroduction. MIT Press, 2018.
[13] Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
[14] Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12(1):219-245. 2000.
[15] Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
[16] William Alexander and Joshua W Brown. Hyperbolically discounted temporal difference learning. Neural Computation, 22(6):1511 1527, 2010.
[17] William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochellc. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:/902 1902006865 2019.
[18] Ruslan L Stratonovich. Conditional Markov processes and their application to the theory of optimal control. 1968.
[19] Harold Kushner and Paul G Dupuis. Numerical methods for stochastic control problems in continuous time, volume 24. Springer Science & Business Media, 2001.
[20] Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions, volume 25. Springer Science Business Media, 2006.
[21] Lev Semenovich Pontryagin. Mathematical theory of optimal processes. CRC Press, 1987.
[22] Kyriakos G Vamvoudakis and Frank L Lewis. Online actor-critic algorithm to solve the continuous infinite horizon optimal control problem. Automatica, 46(5):878-88) 888, 2010.
[23] Leemon C Baird. Reinforvement leamning continuous time: Advantage updating. In IEEE International Conference on Neural Networks. volume 4, pages 2448-24822222 2453. IEEE, 1994.
[24] David H Jacobson. New second-order and first-order algorithms for determining optimal control: differential dynamic programming approach. Journal of Optimization Theory and Applications, 2(6): 411 140, 1968.
[25] Yuval Tassa, Tom Erez, and William Smart. Receding horizon diffcrential dynamic programming. In
Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
[26] Hilbert Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment, 2005(1 1):P1 1011, 2005.
[27] Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. Journal of Machine Learning Research, 11 11:3137-3181, 2010.
[28] Alex Simpkins and Emanuel Todorov. Practical numerical methods for stochastic optimal control of biological systems in continuous time and space. In IEEE Symposiun Adaptive Dynamic Programming and Reinforcement Learning, pages 212-218. IEEE, 2009.
[29] Yuval Tassa and Tom Erez. Least squares solutions of the HJB equation with neural network Value-function approximators. IEEE Transactions Neural Networks, (K44):13111104 2007.

---

[30] Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364 2018.
[31] Michael Lutter, Boris Belousov, Kim Listmann, Debora Clever, and Jan Peters. HJB optimal feedback control with deep differential value functions and action constraints. Conference on Robot Learning. pages 640-650. PMI,R, 2020.
[32] Jiequn Han, Amulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. volume 115, pages 8505-8510. National Academy of Sciences, 2018.
[33] Bastian Alt, Matthias Schultheis, and Heinz Koeppl. POMDPs in continuous time and discrete spaces. In Advances in Neural Information Processing Systems, volume 33, pages 13151-13162. Curran Associates, Inc., 2020.
[34] Richard H Thaler and Hersh M Shefrin, An cconomic theory of self-control. Journal of Political Economy, 89(2):392 406, 1981.
[35] George Ainslie and Richard J Hermstein. Preference reversal and delayed reinforcement. Animal Learning & Behavior, 4442476-482, 1981.
[36] Leonard Green, Nathanael Fristoe, and Joel Myerson. Temporal discounting and preference reversals in choice between delayed outcomes. Psychonomic Bulletin & Review, 13):383-389, 1994.
[37] Steffen Andersen, Glenn W Harrison, Morten Lau, and E Elisabet Rutström. Discounting behavior: A reconsideration. European Economic Review, 71:15-33, 2014.
[38] Jerry B Richards, Suzanne H Mitchell, Harriet De Wit, and Lewis S Sciden. Determination of discount functions in rats with adjusting- 67(3):353-366, 1997.
-amuuut amount procedure. Journal of the Experimental Analysis of Behavior,
[39] Mariheth Coller and Melonie B Williams. Eliciting individual discount rates. Experimental Economics, 2 12):177-127, 1999.
[40] Steffen Andersen, Glenn Harrison, Morten Lau, and Elisabet Rutström. Eliciting risk and time preferences. Econometrica, 76(3):583- 618, 2008.
[41] Jawwad Noor. Hyperbolic discounting and the standard model: eliciting discount functions. Journal of Economic Theory, 144(5)20077-2883, 2009.
[42] Daniel Cavagnaro, Gabriel Aranovich, Samuel M McClure, Mark A Pitt, and Jay Myung. On the functional form of temporal discounting: optimized adaptive test. Journal of Risk and Uncertainty, 52 (3):233-254, 2016
[43] Jorge Chang, Jiseob Kim, Byoung- Zhang, Mark A Pitt. and Jay Myung. Modeling delay discounting using gaussian process with active learning. In CogSci, pages 1479-1485, 2019.
[44] Nicolas Schweighofer, Kazuhiro Shishida, Cheol Han. Yasumasa Okamoto, Saori C Tanaka, Shigeto Yamawaki, and Kenji Doya. Humans can adopt optimal discounting strategy under real-time constraints. PLoS Computational Biology, 2(11) 1)22152, 2006.
[45] Maayke Suzanne Seinstra, Manuela Sellitto, and Tobias Kalenscher. Rate maximization and hyperbolic discounting in human experiential intertemporal decision making, Behavioral Ecology. 29(1): 1911):193-000 193-233, 2018.
[46] Ronald A Howard. Semi-Markovias decision processes. Bulletin of the International Statistical Institute, 40(2):625 652, 1963.
[47] V Korolyuk, S M Brodi, and A F Turbin. Semi-Markov processes and their applications. Journal of Soviet Mathematics, 4(3):244-280 280, 1975.
[48] Sheldon Ross. Average cost semi-Markov decision processes. Journal of Applied Probability, 7(3): 649-656, 1970.
[49] Steven Bradtke and Michael Duff. Reinforcement learning methods for continuous- Markov decision problems. In Advances in Nearal Information Processing Systems, volume 7. MIT Press, 1994.
[50] Anna Jaskiewicz and Andrzej S Nowak. Markov decision processes with quasi-hyperbolis discounting. Finance and Stochastics, 25(2):189-25 2299,229,,0990 2021.
12

---

[51] Tomas Bjork and Agatha Murgoci. A general theory of Markovian time inconsistent stochastic control problems. SSRN, 2010.
[52] Tomas Björk and Agatha Murgoci. A theory of Markovian time-inconsistent stochastic control in discrete time. Finance and Stochastics, 18(3):545-55) 5922, 2014.
[53] Steven R Grenadier and Neng Wang. Investment under uncertainty and time-inconsistent preferences. Journal of Financial Economics, 84(1):2-39, 2007.
[54] Ziran Zou, Shou Chen, and Lei Wedge. Finite horizon consumption and portfolio decisions with stochastic hyperbolic discounting. Journal of Mathematical Economics, 52:70-80, 2014.
[55] A Chunxiang, Zhongfei Li, and Fan Wang. Optimal investment strategy under time-inconsistent preferences and high-water mark contract. Operations Research Letters, 44(2):212-218, 2016.
[56] Shumin Chen, Zhongfei Li, and Yan Zeng. Optimal dividend strategy for general diffusion process with time-inconsistent preferences and ruin penalty. Journal on Financial Mathematics, 911):274-314, 2018.
[57] Hado van Hasselt, John Quan, Matteo Hessel, Zhongwen Xu, Diana Borsa, and André Barreto. General non-linear Bellman equations. arXiv preprint arXiv: /9970006777 2019.
[58] Andrew Y Ng. Stuart J Russell, al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, 2000.
[59] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Interna- rional Conference on Machine Learning, volume 5, 2004.
[60] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K. Dey, al. Maximum entropy inverse reinforcement learning. AAAI, volume 23, pages 1433- 1438, 2008
[61] Divyansh Garg, Shuvam Chakraborty, Chris Cundy. Jiaming Song, and Stefano Ermon. IQ-Leam:: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems, volume 34, pages 4028 4039. Curran Associates, Inc., 2021.
[62] Katja Mombaur, Anh Truong, and lean-Paul Laumond. From human to humanoid locomotion an inverse optimal control approach. Autonomous Robots, 28(3):369-335 2010.
[63] Constantin A Rothkopf and Dana H Ballard. Modular inverse reinforcement learning for visuomotor behavior. Biological cybernetics, 107(4):477-49) 4900, 2013.
[64] Katharina Muelling, Abdeslam Boularias, Betty Mohler. Bernhard Schölkopf. and Jan Peters. Learning strategies in table tennis using inverse reinforcement learning. Biological Cybernetics, 108(5):603-619, 2014.
[65] Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. In Advances in Neural Information Processing Systems, volume 34, pages 9429-9442. Curran Associates, Inc., 2021.
[66] Matthew Golub, Steven Chase, and Byron Yu. Learning an internal dynamics model from control demonstration. In International Conference on Machine Learning, pages 606-614. PMLR, 2013.
[67] Zoe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making. In Advances in Neural Information Processing Systems, volume 33, pages 3442-3455 3453. Curran Associates, Inc., 2020.
[68] Odd Aalen, Ornulf Borgan, and Hakon Gjessing. Survival and event history analysis: process point of view. Springer Science Business Media, 2008.
[69] Floyd Hanson. Applied stochastic processes and control jump-diffusions: modeling, analysis and computation.
computation SIAM, 2007.
[70] Simo Särkkä and Amn Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanee deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
13

---

[72] Herschel Rabitz, Mark Kramer, and D Dacol. Sensitivity analysis in chemical kinetics. Annual Review of Physical Chemistry, 34(1):119441, 1983.
[73] Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-Jacohi deep Q-leaming for deterministic continuous- time systems with Lipschitz continuous controls. Journal of Machine Learning Research, 22:206-L 2021.
[74] Robert C Merton, Option pricing when underlying stock returns are discontinuous. Journal of Financial Economics, 3(1-2):125-144, 1976.
[75] Vijay Mohan K Namboodiri, Stefan Mihalas, Tanya Marton, and Marshall Gilmer Hussain Shuler. A general theory of intertemporal decision-making and the perception of time. Frontiers in Behavioral Neuroxcience, 8:61, 2014.
14


Reviews:
[{"review_id": "wMQKZ-7LpGD", "reviewer": "Reviewer_snXr", "summary": "This paper provides a mathematical derivation of the HJB equation for a general discount factor (i.e., discount factor that is a function of time, as opposed to the traditionally fixed exponential discount rate) in the continuous-time optimal control / reinforcement learning setting. The paper provides two algorithms: one for solving the HJB (computing the value function), and the other for inverse determination of the discount factor parameters given a known reward and time of preference reversal. The validity of the derivations is demonstrated in two toy experiments.", "questions": "See main review. ", "limitations": "Yes, I think the limitations are addressed (but see main review). ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Significance/novelty:** Despite the authors' positioning of this as a \"reinforcement learning\" paper, it appears to be more of an optimal control / behavioral economics paper. The experiments do not use any baselines, and merely exist as empirical validations of the derived HJB + algorithms. The main practical application that is claimed is for the study of human discounting, rather than as work that has applications to training RL agents. This is fine, but it is a bit outside my expertise, and perhaps a bit incongruent with the framing RL framing (e.g., Alg 1. does not take in data, which I would consider to be the defining feature of RL, but rather the system model), and the data used for the \"Inverse RL\" task is rather crude. I would consider the IRL bit to be more of an exercise in elicitation. \n\nIn that regard, I feel the connections to related work are underdeveloped, and I'm not sure how significant the problem treated is. I'm left with the following questions:\n\n- What is the closest work in Optimal Control, and what are the specific contributions of this beyond that? How standard vs non-obvious is the derivation here really? I haven't worked with PDEs/HJB before, so I'm not sure if the derivation is basic, and I would not be surprised if this exists, at least, in a very close form. E.g., In Zou et al., Finite horizon consumption and portfolio decisions with stochastic hyperbolic discounting, 2014 [Not cited], \"*Section 3 derives the HJB equation for sophisticated individuals with stochastic hyperbolic discounting.*\", and a Scholar search for *\"hyperbolic discounting\" HJB* reveals 266 results. I would want to understand how this contribution fits into this other work, rather than the high level, no-contrast description provided at lines 57-68. \n\n- Is there really no comparable elicitation method from behavioral economics that could be contrasted against? A Google Scholar search for *\"hyperbolic discounting\" elicitation* reveals over 5000 results. Several on the first page appear very relevant and aren't cited. Certainly if economists have observed that humans follow a hyperbolic discounting schedule, they must have ways of eliciting this (i.e., doing something comparable to the Inverse RL algorithm proposed in Subsection 4.4) --- how is this problem treated in behavioral economics, and how does this work compare? \n\n- Is this work relevant at all to current (deep) reinforcement learning research? For example, I would say the discussion in Fedus et al. (2019) is complete / well positions the usage of hyperbolic discounting for purposes of reinforcement learning. The Pathworld experiments and the discussion surrounding them make clear when hyperbolic discounting might be useful. It also shows a practical application of the techniques to a relevant modern baseline (albeit the effectiveness has nothing to do with hyperbolic discounting). \n\n**Quality/Clarity** I did not verify the derivations/proofs, and am not familiar with the algorithm techniques (collocation method) to comment on the technical validity. That said, the paper reads well. At a high level (I did understand all the notation, and have reviewed a bit on HJB), the math checks out / \"looks correct\". The writing is overall solid. The introduction could be more concise, with more focus on the relevance of the present work rather than discounting / hyperbolic discounting as a whole. And see my comments on related work above. \n\n**Bottom line**: While this looks correct and is well written / clear, I am a bit wary of recommending accept given the positioning and lack of clear contrast against past work. I would be willing to change my view given confidence from other reviewers on the novelty/positioning issues, and a solid rebuttal from the authors. \n\nEDIT: I found the authors rebuttal responsive with regards to the questions above and increased my score accordingly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZC4OcvXYf1P", "reviewer": "Reviewer_7uF8", "summary": "This paper introduces a reinforcement learning method for computing optimal policies for non-exponential discount functions in continuous-time environments, as well as an inverse reinforcement learning method for estimating the parameters of the discount function based on the learnt policy, assuming that the reward function itself is known. The approach relies on solving a HJB equation. The hyperbolic discount function is given particular attention, but the proposed methods apply to essentially any discount function.\n\nAny discount function besides exponential discounting can induce preferences that are inconsistent over time. This leads to some ambiguity as to what it should mean for a policy or action to be \"optimal\" under such a discount function. In the literature on dynamic choice, there are three natural candidates which are often suggested: naive choice, sophisticated choice, and resolute choice. The notion of optimality that this paper relies on is analogous to resolute choice.", "questions": "The main questions that this paper raises for me are,\n\n1. Can the proposed methods be used in discrete-time settings? I imagine the answer is yes, but I would have liked to see some concrete mention of how to do this (even if just a short remark).\n\n2. How scaleable are the proposed methods? The experiments provided in the paper both involve very small environments, and the paper does not provide the time it took for the algorithms to converge in these environments. The authors do remark that \"when approaching problems with large state spaces, the proposed collocation method is likely to converge slowly\", but I found that I did not get a good sense of what exactly this means in practice. Do these methods only work in toy environments, with states that have one or two parameters, or would they be applicable to environments whose states have at least a few thousand parameters or so?\n\n3. Computing a (resolute) optimal policy, for a given discount function, is equivalent to computing an optimal policy in a modified MDP that, at each time step, terminates with a certain probability, and where the state-space includes the time (this is also pointed out by this paper in Appendix H). Moreover, if we include the time in the state space, then we could alternatively also just directly modify the reward function to reflect the discounting. Does the reinforcement learning method proposed in this paper provide any benefits over either of these two approaches, and if so, what are they? It seems to me that either of these two approaches would be easier to implement, and it would moreover be straightforward to combine either of them with arbitrary state-of-the art methods. I would certainly like to see some discussion of if and why Algorithm 1 improves on these two strategies. If it does not, then that seems to reduce the impact of the paper (although the paper of course also includes an IRL method, which I imagine was the original motivation for this work, given the final remark of the paper).\n\n4. There has been some work on inverse reinforcement learning for non-exponential discounting (see \"Irrationality can help reward inference\" by Chan et al, and \"Learning the Preferences of Ignorant, Inconsistent Agents\" by Evans et al). Moreover, in the fields of preference elicitation and behavioural economics, there has been a very large amount of work on inferring discount rates from behaviour. How does the inverse reinforcement learning method proposed in this paper relate to and improve upon what is provided by this earlier work? In my opinion, an extensive discussion of this question has to be included in the paper.\n\nIf these points are addressed (especially 3 and 4), then that could convince me to increase my rating. I would be particularly keen to see a more detailed comparison between the proposed IRL algorithm and existing methods from the preference elicitation literature. ", "limitations": "I find that this paper provides a fully satisfactory discussion of its limitations, as well as of its potential negative societal impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThere has been some work on reinforcement learning with non-exponential discounting, but that work is fairly preliminary, and this paper certainly approaches the problem from a different angle than those earlier papers. However, I have some questions about the benefits of the approach proposed in this paper, see my question 3 below. On the inverse reinforcement learning side, there has also been some earlier work concerning non-exponential discounting, but to my knowledge, that work only concerns the problem of inferring the reward function when the discount function is known, whereas this paper concerns the problem of inferring the discount function when the reward function is known. The inverse reinforcement learning method is therefore certainly novel in that context. However, in the economics literature, there has been quite a lot of work on the problem of inferring a discount function based on consumer behaviour. I do not know how the inverse reinforcement learning method proposed in this paper relates to that earlier work in economics, see my question 4 below.\n\nQuality:\n\nThe work presented in this paper is of high quality.\n\nClarity:\n\nI found this paper to be very clear, except for my questions 1-2 below. \n\nSignificance:\n\nMy guess is that the problem which this paper addresses probably would be perceived as somewhat niche by the broader machine learning research community. However, it is very relevant to fields such as e.g. preference elicitation. My personal opinion is therefore that the problem is significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PoSH9f7F_rm", "reviewer": "Reviewer_e6eZ", "summary": "This paper generalizes the HJB equation in the infinite time-horizon setting to arbitrary discounting schemes. Based on this HJB equation, a collocation-based deep learning algorithm is presented for reinforcement learning under hyperbolic discounting. Moreover, the authors study the problem of inferring the discount function from data, and demonstrate a deep learning algorithm for accomplishing this task. Lastly, the authors demonstrate the performance of both their RL and discount-function-learning algorithms in two simple domains.", "questions": "Comments and Questions\n======================\n\n  **Line 104**: \"... with the interpretation that the reward becomes\n  unavailable with hazard rate $\\alpha(t)$\" -- Should it not be the case\n  that *all future rewards* become unavailable at that hazard rate? As\n  written, it sounds like reward at time $t$ can become unavailable, but\n  reward at time $t + dt$ can still be available.\n\n  **Equation after Line 127**: I believe the dimensions are incorrect\n  here. The term on the LHS as well as the first two terms on the RHS\n  are scalar-valued, but the final term on the RHS (I believe) has\n  dimensions $m\\times m$ -- the Hessian has dimensions $m\\times m$, and\n  since $G(\\mathbf{x},\\mathbf{u})\\in\\mathbf{R}^m$,\n  $GG^\\top(\\mathbf{x},\\mathbf{u})$ should also have dimensions $m\\times\n  m$.  I believe there should be a trace operator on that final term.\n  Also, it says\n  $R:\\mathcal{X}\\times\\mathcal{U}\\times\\mathbf{R}_0^+\\to\\mathbf{R}$, but\n  in the equations we have the term $R(\\mathbf{x}, u)$ -- an argument is\n  missing (and I believe $u$ should be $\\mathbf{u}$). Is the reward\n  function time-homogeneous?\n\n  **Theorem 1**: It would be helpful to have a refresher on the meaning of\n  $\\alpha_0$ in the statement of the theorem. Also, why does $R$ need to\n  be bounded below in order for the integral to diverge? I suspect what\n  was meant was\n\n  > If $\\alpha_0\\leq 1$, the value function diverges\n  > regardless of the boundedness of $R$.\n\n  Also, I don't think it's correct to say that \"the value function\n  diverges\". Divergence is a property of a sequence, not a function. I\n  believe \"the value function is not well-defined\", like in the first\n  part of the theorem, is better.\n\n  **Line 275**: I suspect continuous action spaces can be supported\n  without much trouble using a similar technique as given in [2], which\n  only requires the assumption that the control signal is Lipschitz\n  continuous.\n\n\nMinor issues / grammatical errors\n=================================\n\n  **Line 59**: \"Solutions approaches that solve the HJB equation directly,\n  include...\" -- that comma shouldn't be there.\n\n  **Line 60**: When talking about path integral approaches, I believe \"A\n  Generalized Path Integral Control Approach to Reinforcement Learning\"\n  [1] deserves a mention.\n\n  *Line 63*: Personally, I find this paragraph to be redundant given the\n  introduction.\n\n\nReferences\n==========\n\n  [1] Theodorou, Evangelos, Jonas Buchli, and Stefan Schaal. \"A\n  generalized path integral control approach to reinforcement learning.\"\n  The Journal of Machine Learning Research 11 (2010): 3137-3181. \n\n [2] Kim, Jeongho, Jaeuk Shin, and Insoon Yang. \"Hamilton-Jacobi Deep\n  Q-Learning for Deterministic Continuous-Time Systems with Lipschitz\n  Continuous Controls.\" J. Mach. Learn. Res. 22 (2021): 206-1.\n", "limitations": "Limitations have been adequately addressed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nIt was immediately evident that the paper is written in great English, making the text very easy and comfortable to follow for the most part. The authors should be commended for this.\n\nI appreciate the authors' ambition to study this problem in continuous time, which is often overlooked. The collocation algorithm, while fairly standard, is not particularly common in RL, and I believe the RL community should be interested by it. Especially given the time-inhomogeneity of the value function with arbitrary discounting, the collocation method seems quite elegant.\n\nMoreover, I do believe the \"inverse RL\" problem studied in this paper -- that being the problem of inferring the discounting function -- is an interesting one. I can imagine the algorithm presented in this paper to be impactful in economics / neuroscience / psychology. \n\n# Weaknesses\nThere are a few mathematical issues and concerns, however they're all minor and surely they can be easily addressed.\n\nI found that the discussion of experimental results is highly qualitative/speculative. Essentially, I don't find that the experimental results do a good job of convincing the reader that arbitrary discounting is valuable. It would have been nice, for example, to see an example where perhaps a nonstandard discounting scheme leads to a value function that is much more intuitively correct than that with an exponential discounting scheme.\n\nFinally, I believe the novelty of the contributions is relatively limited. I do believe some of the contributions are valuable -- particularly the general discounting HJB equation -- however I believe the paper could have a lot more impact with a more developed experimental analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9B58mnSwdhs", "reviewer": "Reviewer_MBPJ", "summary": "The paper studies continuous-time reinforcement learning with general discounting rewards and hyperbolic survival functions under the framework of optimal control (i.e., a generalization of the HJB equation). The extension of the HJB equation leads to a value-based learning method, and the authors discussed how to apply inverse reinforcement learning to infer the discount function. The authors conduct experiments on two simulated problems and demonstrate the effectiveness of their approaches.", "questions": "See the weakness part.", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Advantages:\n\n- The technique contribution of the paper is good. Generalization of the HJB equation makes sense to me and studying continuous time RL is also interesting.\n- The motivation of inverse RL is very strong to me, and I think the proposed method that exploits the structure of the dynamic model is novel.\n\nWeakness:\n\nI’d say that the paper seems over-claimed. The title suggests a general reinforcement learning algorithm, and the paper claims arbitrary discount functions. Unfortunately, these arguments do not hold in my view.\n\nRegarding reinforcement learning,  I would say that the paper only studies a very limited set of RL problem. The requirement of knowing the forward model $f$ and $G$ makes the proposed approach more like a model-based RL or optimal control instead of reinforcement learning. Besides, I believe the method at least requires the state space to be continuous and the drift function twice differentiable so that the value function could be differentiable w.r.t. states. The latter assumption is not explicitly stated in the paper. And with the paper’s general discounting function, it is not easy to ensure that the value function, which is the solution to an ODE, is twice-differentiable w.r.t. the states (as required by the HJB). Additional assumptions like some lipschitz conditions (as in Picard–Lindelöf) might be necessary. Though the conclusion may hold under some mild conditions, I urge the authors to carefully examine those conditions for strictness.\n\nSimilarly, I appreciate the study of hyperbolic discounting, but I don’t think using it to claim an arbitrary distribution is proper. For general discounting, a bounded $R$ may not guarantee a valid value or value gradients. For example, considering a chaotic system, a slight change of the initial state would result in a huge difference in the future. When the discounting function is not carefully specified, the value may have an unbounded gradient in the end.\n\nA minor issue is the lack of human experiments. The key motivation is to study human’s behavior. However, there are no human experiments in the paper. I think this will largely downgrade the contribution of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept