Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies continuous-time reinforcement learning with general discounting rewards and hyperbolic survival functions under the framework of optimal control (i.e., a generalization of the HJB equation). The extension of the HJB equation leads to a value-based learning method, and the authors discussed how to apply inverse reinforcement learning to infer the discount function. The authors conduct experiments on two simulated problems and demonstrate the effectiveness of their approaches.
strengths_and_weaknesses: Advantages:

- The technique contribution of the paper is good. Generalization of the HJB equation makes sense to me and studying continuous time RL is also interesting.
- The motivation of inverse RL is very strong to me, and I think the proposed method that exploits the structure of the dynamic model is novel.

Weakness:

I’d say that the paper seems over-claimed. The title suggests a general reinforcement learning algorithm, and the paper claims arbitrary discount functions. Unfortunately, these arguments do not hold in my view.

Regarding reinforcement learning,  I would say that the paper only studies a very limited set of RL problem. The requirement of knowing the forward model $f$ and $G$ makes the proposed approach more like a model-based RL or optimal control instead of reinforcement learning. Besides, I believe the method at least requires the state space to be continuous and the drift function twice differentiable so that the value function could be differentiable w.r.t. states. The latter assumption is not explicitly stated in the paper. And with the paper’s general discounting function, it is not easy to ensure that the value function, which is the solution to an ODE, is twice-differentiable w.r.t. the states (as required by the HJB). Additional assumptions like some lipschitz conditions (as in Picard–Lindelöf) might be necessary. Though the conclusion may hold under some mild conditions, I urge the authors to carefully examine those conditions for strictness.

Similarly, I appreciate the study of hyperbolic discounting, but I don’t think using it to claim an arbitrary distribution is proper. For general discounting, a bounded $R$ may not guarantee a valid value or value gradients. For example, considering a chaotic system, a slight change of the initial state would result in a huge difference in the future. When the discounting function is not carefully specified, the value may have an unbounded gradient in the end.

A minor issue is the lack of human experiments. The key motivation is to study human’s behavior. However, there are no human experiments in the paper. I think this will largely downgrade the contribution of the paper.
questions: See the weakness part.
limitations: yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper provides a mathematical derivation of the HJB equation for a general discount factor (i.e., discount factor that is a function of time, as opposed to the traditionally fixed exponential discount rate) in the continuous-time optimal control / reinforcement learning setting. The paper provides two algorithms: one for solving the HJB (computing the value function), and the other for inverse determination of the discount factor parameters given a known reward and time of preference reversal. The validity of the derivations is demonstrated in two toy experiments.
strengths_and_weaknesses: **Significance/novelty:** Despite the authors' positioning of this as a "reinforcement learning" paper, it appears to be more of an optimal control / behavioral economics paper. The experiments do not use any baselines, and merely exist as empirical validations of the derived HJB + algorithms. The main practical application that is claimed is for the study of human discounting, rather than as work that has applications to training RL agents. This is fine, but it is a bit outside my expertise, and perhaps a bit incongruent with the framing RL framing (e.g., Alg 1. does not take in data, which I would consider to be the defining feature of RL, but rather the system model), and the data used for the "Inverse RL" task is rather crude. I would consider the IRL bit to be more of an exercise in elicitation. 

In that regard, I feel the connections to related work are underdeveloped, and I'm not sure how significant the problem treated is. I'm left with the following questions:

- What is the closest work in Optimal Control, and what are the specific contributions of this beyond that? How standard vs non-obvious is the derivation here really? I haven't worked with PDEs/HJB before, so I'm not sure if the derivation is basic, and I would not be surprised if this exists, at least, in a very close form. E.g., In Zou et al., Finite horizon consumption and portfolio decisions with stochastic hyperbolic discounting, 2014 [Not cited], "*Section 3 derives the HJB equation for sophisticated individuals with stochastic hyperbolic discounting.*", and a Scholar search for *"hyperbolic discounting" HJB* reveals 266 results. I would want to understand how this contribution fits into this other work, rather than the high level, no-contrast description provided at lines 57-68. 

- Is there really no comparable elicitation method from behavioral economics that could be contrasted against? A Google Scholar search for *"hyperbolic discounting" elicitation* reveals over 5000 results. Several on the first page appear very relevant and aren't cited. Certainly if economists have observed that humans follow a hyperbolic discounting schedule, they must have ways of eliciting this (i.e., doing something comparable to the Inverse RL algorithm proposed in Subsection 4.4) --- how is this problem treated in behavioral economics, and how does this work compare? 

- Is this work relevant at all to current (deep) reinforcement learning research? For example, I would say the discussion in Fedus et al. (2019) is complete / well positions the usage of hyperbolic discounting for purposes of reinforcement learning. The Pathworld experiments and the discussion surrounding them make clear when hyperbolic discounting might be useful. It also shows a practical application of the techniques to a relevant modern baseline (albeit the effectiveness has nothing to do with hyperbolic discounting). 

**Quality/Clarity** I did not verify the derivations/proofs, and am not familiar with the algorithm techniques (collocation method) to comment on the technical validity. That said, the paper reads well. At a high level (I did understand all the notation, and have reviewed a bit on HJB), the math checks out / "looks correct". The writing is overall solid. The introduction could be more concise, with more focus on the relevance of the present work rather than discounting / hyperbolic discounting as a whole. And see my comments on related work above. 

**Bottom line**: While this looks correct and is well written / clear, I am a bit wary of recommending accept given the positioning and lack of clear contrast against past work. I would be willing to change my view given confidence from other reviewers on the novelty/positioning issues, and a solid rebuttal from the authors. 

EDIT: I found the authors rebuttal responsive with regards to the questions above and increased my score accordingly. 
questions: See main review. 
limitations: Yes, I think the limitations are addressed (but see main review).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a reinforcement learning method for computing optimal policies for non-exponential discount functions in continuous-time environments, as well as an inverse reinforcement learning method for estimating the parameters of the discount function based on the learnt policy, assuming that the reward function itself is known. The approach relies on solving a HJB equation. The hyperbolic discount function is given particular attention, but the proposed methods apply to essentially any discount function.

Any discount function besides exponential discounting can induce preferences that are inconsistent over time. This leads to some ambiguity as to what it should mean for a policy or action to be "optimal" under such a discount function. In the literature on dynamic choice, there are three natural candidates which are often suggested: naive choice, sophisticated choice, and resolute choice. The notion of optimality that this paper relies on is analogous to resolute choice.
strengths_and_weaknesses: Originality:

There has been some work on reinforcement learning with non-exponential discounting, but that work is fairly preliminary, and this paper certainly approaches the problem from a different angle than those earlier papers. However, I have some questions about the benefits of the approach proposed in this paper, see my question 3 below. On the inverse reinforcement learning side, there has also been some earlier work concerning non-exponential discounting, but to my knowledge, that work only concerns the problem of inferring the reward function when the discount function is known, whereas this paper concerns the problem of inferring the discount function when the reward function is known. The inverse reinforcement learning method is therefore certainly novel in that context. However, in the economics literature, there has been quite a lot of work on the problem of inferring a discount function based on consumer behaviour. I do not know how the inverse reinforcement learning method proposed in this paper relates to that earlier work in economics, see my question 4 below.

Quality:

The work presented in this paper is of high quality.

Clarity:

I found this paper to be very clear, except for my questions 1-2 below. 

Significance:

My guess is that the problem which this paper addresses probably would be perceived as somewhat niche by the broader machine learning research community. However, it is very relevant to fields such as e.g. preference elicitation. My personal opinion is therefore that the problem is significant.
questions: The main questions that this paper raises for me are,

1. Can the proposed methods be used in discrete-time settings? I imagine the answer is yes, but I would have liked to see some concrete mention of how to do this (even if just a short remark).

2. How scaleable are the proposed methods? The experiments provided in the paper both involve very small environments, and the paper does not provide the time it took for the algorithms to converge in these environments. The authors do remark that "when approaching problems with large state spaces, the proposed collocation method is likely to converge slowly", but I found that I did not get a good sense of what exactly this means in practice. Do these methods only work in toy environments, with states that have one or two parameters, or would they be applicable to environments whose states have at least a few thousand parameters or so?

3. Computing a (resolute) optimal policy, for a given discount function, is equivalent to computing an optimal policy in a modified MDP that, at each time step, terminates with a certain probability, and where the state-space includes the time (this is also pointed out by this paper in Appendix H). Moreover, if we include the time in the state space, then we could alternatively also just directly modify the reward function to reflect the discounting. Does the reinforcement learning method proposed in this paper provide any benefits over either of these two approaches, and if so, what are they? It seems to me that either of these two approaches would be easier to implement, and it would moreover be straightforward to combine either of them with arbitrary state-of-the art methods. I would certainly like to see some discussion of if and why Algorithm 1 improves on these two strategies. If it does not, then that seems to reduce the impact of the paper (although the paper of course also includes an IRL method, which I imagine was the original motivation for this work, given the final remark of the paper).

4. There has been some work on inverse reinforcement learning for non-exponential discounting (see "Irrationality can help reward inference" by Chan et al, and "Learning the Preferences of Ignorant, Inconsistent Agents" by Evans et al). Moreover, in the fields of preference elicitation and behavioural economics, there has been a very large amount of work on inferring discount rates from behaviour. How does the inverse reinforcement learning method proposed in this paper relate to and improve upon what is provided by this earlier work? In my opinion, an extensive discussion of this question has to be included in the paper.

If these points are addressed (especially 3 and 4), then that could convince me to increase my rating. I would be particularly keen to see a more detailed comparison between the proposed IRL algorithm and existing methods from the preference elicitation literature. 
limitations: I find that this paper provides a fully satisfactory discussion of its limitations, as well as of its potential negative societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper generalizes the HJB equation in the infinite time-horizon setting to arbitrary discounting schemes. Based on this HJB equation, a collocation-based deep learning algorithm is presented for reinforcement learning under hyperbolic discounting. Moreover, the authors study the problem of inferring the discount function from data, and demonstrate a deep learning algorithm for accomplishing this task. Lastly, the authors demonstrate the performance of both their RL and discount-function-learning algorithms in two simple domains.
strengths_and_weaknesses: # Strengths
It was immediately evident that the paper is written in great English, making the text very easy and comfortable to follow for the most part. The authors should be commended for this.

I appreciate the authors' ambition to study this problem in continuous time, which is often overlooked. The collocation algorithm, while fairly standard, is not particularly common in RL, and I believe the RL community should be interested by it. Especially given the time-inhomogeneity of the value function with arbitrary discounting, the collocation method seems quite elegant.

Moreover, I do believe the "inverse RL" problem studied in this paper -- that being the problem of inferring the discounting function -- is an interesting one. I can imagine the algorithm presented in this paper to be impactful in economics / neuroscience / psychology. 

# Weaknesses
There are a few mathematical issues and concerns, however they're all minor and surely they can be easily addressed.

I found that the discussion of experimental results is highly qualitative/speculative. Essentially, I don't find that the experimental results do a good job of convincing the reader that arbitrary discounting is valuable. It would have been nice, for example, to see an example where perhaps a nonstandard discounting scheme leads to a value function that is much more intuitively correct than that with an exponential discounting scheme.

Finally, I believe the novelty of the contributions is relatively limited. I do believe some of the contributions are valuable -- particularly the general discounting HJB equation -- however I believe the paper could have a lot more impact with a more developed experimental analysis.
questions: Comments and Questions
======================

  **Line 104**: "... with the interpretation that the reward becomes
  unavailable with hazard rate $\alpha(t)$" -- Should it not be the case
  that *all future rewards* become unavailable at that hazard rate? As
  written, it sounds like reward at time $t$ can become unavailable, but
  reward at time $t + dt$ can still be available.

  **Equation after Line 127**: I believe the dimensions are incorrect
  here. The term on the LHS as well as the first two terms on the RHS
  are scalar-valued, but the final term on the RHS (I believe) has
  dimensions $m\times m$ -- the Hessian has dimensions $m\times m$, and
  since $G(\mathbf{x},\mathbf{u})\in\mathbf{R}^m$,
  $GG^\top(\mathbf{x},\mathbf{u})$ should also have dimensions $m\times
  m$.  I believe there should be a trace operator on that final term.
  Also, it says
  $R:\mathcal{X}\times\mathcal{U}\times\mathbf{R}_0^+\to\mathbf{R}$, but
  in the equations we have the term $R(\mathbf{x}, u)$ -- an argument is
  missing (and I believe $u$ should be $\mathbf{u}$). Is the reward
  function time-homogeneous?

  **Theorem 1**: It would be helpful to have a refresher on the meaning of
  $\alpha_0$ in the statement of the theorem. Also, why does $R$ need to
  be bounded below in order for the integral to diverge? I suspect what
  was meant was

  > If $\alpha_0\leq 1$, the value function diverges
  > regardless of the boundedness of $R$.

  Also, I don't think it's correct to say that "the value function
  diverges". Divergence is a property of a sequence, not a function. I
  believe "the value function is not well-defined", like in the first
  part of the theorem, is better.

  **Line 275**: I suspect continuous action spaces can be supported
  without much trouble using a similar technique as given in [2], which
  only requires the assumption that the control signal is Lipschitz
  continuous.


Minor issues / grammatical errors
=================================

  **Line 59**: "Solutions approaches that solve the HJB equation directly,
  include..." -- that comma shouldn't be there.

  **Line 60**: When talking about path integral approaches, I believe "A
  Generalized Path Integral Control Approach to Reinforcement Learning"
  [1] deserves a mention.

  *Line 63*: Personally, I find this paragraph to be redundant given the
  introduction.


References
==========

  [1] Theodorou, Evangelos, Jonas Buchli, and Stefan Schaal. "A
  generalized path integral control approach to reinforcement learning."
  The Journal of Machine Learning Research 11 (2010): 3137-3181. 

 [2] Kim, Jeongho, Jaeuk Shin, and Insoon Yang. "Hamilton-Jacobi Deep
  Q-Learning for Deterministic Continuous-Time Systems with Lipschitz
  Continuous Controls." J. Mach. Learn. Res. 22 (2021): 206-1.

limitations: Limitations have been adequately addressed.
review_title: null

Meta-review: metareview: This paper studies various forms of discounting in continuous time, with a deep learning solution. One of the motivations for doing so is to broaden the range of inverse RL algorithms.

Overall, the reviewers appreciated the perspective taken in this paper and the proposed application of the idea in an IRL context. There was some discussion on whether actual human data experiments were necessary; I note that the experimental results are currently fairly preliminary. The use of the term "reinforcement learning" is also somewhat misleading given that this is closer to more traditional OR work, including the assumption that the MDP parameters are known. However, there was general agreement that this paper plays a useful role in bridging different fields and makes a good contribution. 

The authors are encouraged to give a more complete discussion of how this work relates to other techniques such as preference elicitation.
confidence: Less certain
Decision: Accept