Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an algorithm and regret analysis of risk sensitive reinforcement learning. Their regret guarantee is significantly tighter than prior work; in particular, it helps close the gap compared to the lower bound.
main_review: Strengths
- Important problem
- Significantly tighter regret analysis compared to prior work

Weaknesses
- Unclear connection to distributional reinforcement learning
- Unclear exactly what novelty is being claimed in terms of connection to entropic risk measure

Overall, I think this paper is studying an interesting and important problem, and includes several new insights that allow them to improve upon prior work.

My main concern with this paper involves some significant issues with its exposition, in particular, regarding the claimed connection to distribution reinforcement learning. First, the connection to distributional reinforcement learning is itself unclear. As far as I understand, the connection is simply that the exponential value function is the moment generating function of the distributional value function. However, this connection seems to be completely irrelevant to the remainder of the paper. They also do not make this connection very precise, and it is also unclear to me whether this connection is novel.

The authors also claim in their contributions that the connection is in terms of the “entropic risk measure”; I didn’t understand what the authors meant here.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work a theoretical analysis is conducted in the context of risk-sensitive reinforcement learning with entropic risk measures. This allows connecting distributional RL and risk-sensitive RL. Two algorithms are proposed with a novel exploration scheme based on two bonuses. Regrets bounds are provided that improve previous analyses by an exponential term.

main_review: Originality:

The paper relies heavily on  a previous paper [16]. However, the authors improve [16] by a simple procedure re-formulating a recursion using an exponential form. The result of this re-formulation gives insight on how to obtain a tighter bound which translate in very simple modifications to existent algorithms (RSVI and RSQ from [16]). Although the change is small it has considerable implications in the regret bound.

I also think the literature is fairly represented, specially the section commenting on the paper [16] since it is the closest one to this work.

Quality: 

- I believe the work is technically sound. 

- I am unsure about the connection to distributional RL claimed by the authors. Maybe they could elaborate more on that? As far as I can see Equation 4 is just a simple re-arrangement of Equation 8 which is very well known in the community (see N Tishby, PA Ortega, J Grau-Moya, R Fox, D Polani works to name a few) and also derived in [16]. How exactly this connects to distributional RL?

- It would be interesting to see if the authors can comment how their work could be extended to infinite horizon settings or to the online setting (non-episodic). Since the algorithm requires revisiting states at time "h" in order to have good estimates of Equation 5 I wonder if the presented work breaks down in this case.

- Shouldn't Line 8 of Algorithm 1 contain a log e.g. 1/beta log G? Hand-wavy explanation: Since V=max Q and exp(beta Q) = G--> V = max 1/beta log G?

- Actually looking again at this line, the authors could comment what happens when there is only 1 sample to estimate w? In that case one would obtain a risk-neutral estimate of V right?

Clarity: 

I want to thank the authors for producing such a well explained work. It is well organized, the explanations of the problems in [16] were good, and the method the authors used to solve it was also very well explained.

Significance: 

I think since the bound improvements are fairly easy to re-construct, this work might be a reference  that future researchers might use. 


limitations_and_societal_impact: I think the authors could have done a better job stating the limitations of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers risk-sensive decision-making in reinforcement learning as modeled by an exponential utility function. In contrast to a previous work by Fei et al. (2016), the authors propose to consider the exponential Bellman equations, which allows to provide a smaller regret upperbound of their proposed algorithms.
main_review: The paper is generally well-written and the contributions (Algorithms with doubly decaying bonus and theoretical analysis that reduces the gap between the upperbound and lowerbound) are clearly presented. However, I think the exposition would be clearer if Sec.3.3 happens before Sec.3.2 because the exponential Bellman equation makes sense only after the corresponding value function has been introduced.

Regarding the related work about distributional RL, the authors may want to also cite the following paper:
D. Yang, L. Zhao, Z. Lin, T. Qin, J. Bian, and T. Liu, “Fully Parameterized Quantile Function for Distributional Reinforcement Learning,” NeurIPS 2019

Typos:
- line 96: large -> larger
- (2): Z_{H+1} = 0 should be added
- (3): Z_{h+1} -> Z
- line 217: differerent
- Algorithm 1 should also include the case when \beta < 0 in lines 8 an 11
- line 286: aveage
- line 288: miminial 
- line 306: Algorithm -> Algorithms
limitations_and_societal_impact: No suggestion is needed since the work is more fundamental.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the exponential Bellman equation that arises in risk sensitive reinforcement learning when using entropic risk measure as the risk measure. It claims to "offer a route for connecting distributional RL and risk sensitive RL", although this question is never seriously addressed. The main result of their analysis consists in design and analyse two novel RL algorithms that employ a novel exploration mechanism. The regret bounds on this mechanism improves by an exponential factor.
main_review: 
Generally, the paper is well written and tackles an interesting problem. I consider the main contribution to be the said algorithms that exploit a new exploration strategy and its analysis in terms of regret, which leads to a regret bound that is O(exp(H)) instead of O(exp(H^2)) for previously proposed algorithm for this problem. From a theoretical point of view, this seems to be a relatively important contribution. On the other hand, I find that the claimed second contribution which relates to distributional RL is rather under-developed and trivial in its current state. The paper also might also lack a concrete application where one can confirm that the theoretical performance improvement translates in actual speed-ups in convergence rate during training.

Regarding the connection with distributional RL, the authors claim in the abstract and introduction that the paper answers to the need of identifying an algorithm that can optimize non-heuristically risk-sensitive policies in a distributional RL framework. Yet, the only support to this claim comes in the form of an observation that the use of a recursive entropic risk measure in distributional RL might motivate encoding the distribution of the "total cost to go" through its moment generating function. Implications of this observation are completely missing from the paper while the details of a rigorous algorithmic method to do risk-sensitive distributional RL are undisclosed.


Other comments:

Why focus on finite horizon MDPs? This assumption significantly restricts the scope of the paper.

Equation (4) is misleading as it seems to indicate that the same Q_h function needs to satisfy the equality constraint for all values of \mu. In general, I suspect that this is impossible to satisfy and that the authors meant that given any fixed \mu, one could impose the condition described in equation (4). In particular, in a distributional RL setting, my understanding is that the Q_pi function would need to have four input parameters: h, s, a, \mu.

limitations_and_societal_impact: I do not see anything special to report here.
review_title: null

Meta-review: Reviewers believe that the improved regret bound in this paper makes a significant contribution and that the paper should be considered for acceptance on the basis of that contribution alone.  

However, the one sticking point during reviewer discussion concerned the distributional RL connections in the paper.  More precisely, a long nested discussion starting with nUcE's initial review concerns about the mostly undeveloped link to Distributional RL (also reflected in other reviews) and continued towards the end of the discussion chain with reviewer tvfS.  The authors presented a detailed mathematical presentation in their final nested comment that convinced reviewer tvfS of the connection being claimed.  

While ultimately the reviewer concerns were addressed with this final author response, it is critically important that this mathematical presentation and discussion be included on revision, perhaps in an Appendix.  Furthermore, while reviewer tvfS is satisfied with the author response, reviewer nUcE would still prefer that a revised paper focus on the regret bound contributions and focus on the distributional RL discussion as a "a side remark that opens to future interesting research directions".  The authors are asked to carefully consider all of these comments as they prepare their final version.
Decision: Accept (Poster)