Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers finite episodic Markov decision processes aiming with risk-sensitivity. Two properties are identified to develop both the model-based and model-free algorithms for risk-sensitive distributional dynamic programming.  Both upper and lower bounds for regret haven been proved.
strengths_and_weaknesses: Strengths:
1. A specific entropic measure is used, enable the derivations of useful distributional Bellman optimality equation as well as risk-sensitive distributional dynamic programming. This contribution is very interesting. 

2. OFU principle has been used to develop deep RL algorithms with provable regret bounds. 

3. An improved lower bound has been obtained when beta is positive. 

4. The paper is well written. The presentation is clear.


Weakness:
1. This paper focuses on the episodic MDP setting. The most common setting in the risk-sensitive control literature is the setting with continuous state/action spaces (see Jacobson 1973 or Whittle 1990). Then stability also becomes an important issue. It is unclear how useful the results in this paper will be for continuous control applications. Maybe rephrase and focus on "risk-sensitive MDP"?

2. This paper proposes two algorithms with provable regrets. However, it is unclear whether the proposed algorithms will really perform well on any MDP tasks. Numerical examples are needed in this case. This is different from analyzing existing algorithms which already perform well.

3. CVaR vs Entropic Risk Measure (ERM):  CVaR has been used a lot in deep RL tasks and seems to be a better risk measure than ERM for deep RL. One reason why ERM was popular in some early risk-sensitive control literature is that for linear dynamics, ERM leads to a dynamical programming solution with relatively simple analytical form. In the deep RL setting, it is unclear whether ERM has any practical advantages over CVaR.

4. The regret analysis technique is not that new (similar analysis has been used to study episodic MDPs (without ERM) before).




questions: How does the performance of the proposed algorithms compare to that of DRL algorithms using CVaR? Is it possible to extend the proposed algorithms for continuous control tasks, such as Cartpole or more complicated robotic learning application? Can the authors clarify the unique novelty of the regret analysis technique?


limitations: Yes, the authors have discussed the limitations of their theory. However, I mean some study on the actual performance is necessary for this paper. The proposed regret guarantees are meaningful only when the algorithms actually do perform well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is about distributional reinforcement learning (DRL), which aims at learning optimal policies (in terms of total payoff) in a Markov decision process. While the classical reinforcement learning (RL) paradigm is based on the expected total return, in DRL one is interested about the whole probability distribution of this total return. 

In this work, the idea is to leverage this distributional information for risk-sensitivity purpose.
The focus is on the entropic risk-measure (ERM) of the total reward in an episodic MDP. Properties of ERM are given (additivity and monotonicity preserving) leading to an optimality principle akin to the classical one based on expected value.
Two algorithms (one model-free and the other model-based) called RODI are proposed: they both rely on the OFU principle applied to ERM.
While most risk-aware DRL papers have no regret analysis, here both upper and lower regret bounds are provided.
strengths_and_weaknesses: The paper is well-written and motivated by risk-sensitive applications. The DRL perspective makes a lot of sense to tackle risk-aware problems. The main strength of the paper is that its algorithmic RODI procedures are naturally derived from DRL applied to ERM and combined with the OFU principle.

The main weakness is the lack of numerical experiments, which is balanced by the theoretical nature of the paper.
questions: (Minor) questions/comments:

- line 80: more accurate to say that P_h maps state-action pair to probability distribution over S, i.e. P_h : SxA -> Delta(S)
- line 108: replace Z(s) by Y(s)
- footnote page 4: strange formulation? maybe rather say: "The algorithms for random reward enjoy regret bounds of the same order"
- notational issue: at line 176, reference to "c_h^k" but in Algorithm 1 it is just written "c_h". Same holds for eta_h. Is it normal or a notation issue?
- what is f_h in Proposition 2?
limitations: The scope of the paper is limited to the ERM criterion, but it is a reasonable "limitation" in order to design a risk-sensitive DRL approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper deals with a risk-sensitive reinforcement learning problem where the risk functional is the entropic risk. The authors provide both the regret upper bounds for a proposed model-based algorithm and a model-free algorithm and a tighter regret lower bound compared to prior works. 

strengths_and_weaknesses: Strength: The ideas in the paper are clearly explained. A tighter lower bound is presented in the paper. 

Weakness: Given that prior works, e.g., [22, 23], have also used optimism to solve the exact same problem and achieve similar regret guarantees, it is unclear how much novelty/significance the newly proposed algorithms are. 

questions: - The authors suggest that the lower bound proof in [22] has a mistake. Can the authors elaborate more on this?
- Can the authors compare their proposed algorithms with that proposed in [22] in a more detailed way to explain how optimism may have been used differently in this paper and what that implies?


----

Stylistic suggestions:

- Using ERM to refer to entropic risk measure may be confusing since ERM is known to be Empirical Risk Minimization in standard ML literature.
- \citep should be used for the first two citations in L101.
- In L168, (s,a)s may be confusing to the reader.

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes to build a distributional dynamic programming framework to optimize the entropic risk measure of the return. Under the new framework, both model-based and model-free algorithms are proposed, which achieve regret bound that matches one of the existing works in risk sensitive control. The lower bound of rish sensitive control problem is also improved using the new framework.
strengths_and_weaknesses: Strengths: 

1. The proposed distributional dynamics programming framework is novel to my best knowledge. Differences in the analysis between distributional RL and previous work [22] are discussed, which shows that the distributional analysis can avoid the explode factor and shows the significance of the proposed framework in theory.

2. The presentation of the paper is clear. Key steps in the analysis are highlighted, with detailed discussion on how they would affect the final regret bound. Algorithm designs are also explained in detail.

Weaknesses: 

1. Since this work is highly relevant to [22], the significance of the work can benefit from a more comprehensive comparison  w.r.t. to previous work [22]; See Questions 1 and 2. Also, further discussion on another relevant work [1] can also improve the significance. 

2. The novelty of the proposed algorithms can benefit from further discussion. The equivalence of the proposed model-based algorithm to ROVI is discussed, which raises a natural question: if distributional RL is somewhat equivalent to the existing algorithms, existing analysis could also apply to DRL, then is the claim in line 10 "this is the first regret analysis of DRL" still valid?

[22] Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
questions: 1. In Appendix A of [22], they also provide connections between risk-sensitive RL and distributional RL through the exponential Bellman equation. How is the proposed framework related to theirs? It would be helpful if there's a section dedicated to explain the relation and potential difference w.r.t. their analysis.

2. If Alg. 2 is actually equivalent to Alg. 3, what is the benefit of distributional framework over existing risk sensitive control approaches? E.g., what are the cases when the proposed framework is more favourable, either theoretically or empirically? 

3. Does the improvement in the lower bound in Section 5.3 come from the newly proposed framework, or just different proof techniques? 
limitations: The limitation of the proposed method are discussed, but not adequately.
review_title: null

Meta-review: metareview: While this work provides interesting insights on  distributional reinforcement learning for risk-sensitive control, it is unclear how beneficial these results are given the closely related work [22] (on the same problem with the same regret bound). The authors mentioned in the rebuttal that their algorithm may motivate the design of similar algorithms for other risk measures, but no concrete discussions and examples were provided. We believe that the paper would benefit from another round of revision to properly address these issues and make its contributions move convincing.
confidence: Certain
Decision: Reject