Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers learning a risk-sensitive policy based on CVaR via distributional RL. A modified distributional optimal operator is proposed with fixed-point satisfying CVaR optimality and an algorithm is proposed which combines QRDQN and the proposed operator. Experiments are executed on both synthetic and real-world data to present the performance of the proposed algorithm.
strengths_and_weaknesses: Strengths: 
This paper is well written and easy to follow. It indicates that the standard distributional Bellman operator does not work for the CVaR optimal policy and makes a proper modification. 

Weakness: 
The theoretical results are not solid enough to guarantee the convergence of the algorithm. 
questions: 1. Although the operator $\tilde{\mathcal{T}}_{\psi}$ defined in (10) may not result in convergence of distribution, I was wondering if $C_{\alpha}(\tilde{\mathcal{T}}_{\psi} U)$ converges to $C_{\alpha}(Z^{\pi^*})$ for all $U$. 
2. The authors compare their proposed algorithm, which is based on QRDQN (Dabney et al. 2018b) with IQN(Dabney et al. 2018a). I think it would be more fair to use the same baseline structure to approximate the distribution function, either IQN or QRDQN.
3. The experiments on Atari games show that the proposed algorithm achieves a more concentrated distribution. Is there any benefit or effect with the concentration?
limitations: This paper requires that the rewards are drawn from a countable set and this is not a common assumption.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a new method for robust reinforcement learning that optimizes the dynamic and static CVaR. First the paper proves that the current action selection rule in distributional RL does not converge to the CVaR objective. The paper presents a new method that they show learns to optimiz static and dynamic CVaR. Experiments show that the proposed approach does better at the task of option trading than risk neutral and markov action selection startegies
strengths_and_weaknesses: + Interesting and relevant problem
+ Interesting theoretical and experimental results
- It is difficult to see how the method compares or positions itself to other papers in the robust RL literature
- The evaluation seem to be missing some relevant baselines as the paper points out itself that there is related work that can tackle static and dynamci CVaR
questions: - What is meant by distributional RL does not converge to static or dynamic RL 
- What about other literature on robust policy optimization? The paper can compare their method to these existing approaches 
    -  Risk-constrained reinforcement learning with percentile risk criteria
    - Risk-sensitive and robust decision-making: a CVaR optimization approach
    - Tractable objectives for robust policy optimization

limitations: I feel the paper currently lacks a good contrast to existing methods for robust RL, principally and experimentally.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors study the problem of learning risk-sensitive policies in the distributional RL setting. They adopt Conditional Value at Risk (CVaR) as the risk measure throughout the paper. They begin with a problem overview and discuss the differences between static and dynamic CVaR problem formulations. After introducing the requisite background on distributional RL, the authors propose a naïve method of learning a risk-sensitive policy, called the Markov action-selection strategy, where the action selection procedure aims to maximize the $\alpha$-CVaR over return distributions at a given state. It is then shown that a distributional Bellman operator employing the Markov action-selction strategy does not converge to a dynamic or static CVaR optimal policy, even if the optimal CVaR policy is stationary and Markov. 

With this result, the authors’ aim is thus to determine if a static CVaR policy may be properly optimized in the distributional RL framework. They begin by constructing an augmented MDP from the original, with the addition of a thresholding parameter. This parameter is initially set equal to the $\alpha$-quantile of the initial return distribution and is repeatedly updating by subtracting off observed rewards, thereby effectively keeping a history of the “return-to-go” to reach the initial threshold. The introduction of this new parameter poses a new problem: it is no longer sufficient to retain a single value distribution for the original MDPs state-action pairs, since the value distributions in the augmented MDP depend on the threshold parameter, which may be observed to be many different values for any given state-action pair. To navigate this problem, the authors assume the existence of a CVaR optimal augmented policy that is stationary and Markov in the original MDP, and show that the stationarity and Markov properties allow for one to recover an optimal CVaR policy in the original MDP. They go on to show that given the optimal original policy’s value distribution, Algorithm 1 executes the optimal original policy by keeping track of the thresholding parameter’s history. It is shown thereafter that the action selection procedure in Algorithm 1 may be used in a distributional Bellman operator, and that this new operator has a fixed point that is equal to the return distribution associated with the optimal CVaR policy from the original un-augmented MDP. The authors note here that it is still an open question whether the new Bellman operator is a contractive operator on the entire space of return distributions i.e. that starting from any return distribution, the repeated application of the operator would result in the eventual fixed point that is equal to the return distribution of the CVaR optimal policy. 

The authors then present an algorithm that optimizes a neural network policy using the newly proposed distributional Bellman operator. Experiments are presented showing that the proposed algorithm outperforms the Markov action-selection and risk neutral policies in a synthetic data experiment, an options trading experiment, and in Atari games.

strengths_and_weaknesses: Strengths:

-The paper introduces novel and interesting theoretical results that advance the theory of CVaR distributional RL

-The research presented in the paper lays groundwork for important future research, e.g. proving the contractive property of the proposed Bellman operator.

-The paper does a very good job of clearly stating the need, contribution, and relevance of the work. This is reinforced by the detailed background provided in Section 2


Weaknesses:

-The paper ends abruptly and has no concluding remarks. 

-The discussion and analysis of the empirical results is limited. There is very little explanation offered that would allow the reader to construct an intuitive reasoning for the obtained results.

questions: N/A
limitations: -The paper does discuss the theoretical limitations of the work, namely the absence of a proof of contractivity of the proposed Bellman operator.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors tackle the challenge of learning a risk averse policy (CVaR) in a distributional RL setting.
While prior works have presented a naive method for optimizing the CVaR, this is shown (both theoretically and practically) to not optimize the correct objective.
In this work, they present a theoretically sound method for iteratively learning a CVaR maximizing policy and back these claims with experiments.
strengths_and_weaknesses: Strengths:

- Present a theoretically backed 1-step iterative update scheme for learning a CVaR maximizing policy in a distributional RL setting.
- Present a comparison and analysis of prior methods showing that naive application of the 1-step CVaR does not solve the objective.
- Experiments on synthetic data + option trading + ATARI.

Weakness:

- While this work performs analysis on atari, this analysis is very limited. While I appreciate the simpler experiments that enable a more theoretical analysis of the method, complex domains enable a more convincing empirical justification.
I'd like to see more experiments in such domains. If compute is an issue, the authors can always resort to tasks such as continuous control / minatari / cartpole / and other simpler tasks.
- Paper lacks a conclusion and future work section. While there are no rules that require such structure, I do feel that this is warranted and well accepted in the community. I'd propose to move some proofs to the appendix in order to allocate room for discussion.
questions: 1) The comparison to Dabney et. al. 2018 (a) seems slightly misleading.
The goal of a risk-sensitive method is to maximize the lower-tail of the returns. Dabney et. al. 2018 (a) might not optimize the CVaR properly, but their results do show competitive performance in ATARI domains (often outperforming mean optimization).

In these experiments, we see that Markov selection is under-performing compared to risk-neutral.

What happens in simpler domains / when the algorithm is run using the same parameters as Dabney et. al. 2018 (a) and for a longer period of time (commonly 200m steps)?

2) What happens in other domains? does this method still work or is it limited to Asterix?
limitations: -
review_title: null

Meta-review: metareview: This paper proposes a new action selection approach for risk-averse distributional reinforcement learning optimizing CVaR. It first shows that the action selection schemes used in existing approaches do not converge to the desired policies and subsequently shows that the fixed-point of the Bellman operator with the new action selection scheme is the desired optimal CVaR policy as long as it is stationary. It finally provides empirical results showcasing the benefits of the proposed approach.

The reviewers had mixed initial views on this paper. On the positive side, they found the paper to be well written and appreciated the new insights into the convergence of the existing action selection scheme as well as the more principled proposed scheme. On the negative side, there were concerns that (1) the paper does not actually convergence of the algorithm, only a fixed point, (2) that the paper does not provide sufficient discussion of the implications of the presented results, e.g. in the form of a conclusion and (3) that a comparison to CVaR optimization approaches that are not based on distributional RL is missing. 
The authors' response addressed serval of these concerns so that all reviewers view this paper positively. Although, this paper still remains borderline and some concerns remain, the AC concurs with the reviewers that this paper has sufficient merits to be accepted, hence a recommendation for acceptance.
confidence: Less certain
Decision: Accept