Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - This paper studies the problem of optimizing CVaR-alpha of returns using the policy gradient method. 
- The paper identifies and tackles two deficits of existing PG approaches to CVaR optimization:
1. the "blindness to success" phenomenon, which clips returns above the alpha-quantile of the return distribution and causes the corresponding gradients to be uninformative w.r.t. high-return scenarios -- the approach suggested by the authors is to start with a higher-alpha (e.g. 1) to allow PG to learn the high return scenarios, then gradually decrease the risk tolerance to the desired level as training progresses
2. sample efficiency, which is addressed by a cross-entropy method that learns to weigh low-return experience more than high-return ones during training
- Finally, the paper provides theoretical and empirical results on three domains that argue in favour of these two strategies 
strengths_and_weaknesses: Strengths:
- the problems described pertaining to the optimization of CVaR seem important to address
- the application of CE and its corr. Proposition 1 is novel to my understanding, and Theorem 1 is an important and interesting result that has standalone value for the general research area
- the Algorithm choices are clearly motivated by and well connected to the theory  (Th. 1, Pr. 1, Lem. 2)

Weaknesses:
- a number of important algorithms discussed in the related work section are not compared against in the empirical evaluation: specifically, the paper by Keramati et al., 2020 appears to be similar in terms of the problem being tackled as well as their methodology
- while I think the CE approach is interesting and appreciate the authors' theoretical insight and strong motivation for tackling the problem, my biggest concern is that it is largely unclear to me whether DRL and related approaches are already solving (or could readily solve) the same problem, and the distinction w.r.t. current work is a bit blurred (see question 1 below as well). I am willing to revise my score if additional experiments/discussion in the paper disprove this claim. 
- the experiments are fairly low dimensional and it is difficult to determine if the effects of functional approximation errors could indirectly solve one of the problems this paper is tackling, on more challenging domains.

Organization:
- I think it would be helpful to the reader to provide a stylized example demonstrating empirically how CVaR optimization naively leads to the problems claimed in the intro. this might make it clear to the reader that the benefits seen in figure 1 do indeed arise from tackling the problems as claimed. 
- I would also suggest to consider moving the discussion of blindness of success before the algorithm presentation, and use it as principled explanation for why a larger alpha will be necessary. 
questions: 1. clearly, the authors assert some additional assumptions about the structure of the problem to allow the CE method to gravitate towards the high risk areas. in contrast, distributional RL does not make such assumptions, and it can reduce the model uncertainty through bootstraps (unlike PG which inherently has high variance). how does the proposed cross entropy approach differ from learning a distribution over return? what is the advantage and the drawback of making such assumptions compared to DRL? 
2. the cross entropy and "alpha-annealing" approaches seems to be largely orthogonal to one another. is there a stronger connection between them w.r.t. bias-variance trade-offs? 
3. in relation to the weakness point made above, does the existence of function approximation errors limit the effectiveness of the alpha reduction method?
4. In sect. 4.2., variance reduction is connected to sample efficiency by showing that reducing variance reduces one term in the general error bound [Xu et al., 2020] . however, viewed in the same lens, doesn't annealing alpha play a contradictory role by increasing the bias of the CVaR estimate J_\alpha, and hence increasing the first term in that bound? 
limitations: - limitations are briefly discussed in the conclusion of the paper, but it would be nice to discuss them in more details
- while this work is theoretical/abstract in nature, there could be some societal impacts in areas where RL is starting to become more practical (e.g. financial trading, health-care) - the implications of this with regards to the modeling assumptions (e.g. contextual MDP) used in this work should probably be mentioned
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors provide different contributions, all related to the CVaR policy gradient method:
1) They analyze the phoenomenon named "blindness to success", which affects the CVaR gradient optimization.
2) They propose soft risk scheduling as a heuristic way of circumventing the problem.
3) They extend the CE method for their particular setting, obtaining the CeSor algorithm, which consists in applying the two enhancements on top of the original GCVaR algorithm.
4) They analyse the theoretical advantages in employing the CE method in the exact case, showing that it allows to reduce the policy gradient variance, thus, to reduce the sample complexity of a variance reduced PG algorithm.
5) They compare the performance of the proposed method with the original version on some simple domains, moreover, they provide an ablation study in order to analyse the contribution of the different components of the algorithm.

strengths_and_weaknesses: # Strenghts
The has the following points of strength:
- The empirical analysis shows the advantages of employing both the proposed enhancements, and an ablation study clarifies that applying just one of them alone is not sufficient to reach the optimal policy.
- The problem of blindness to success is analysed in a formal and exhaustive way.
- The theoretical contribution about CEM is sound and it allows to clearly highlight the advantages of employing this sampling strategy.

# Weakesses
The article presents the following weaknesses:
- The soft risk approach, while intuitive and justified from an empirical viewpoint is not analyzed from a theoretical perspective, thus, it can be considered just as an heuristic to avoid the problem.
- The theoretical results provided for the CE method hold only in the exact case, i.e., when the quantile extimation has no error and the CE method allows to match exactly the desired distribution. 
- The role of the hyper-parameter $\beta$ is to guarantee a minimun number of samples in the CE update. However, this can introduce a bias in the distribution found by CE. The sensitivity of the approach w.r.t. to this hyper-parameter is not discussed.

## Minor
- Arguably, from the point of view of exposition, the article may benefit from a gradual introduction of the two main innovations propose the application of CE method and the risk-scheduling, instead of directly introducing the whole algorithm.
- Section 5.1 comment the performance of CeR, however, they are only showed in the appendix. 

questions: - Can the authors please discuss how sensitive the approach is to the hyper-parameter $\beta$?
- In particular, is the choice of $\beta$ important to guarantee convergence to the optimal solution? 
- If the approach is sensitive to this hyper-parameter, how should it be chosen?
limitations: As highlighted by the authors too, the main limitation is constituted by the necessity of having the possibility of conditioning sampling w.r.t. contexts. This requires to have a high-level control over the environments, an hypothes which can hold true in simulation, but which is usually more difficult to enforce in real-world environments.

Another limitation is constituted by the assumptions of Proposition 1, which are almost never satisfied in practical applications of the algorithm, as highlighted by the authors too.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies policy gradient under the conditional value at risk (CVaR) objective. The proposed method contains two important components: (1) using CEM for the better sampling of data so as to obtain a better estimate of the value at risk; and (2) using soft-risk scheduling on alpha (the CVaR parameter) to address the blindness to success problem. 

strengths_and_weaknesses: Strength: 

The paper proposed interesting heuristics to address the problem that CVaR is hard to optimize (especially in the RL setting). Algorithm 1 seems to be generic to any CVaR policy gradient method, which is ideal. The experiments have demonstrated the efficacy of the proposed methods.


Weakness: 

The problem of estimating the VaR and the problem of zero gradients for optimizing CVaR is not new in the risk-sensitive learning field. For example: In "Adaptive Sampling for Stochastic Risk-Averse Learning (Curi et al.)", the authors have considered using adaptive sampling of the data to overcome these issues (both the estimation of VaR and the zero-gradient problem).  In "On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk (Huang et al.)", the authors have showcased the zero-gradient problem of optimizing CVaR in a bandit setup (where the Markov CVaR is equivalent to CVaR). The proposed heuristics are interesting but it is unclear why certain design choices (e.g., the particular schedule of alpha') are made.
questions: - Figure 2 is a nice illustration. But could the authors provide how it is generated in detail? For example, how are R and C obtained for each method?
- Is there any additional reasoning for choosing alpha' the way presented in L13?
- This is just a small suggestion: For theorem 1, it might be more readable for the users to present it directly for B instead of negating the B. How should one interpret m_0? Does it depend on \alpha itself?
- For the experiments:
  - Why is GCVaR chosen as the PG method? 
  - How big of a role does clipping play? Have the authors compared the performance of GCVaR without clipping with different methods?
  - In the first experiment, the authors have done an ablation study and compared CESoR with SoR and CEM separately. What about the second and third experiments? How does  CESoR compare with SoR and CEM?

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors study risk-averse reinforcement learning with the specific problem of how to keep risk-averse while using the collected samples efficiently. To this end, a soft risk-level scheduling mechanism is proposed where all samples are used at the initial stage, and it gradually shifts to high-risk (lower return quantile) samples. Additionally, a cross entropy method (CEM) is used to sample risky trajectories among soft risk samples. The proposed method is evaluated on three benchmarks to show its effectiveness.
strengths_and_weaknesses: **Strengths**

The risk-averse property plays an essential role in reinforcement learning (RL), especially for real-world problems. Therefore, the research topic of this paper is interesting and important for the RL community. The proposed mechanisms for improving risk sample usage are effective on the evaluated benchmarks. The authors also give a theoretical analysis of the ignorance of high-return samples, which leads to a local optimum.

**Weaknesses**

The writing need to be polished carefully. Although the paper is well-motivated, the methodology is hard to follow. Specifically, it is hard to understand how soft risk-level scheduling works together with CEM since the optimization in Algorithm 1 line 11:12 is independent of policy learning line 13:15. Additionally, a lot of details of the methodology, e.g. CEM are introduced in Problem Formulation sections, which further prevent the understanding. For the novelty, it seems unclear. The main contributions seem to be a heuristic risk quantile hyper-parameter scheduling and a sampling-based previous CEM.


Minor:

1. Symbol 1_{R(tau)<q_alpha} is undefined in Eq. 3. It seems to be the indicator function.
2. Grammar error: Are —> is in line 133.

questions: 1. Some risk may caused by an insufficient exploration of the environment. What is the performance if using more advanced RL method, e.g. SAC instead of policy gradient?
2. How to understand the relationship of CEM and soft risk-level scheduling in Algorithm 1? It seems they are independent.

Update after the rebuttal.
It is clear now about the contributions and the relationship between CEM and soft risk-level scheduling.
limitations: No.
review_title: null

Meta-review: metareview: Overall, the reviewers were satisfied with the author response and overall recommend acceptance.  However, there were many discussion points and nuanced details that arose during post-rebuttal author-reviewer discussion.  Reviewers would like to see these discussion points, clarifications, and requests for revision addressed in the camera-ready.  To this last point, I specifically highlight the writing/illustrative example discussion that the authors had with reviewer QDcc.  I fully agree that refactoring a paper is challenging, but ultimately, the suggested modifications will improve the accessibility of the ideas and contributions in the paper.
confidence: Certain
Decision: Accept