Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an alternate approach for distributional DRL via proposing an objective inspired from Cumulative Prospect Theory (CPT, Tversky and Kahneman, 1992). They use this distributional objective in conjunction with policy gradient methodology to propose a distribution policy gradient method for risk-sensitive RL. Under their approach, the distribution of the returns is optimized to maximize some chosen function of its CDF. They experiment with different such possible distributional objectives (risk profiles) on the OpenAI Safety Gym environments and show that their approach performs better than PPO. 
main_review: ### Strengths: 

- The idea of using CPT in context to risk-sensitive RL seems promising. I also believe the approach is novel to the best of my knowledge. 

- The paper is written clearly and with sufficient empirical rigour in the experiments conducted.

### Concerns:

- I think the clarity of some parts can be improved. A major component of the work is Eq 4, where the second transformation is not clear. Is it a property of the CDF or is it derived from the approach of Prashanth et al (2016.)? I think this is an integral component of the work and it should be clarified in more detail. 

- The authors claim that the expression in Eqn 11 has a lower variance than expression in Eqn 6 but there is no accompanying proof. A formal explanation of this claim will help to strengthen the work.

- I also don’t understand why risk-sensitive DRL and constrained DRL approaches were excluded from the baselines. The comparison with PPO/TRPO demonstrates that the proposed approach can work better than the unconstrained methods. However, it is not clear what is the benefit of the proposed approach compared to the approaches in constrained RL literature. Some experiments with simple approaches like Lagrangian-PPO, CPO (which are both benchmarked and included with OpenAI gym) can help to make this distinction clear. Another possible method that can be included as a baseline is Cvar based objective (Chow et al 2015).



### Minor comments:

I think the notation for denoting CDF $P_{\theta}$ is a bit confusing as it gives the impression of parameterized transition model. I would encourage the authors to explore different notations for this quantity.

summary_of_the_review: I think the paper presents a really interesting approach to distributional DRL in context to risk-sensitive RL. Although there are some issues with the clarity, my biggest concern is that advantage of the proposed approach is not clear when compared to the existing literature on risk-sensitive and constrained RL. Although the paper could use more empirical strengthening, nevertheless, I think the will be useful to the community and hence recommend weak acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Risk objectives have long been investigated in reinforcement learning (RL). Most of the focus has been on classic risk measures, like exponential utility, value-at-risk (VaR), conditional value-at-risk (CVaR), leaving out, however, the cumulative prospect theory (CPT) developed by Tversky and Nobel Prize Kahneman in 1992, which has not yet been considered.

The advantage of CPT is to better model human decision-making, still allowing a wide class of risk measures, based on the utility $u$ and weighting $\omega$.

Hence, the authors consider a new risk-aware objective. Following some derivation, they compute a sample-based estimation of the gradient of this new objective w.r.t. the policy parameters.

The authors propose a PPO-like algorithm (called C3PO), which incorporates the new, risk-aware, gradient estimator.

They perform an empirical analysis on some tasks of "Safety Gym", showing that proper risk-awareness helps increase the performance of classic PPO.
main_review: STRENGTHS

The authors consider an interesting problem. I believe that risk awareness is important in RL. The utilization of a different risk measure, other than the one already explored, can be beneficial to obtain better risk awareness of the learning agent. 

WEAKNESSES.

The paper is difficult to follow. Mathematical notation lacks precision (for example it is unclear whether the authors consider a finite horizon, or an episodic or a continuing problem in Equation 1). Most of all, concepts related to risk measures should be better explained. When one looks at the "CPT objective" at the beginning of page 3, it is unclear which roles play the utility function and the weighting. I do understand that probably the authors are very familiar with the underlying theory, but most people in our field (Rl) might be not. Would be nice to give an intuition.
Further, many passages seem unnecessary, or they are poorly explained. 

Example: why do the authors bother giving us two different forms of $\nabla_\theta P_\theta$ in Equations 4 and 5? And why the same happen in 7? Is there any utility in showing these two different versions of the same quantity?

Also, it is not clear to me whether Equation 6 is an unbiased estimator of Equation 3. Can the authors give an intuitive explanation of why $\omega'(i/N)$, and can they show the unbiasedness of such estimator? 

Section 3.3 starts with "unfortunately ... (6) suffers from high variance". Is this due to the log-likelihood derivative (as in classic REINFORCE) or there is an additional motivation caused by the new objective?

Derivation in (9) is known by most of the people in our field. The authors can just cite classic works on baseline subtraction. 
Further, starting from Equation 9, and in many equations that follow, $\mathrm{d}$ is missing in the integrals. 

I am not convinced that 

$$
\left(\omega '\left(\frac{i}{n}\right) + \omega '\left(\frac{i-1}{n}\right)\right)
$$

can be pulled out of the expectation $\mathbb{E_{\tau_i}}$. Can the authors clarify the passages in 13?

Why the approach in (14) should minimize the variance? While subtracting the value function $V$ (or utility function, here), is a standard technique, remains unclear whether with the influence of 

$$
\left(\omega'\left(\frac{i}{n}\right) + \omega'\left(\frac{i-1}{n}\right)\right)
$$

still makes $V$ a proper baseline subtraction.

Further, the usage of the utility function $u$ is unclear. In Equation 3, the utility takes as input a scalar function (the reward), i.e., $u(r(\tau))$. It is unclear how the per-step utility $u(\mathbf{s}, \mathbf{a})$ in Equation 14 is defined, and it is also unclear how $u(\mathbf{s})$ in Equation 15 is defined. 

Can the authors explain the usage of the euclidean norm in Equation 15? The value function and the utility are scalars, why do we need that? Did the author mean to write something like

$$
\mathcal{L}(\phi) = \sum_{i,t} \mathbb{E}\left[V_\phi(\mathbf{s}_{i, t}) - \sum_{t'=t}^{T_i} u(\mathbf{s}_{i,t'})\right]?
$$

It would be nice to see the derivation of the value function and the advantage in this new setting of the risk measures, for example introducing a risk-aware Bellman equation. It is otherwise difficult to understand why the advantage "is computed with the standard GAE except with per-step utilities in places of rewards". The correctness of this approach might be trivial for the authors, but not for me. 

In Section 4, could the authors clarify what ar the robots "Point" and "Car", and what are the tasks "Goal", "Push" and "Button"? It would be nice to know what are these tasks about, and the dimensionality of action and state spaces.

One thing that I think was really not clear, is what should I expect from the experiments. What should be the idea of using the CPT objective instead of other measures? How this should help?

I have seen that making the agent risk-aware can lead to better performances, as highlighted in your experiments, but shouldn't this effect appear also with other risk measures (like VaR or CVaR)? Could the author clarify this point? I think that a comparison with other risk measures would have been beneficial. 

Further, uncertainty comes both as an aleatoric component (stochasticity of the environment) and epistemic (trust in my own model). Looks like that this work is focused on the aleatoric component (which is fine). Do the authors plan to extend their work also to the epistemic uncertainty?


summary_of_the_review: STRENGHT:

The authors propose a new policy gradient algorithm based on a risk measure that has not been considered in RL so far. The considered risk-sensitivity measure comes from the cumulative prospect theory and aims both to mimic human decision making, and to generalizes a large class of other risk measures.

I believe that risk-aware RL is important, as intelligent beings rarely optimize for the "average" case, but they often act optimistically or pessimistically, usually giving higher weights to rare events.

WEAKNESSES:

The paper lacks a proper background, clarity, and precision. The mathematical notation should be improved, and while some passages could be skipped, since they are widely known by most of the audience, some passages instead remain obscure and need further clarification. I think that some passages are wrong, and I am waiting for the authors' response.

The main objective of the paper remains unclear. What is the benefit of utilizing this method in RL? Why are other risk measures (like VaR or CVaR) not enough? 

The authors did not compare their risk measures with others, which I think was instead necessary. 

UPDATE
---------

As the author improved the clarity of their submission and clarified some of my doubts (for example explaining the lack of comparison with VaR & CVaR), I am raising the score from 3 to 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers a generalization of the policy gradient method to optimize for arbitrary utility functions with weightings that depend on the entire CDF (rather than the expected reward). This generalization has two aspects: (1) a utility function on top of the trajectory reward and (2) a weighting function for the CDF of the trajectory reward with respect to which the expectation is performed.  The paper derives an expression for the policy gradient and also generalizes the standard variance reduction baselines. Inspired by the PPO loss, the authors then propose a clipped version of the policy gradient calling it C3PO and evaluate this on some benchmarks from the OpenAI Safety Gym, where it is found that the conservative weightings can offer improvements over the standard formulation.
main_review: Strengths: The generalization to consider the entire CDF with arbitrary weightings appears to be a novel contribution to the literature. Furthermore, the authors also propose a simple to instantiate procedure for these weightings and a sample based algorithms that just sorts the trajectory outcomes. 

Other comments:
* How robust are these results to the more common deep RL benchmarks outside of the safety gym? 
* Once the utility is assumed to additively decompose across each state, action is this not equivalent to simply defining the utility as the reward?
* The variance reduction is basically the same as the usual policy gradient (and follow directly from the fact that the expectation of the gradient of log prob is 0).
* In Algorithm 1, where is $\hat{u}$ coming in? Is this just the sum over t of $u(s_{i,t})$? There seems to be some confusion over the dependence on both state and action or only state for the utility -- same issue between (14) and (15).
* A $\frac{1}{N}$ factor appears to be inconsistent across Eqs (11), (12) and (14)
* It's not clear how important the particular choice of -0.025 for the negative reward is for the results or how it was chosen. In particular, this seems like it would have a complex relationship with the $\eta$ weighting. 
* Figure 5 captions says 6 environments, but there is only three in the figure.
summary_of_the_review: The main technical contribution is a novel generalization of the policy gradient expression to consider CDF weightings and a sample based algorithm that weights sorted trajectories to compute the gradient. The variance reduction extensions are fairly trivial. The empirical evaluations are limited to the open AI safety gym and are conducted on a PPO inspired version of the policy gradient result. These look promising, but the role of an arbitrarily chosen reward modification makes it hard to interpret how much the results depend on this  and how robust they are.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The article propose a policy gradient method for optimizing a CDF based criterion, inspired by CPT. 
By varying the weighting function inside the objective, it is possible to change the risk-aversion of the agent.
The authors derives the policy gradient for the aforementioned objective and propose an estimation technique for it.
Then, they propose an algorithm which extends PPO for optimizing their objective. Empirical analysis is carried on to evaluate the approach on some modified Safety Gym environments, in which a fixed negative rewards corresponded to adverse events.
The authors evaluate different objectives obtained by employing a Wang weighting function, with different values of the parameter $\eta$. They show that optimizing a cautious (or risk-averse) objective allow to obtain better results in terms of average reward w.r.t. optimizing an aggressive (or risk-seeking) one. By further exploring the parameter space, the author demonstrate that some risk-averse values of the parameters allow to outperform also the risk-neutral version of PPO w.r.t. the average reward objective.
main_review: ## Strenghts
The paper aims at solving an interesting problem, which is the optimization of a CPT inspired objective with a policy search approach. The authors derive a policy gradient for their objective and propose and estimator for it. They derive a PPO-like algorithm, which is then employed in the experiments.
The experimental analysis seems to be fair and sound: the experimental setting is described in a clear way, the experiments are repeated to take into account for the randomness in the weight initialization and the stochasticity of the performance is correctly represented in the plot with shaded areas, and by showing quantiles in the tables. The author included also the distribution of the returns, which helps understanding the impact of the risk-averse optimization.


## Weaknesses
While the author correctly described the state-of-the-art in the fields, some connections with related works are still not completely clear. For instance, it is not clear how much the proposed objective differs from distortion risk-measures (Wang, (1996), Dabney, (2018)). Furthermore, the author says that their estimation approach is similar to the one used in (LA Prashant, 2016), but it is not clear to which extent the developed techniques are different from the ones proposed there. It would help the reader to explicitly describe these connections with prior works, in order to ease the understanding of the authors contribution.
From a methodological viewpoint, the policy gradient derivation lacks a clear separation between the exact gradient formula and its estimator. Moreover, the authors do not show that expression (6) represents an unbiased estimate of the gradient, as they claim. Results in (LA Prashant, 2016) seems to suggest that the estimate would be instead only consistent, hence, asymptotically unbiased.
For what concerns the experimental analysis, the authors focus mainly on evaluating their results under the average reward objective. This is not fair in principle, since the proposed approach is actually optimizing a different objective. It would be interesting also to see how the algorithm performs w.r.t. the objective it is optimizing, including its learning curves. Finally authors suggest that the positive effect due to risk-averse optimization is similar to what happens with prioritized replay. However, while the latter technique gives an in higher weight to errors in value function estimation (high TD-errors), risk-averse optimization gives an higher weight to actual lower returns, thus, their objectives are substantally different.
summary_of_the_review: While the paper analyses an interesting problem and its experimental analysis is sound, it is not clear how novel its derivation is. Moreover, the author did not provide a clear picture on the policy gradient method they propose, for which further proofs should be included. The main claim of the authors, that some degree of risk-aversion may help risk-neutral optimization, is not novel from an empirical point of view, but it is still largely unexplained from the theoretical one, for which, unfortunately, no contribution is given in this work.
Due to a balance between the aforemetioned weaknesses and strenghts I suggest a weak reject.
review_title: null

Meta-review: This paper introduces a new approach for risk sensitive RL by using an objective that depends on the full distribution and can apply a weight to the resulting trajectory. The reviewers thought that focusing on more general and expressive objectives for RL is well motivated. However, they had a number of concerns of the current paper state, including its clarity in a number of sections and its relation to other work in risk-sensitive RL. The authors provided thoughtful responses but some concerns lingered around the prior concerns.
Decision: Reject