A Risk-Sensitive Policy Gradient Method
Authors: Jared Markowitz
Ryan Gardner
Ashley Llorens
Raman Arora
I-Jeng Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: deep reinforcement learning
policy gradient
risk-sensitive
ai safety

Abstract: Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also wastes an opportunity for the agent to modulate behavior based on distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach, whereby the distribution of full-episode outcomes is optimized to maximize a chosen function of its cumulative distribution function (CDF). This technique allows for outcomes to be weighed based on relative quality, does not require modification of the reward function to modulate agent behavior, and may be used for both continuous and discrete action spaces. We show how to achieve an unbiased estimate of the policy gradient for a broad class of CDF-based objectives via sampling, subsequently incorporating variance reduction measures to facilitate effective on-policy learning. We use the resulting approach to train agents with different “risk profiles” in penalty-based formulations of six OpenAI Safety Gym environments, finding that moderate emphasis on improvement in training scenarios where the agent performs poorly generally improves agent behavior. We interpret and explore this observation, which leads to improved performance over the widely-used Proximal Policy Optimization algorithm in all environments tested.

Under review as a conference paper at ICLR 2022
A RISK-SENSITIVE POLICY GRADIENT METHOD
Anonymous authors
Paper under double-blind review
ABSTRACT
Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating policy. This differs from human decision decision-making where gains and losses are valued differently and outlying outcomes are given increased consideration. It also wastes oppor- tunity for the agent to modulate behavior based on distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose more direct approach. whereby the distribution of full-episode outcomes is optimized to maximize chosen function of its cumulative distribution func- tion (CDF). This technique allows for outcomes to weighed based on relative quality, does not require modification of the reward function to modulate agent behavior, and may be used for both continuous and discrete action spaces. We show how to achieve asymptotically consistent estimate of the policy gradient for a broad class of CDF-based objectives via sampling, subsequently incorporating variance reduction measures to facilitate effective on-policy learning. We use the resulting algorithm to train agents with different "risk profiles" in penalty-based formulations of six OpenAl Safety Gym environments, observing that moderate emphasis on improvement in training scenarios where the agent performs poorly both increases the accumulation of positive rewards and decreases the frequency of incurred penalties. We found that, in all environments tested, the same risk profile can be used to produce both stronger overall performance than standard Proximal Policy Optimization (PPO) and higher levels of positive reward than PPO constrained by Lagrangians maintain the same cost levels.
INTRODUCTION
While deep reinforcement learning (DRL) has been used to master an impressive array of simulated tasks in controlled settings, it has not yet been widely adopted for high-stakes, real-world applications. One reason for this gap is the lack of distributional perspective in standard artificial agents. Endowing agents with such perspective could make their cecision-making more robust, potentially leading to increased safety, increased trust from humans, and more widespread real-world adoption.
In reinforcement learning (RL), risk arises due to uncertainty around the possible outcomes of an agent's 's future actions. It is a result of randomness in the operating environment, mismatch between training and test conditions, and the inherent randomness of stochastic policy. Risk-sensitive policies, or those that consider more than mean over the distribution of possible outcomes, offer the potential for added robustness under uncertain and dynamic conditions. There an evolving landscape of algorithmic paradigms for handling risk in RL, from constraint-
approaches adapted from
optimal control (Achiam et al. 2017; Chow et al. 2019 Ray et al. [019 Zhong et al. (020) to adversarial approaches emerging from AI Safety (García & Fernández) 2015 Amodei et al. (016)). Within this landscape, learning approaches that optimize distributional measures offer the ability to express design preferences over the full distribution of potential outcomes, through the specification of a sensitivity criterion.
Distributional RL has been studied for value-based methods, with popular strategy being to use the distributional Bellman equation to estimate the distribution of Q-values for each member a discrete set of potential actions (Bellemare et al. 2017; Dabney al. [2018a b). However, distributional RL has not been widely explored for policy gradient methods, which could permit direct optimization of risk sisk-sensieive measures and naturally accommodate both discrete and continuous action spaces.

---

Under review as a conference paper at ICLR 2022
In the following, we introduce novel framework for risk-sensitive learning using policy gradients. Our approach allows agents to be trained with different risk profiles through design-time specification of both utility and weight functions, with the latter being defined over the estimated distribution of full-episode rewards. This framework enables agent-hased learning that captures aspects of human decision-making. such overemphasis of rare occurrences and diminishing marginal utility relative to a reference outcome [Kahneman & Tversky, 1979). It also allows implementation of another key strategy of human learning: emphasizing improvement on tasks where one is deficient. We demonstrate the ability of our algorithm to use this strategy to improve performance relative to both unconstrained and constrained methods in six OpenAI Safety Gym environments (Ray et al. 2019).
2 RELATED WORK
Constrained RL offers set of approaches to safe exploration (García & Fernández (2015) that aim to enforce explicit constraints throughout the learning process via methods including Lagrangian constraints (Ray et al. (2019) and constraint coefficients (Achiam et al. (0177)) Differing from the safe exploration scenario, we here consider problems with distinct training and test phases where agent performance is to be evaluated. Our experiments indicate that diskssensitive learning can offer performance improvements over constrained learning in scenarios where safety constraints need not be enforced during training.
Distributional approaches to risk-sensitive RL have primarily been explored in the value-based setting. Therein, the value distribution has been explicitly modeled through categorical techniques (Bellemare et al. 2017) or quantile regression (Dabney et al. 2018a) and used to improve both value predictions and overall performance. Recent works utilize distributional modeling in the actor-critic setting to enable application to continuous action spaces, again demonstrating improved performance over baseline approaches (Ma et al.) 22200] Zhang et al.) 2021 Duan al. [202I). In value-based approaches, risk-sensitivity criteria are applied at run time as nonlinear warping of the estimated value distribution.
Policy gradient approaches offer additional promise for risk-sensitive RL, but require direct optimiza- tion of parameterized policy with respect to distributional objective. Some existing methods are limited to specific class of learning objective, such as the set of concave risk measures that allow a globally-optimal solution (Tamar et al.) 2015) Zhong et al.] 2020). Others allow broader class of measures but are more restrictive in the class of policies that can be represented (Prashanth et al., 2016). We aim for a risk-sensitive policy gradient approach that both offers significant flexibility in the choice of learning objective and can learn policies parameterized by deep neural network.
Various measures have been considered in the context of risk-sensitive RI., including exponential utility (Pratt 1964). percentile performance criteria (Wu & Lin 1999), value-at-risk (Leavens) (1455. conditional value-at-risk (Rockafellar & Uryasev] 2000). and prospect theory (Kahneman & Tversky 1979). In this work, we consider class of risk-sensitivity measures motivated by Cumulative Prospect Theory (CPT) (Tversky & Kahneman] 1992). CPT uniquely models two key aspects of human decision-making: (I) utility function u, computed relative reference point that induces more risk-averse behavior in the presence of gains than losses and (2) a weight function w that prioritizes outlying events. Specific forms of u and w are given in Tversky & Kahneman (1992) (and in Appendix A.4). but the general form of CPT admits a wide variety of iiskiicnstvive objectives.
Here we show how to train agents optimize this class of objectives through sampling- estimation of their policy gradients and requisite variance reduction. The final algorithm resembles well-known on-policy approaches such as Proximal Policy Optimization (Schulman et al. (017b) and is similarly widely applicable. Although we do not explore it here, the incorporation of an appropriate risk-sensitivity criterion could additionally enable risk-aware exploration and adversarial training for increased robustness (Pinto et al. 2017 Parisi et al. 2019 Zhang et al. 2020).
3 SENSITENE POLICY OPTIMIZATION
In this section we formalize the class of distributional objectives to be considered, derive sampling- based approximation of its policy gradient, enact variance reduction on this estimate, and use the result to produce practical learning algorithm.
2

---

Under review as a conference paper at ICLR 2022
3.1 PRELIMINARIES; PROBLEM AND NOTATION
Standard deep reinforcement learning seeks to maximize the expected reward of an agent over encountered trajectories; that is, it maximizes the objective
((0) = Ex)
Σ r(s,, ar)
Here (D( the distribution over trajectories 7 = $1, a1,..., ST, ar induced by a policy parameter- ized by 0: Sr, a,, and r(sr, ar) denote the state, action, and reward time t, respectively. To enable the incorporation of distributional context, we instead consider the objective
((0) - f to0 u(r(T)) (त(c)
d (e-((w(Porrt (w((Par(ti) (
where u(r(T)) is the utility associated with full-trajectory reward r(T) = E, r(s, a.) and w is piecewise differentiable weighting function of the CDF of trajectory reward Po(r(T))
1 po(r))dr'.
Equation 2is inspired by CPT Tversky & Kahneman 1992). which includes a pair integrals of this form. It was chosen for its generality; by using different utility functions u and weight functions w one may represent all the risk measures mentioned Section and all of the risk measures evaluated by Dabney al. (2018a). The form (2) reduces (I) when w. and mapping. While designed for the episodic setting, the objective (2) may be considered for infinite horizons through the choice of appropriately long windows.
both the identity
3.2 RISK-SENSITIVE POLICY GRADIENT
To optimize the objective 22, first derive approximation its gradient with respect to the policy parameters 9. Working toward representation that can be sampled, we assert the independence of the reward and use the chain rule write
VaJ(O) - / u(r(T))
d dr(r)
( (c((RArtt)) (r(()))))r(().))))))))))
dr(T).
where w' is the derivative of with respect to Po(r(T)). The gradient of the CDF may be written follows:
VoPo(r(T)) Vo / po(r')dr'= Vo voJ. ((())))(((()))
-Va
(4)
{ /uutt:/fnD-aaatooooooo//tttt///- loggo"o
Here the first equality is the integral representation of (o(r(r)), the second uses the Heaviside step function to select all trajectories with total reward S r(T), the third follows from the independence of reward on 0, and the fourth follows from the expression for the derivative of the natural logarithm. In the following, we also use the complementary expression
voPa(r(T)) Vo (1 po(r")dr) -
(((r)
(5)
Either form, or a combination of the two, may be substituted into (3) and the result sampled over N trajectories by first ordering trajectories =I..N N by increasing reward r(7). Then
N VoJ(O) ~ Σ (((((T)) w/
( (.oPo((TT)) - w' "'' ) (6)
3

---

Under review as a conference paper at ICLR 2022
where the term w' (r = 0. This ordering scheme produces asymplotically consistent
estimate, as shown in the context of CPT value estimation by Prashanth et al. (2016). VoPo(r(T))
may be sampled in one of two ways, based on either
T
T
VoPo(r(T.)) N
Vo log (a(a,)))))))
Vo log mo(aaalss))
N
j=++1
The expression may used to train policy that optimizes the distributional objective in manner similar to REINFORCE (Williams) 1992).
3.3 VARIANCE REDUCTION
Reducing the variance sample-based gradient estimates enables faster learning. Here we take several steps reduce the variance of (6), as has been done with the policy gradient estimate of REINFORCE (Williams 1992).
First, note that tross-trajectory terms of the form [(T,; aj.t, 8j,a) u(r(T.)) Vo log To (a ($3.)) while nonzero, do not contribute to the gradient estimate in expectation when + j. A proof of this assertion given in Appendix A.1 Using (4) for the first term of (6) and (5) for the second allows us to write
T
VoJ(O)
u(((n)))
w'
V log
N
N
(8)
1
+w/
Vo log no (a,,1(8.))
N
N
j=
Removing cross-trajcctory terms gives
VoJ(O) ((r((s) ((w' ww
T. Ve log To (ar.)())).)
w/
N
Note that the weight coefficients (w ( + (-x)) should be normalized over each batch. The expression (9) is equivalent to (6) in expectation, but with reduced variance (see Topendix[A.]]fl justification). It has clear intuition trajectories are assigned utilities based on their rewards and their contributions to the gradient are scaled by the derivative of the weight function, just as they are in Cumulative Prospect Theory (Tversky & Kahneman, 1992).
Standard variance reduction techniques may be applied to this simplified form. Without further assumption introduction of additional bias, static baseline b can be employed:
(urtr)))-b) w/
(+) N
VoJ ~
+ w
V log To (a
(10)
Justification for this assertion is given in Appendix AA. Learning may be further improved if we additionally assume that utility may be allocated are computed as the difference between what the full-episode utility would be the episode were to end given time step and what it would have been had the episode ended at the previous time step. While not applicable in cases where episode utility is adjusted based on final outcome, this assumption has the significant benefit of modeling the temporal allocation of rewards and aligns with the standard formulation of RL. With it, the variance of (9) may be further reduced through the incorporation of atillity-to-go and deeendde baseline (s
per-step basis. In this case, per-step utilities
VoJ(O))
Vo log To (an.(ss.)
Σ as (a..:)-Vooss.) (11)
N
N
Here u(s,,r, ai,e) the per-step utility. The value function Vo(s.,) parameterized by & and trained via regression to minimize
(Vo(s..)
Tc Σ
((d) Σ
u( a,,))
(12)
8,
4

---

Under review as a conference paper at ICLR 2022
A standard argument, similar to the approach taken in (Achiam 2018), can be used show that the incorporation of utility-to-go does not change the expected value of ( The addition of a state-dependent baseline also does not introduce additional bias, justified Appendix A.2
Finally, discount factors, bootstrapping, and trust regions may be used to provide additional variance reduction, just as they in conventional on-policy learning (Appendix A.2).. These measures may introduce additional bias to the policy gradient estimate, but typically lead to more sample- efficient learning. In our experiments, we evaluate the use of generalized advantage estimation (GAE; (Schulman et al. 2016)) based on the utility-tO-go as well as the clipping-base trust regions of Proximal Policy Optimization (PPO; (Schulman et al. 2017b)). Incorporating these in the policy gradient yields
VoJ(O)
= N w
N Σ 1=1
2 N
+
Σ
I
Vo stip
log (o(a,,, A7 AT((s,,,a..)) ).:)). (13)
N
where (s an.s) is the standard GAE except with per-step utilities in place of rewards. Trust regions are implemented similarly to PPO, pessimistically clipping policy updates to be within multiplicative factor of c of the existing policy:
Lehp min log [ ar,e);
(14)
(a, (8t,e) 1 + Tesa (as.))s..) A (Sa,e, (.s,,,a).))) a..))
log
clip
Teaa (as,
They are used to perform multiple policy updates using the same batch of data, providing learning that is longer strictly on-policy but that can be significantly more sample efficient. When following this route, we apply the same early stopping mechanism, based on the Kullback-Leibler divergence (DKl) between old and new policies, as was used by Ray et (0199)
3.4 LEARNING ALGORITHM
The above sample-based estimate of the policy gradient may be used train agents to maximize distributional objectives of the form 2 The resulting method, Cumulative Prospect Proximal Policy Optimization (C3PO), is given in Algorithm aand mirrors standard on-policy learning.
Algorithm 1 Cumulative Prospect Proximal Policy Optimization (C3PO)
Require: Policy: initial parameters Co. learning rate 0g, updates per batch Mo
Require: Value: initial parameters Φo. learning rate Ogg updates per batch Ms
Require: Early stopping threshold DRL, stop- discount factor 7 for 1,2,... do
Collect set of episodes Dj (T.) ) by running policy x(ax) in the environment
Compute per-step utilities u(s,,t. a,,))
Fit value function by regression:
for m = 1,... Mo do
o T. Eis
Eie (Vo(su))- Et "u(s,,,,,
end for
Update billity-bssed advantage estimates A a), using new (s) Compute weight coefficients based ordered episode outcomes and normalize Update policy, using KL-based early stopping:
for m 1.... Mg do
if ( ) < DKL, stop then
0 - 0 a9 (w else
(w ( w ()
V VoLeii (log To (og (ai, (8.,,,,,
break end if end for end for
5

---

Under review as a conference paper at ICLR 2022
Weiht Farziions
Wiight Confsolntte
08
Wong 1711
0..2
CFT
More 0010
Wong -8.21
Wore (o -039
0.2
00
Probabiitty
Probabiity By
Figure I: Example weight functions and their resulting coefficients in the policy gradient estimate In these plots, outcomes increase in quality from left to right. As in CPT (Tversky & Kahneman 1992), weight coefficients are proportional to the derivative of the weight function.
Beyond the utility and weight components, Algorithm I differs from conventional methods in the requirement to collect full episodes of data in each batch. This requirement can be removed if outcomes can be defined over partial rather than full episodes, an assumption that is often viable and matches human decision- maakong. For instance, while out of scope for this work, our approach could be applied the Atari suite (Bellemare al.] (213)) by considering the outcomes of fixed-length windows and restructuring Algorithm| to mimic minibatch PPO (Schulman et al. 2017b).
4 EXPERIMENTS
To evaluate our approach, sought to both establish that it can effectively optimize different distributional objectives and explore the impact of using different objectives on agent outcomes. We found the OpenAl Safety Gym (Ray al. 2019) be suitable for these purposes. Safety Gym is configurable suite of continuous, multidimensional control tasks wherein different types of robots must navigate through obstacles with different dynamics to perform different tasks. By including both positive and negative events in each training scenario, it allowed us to evaluate how our various agents handled risk. Safety Gym also highly stochastic: the locations of the goals and obstacles are randomized, leading to outcome variability and forcing the agent to learn generalized navigation strategy.
Safety Gym logs adverse events but does not incorporate them into the reward function. As our method relies solely on the training signal from the reward, we assigned each logged adverse event fixed, negative reward contribution in experiments using it or other unconstrained agents. Our initial experiments were conducted with a reward contribution of -0.025, which was found to allow agents to prioritize reaching goals but deter them from collisions with obstacles. To further emphasize obstacle avoidance, we doubled this contribution to -0.05 in our experiments using cautious weightings. These choices and the role they play are further discussed in Section
To highlight distributional differences, we focused on the publicly available, obstacle-rich level 2 environments [ Avoiding the longer compute time of the "Doggo" robot, evaluated the "Point" and "Car" robots on each task ("Goal", "Button", and "Push"). Further details on these environments and our rationale for choosing them are given in Appendix A.3
In all experiments, we evaluated five random seeds and matched the hyperparameters used in the baselines accompanying Safety Gym (Ray et al. 2019) as closely as possible. The neural networks used to model both policy and value were multilayer perceptrons (MLPs), with two hidden layers of 256 units each and tanh activations. As in Ray et al. (2019). the policy network outputs the mean values of multivariate gaussian with diagonal covariance. The control variances are optimized but independent of state. The full complement variance reduction measures were used throughout; see Appendix A.4 for experimental justification of this choice.
4.1 DIFFERING OBJECTIVES
Agent performance was explored under four different distributional objectives. In addition to expected reward and CPT (configured to match the original form of Tversky & Kahneman (1992) and as given in Appendix A.4) optimized for cautious (n 0.75) and aggressive (n) -0.75) versions of the
In Safety Gym, the default environments have three levels (0, 2); obstacle density increases with level.

---

Under review as a conference paper at ICLR 2022
CarButton22 Episode Rewards
CorBucton! Episode Costs
CarBurnor2 Outcome Dienher
046
CPT
Mon Eaiecc Recuel
12.5
145
Beng On 1791
08.0
0.44
Hang O1 (AT3]
1.5
50
640
Deae 000
2.5
1.00
0.0
EX0
-75
50
Training Experferon
Traking
Spisde Rowid
Figure 2: Impact of different distributional objectives in one environment (CarButton22. The shading in the first plots (and subsequent learning curves) reflects the standard deviation associated with running over random seeds. Left: Net reward (positive reward minus penalty) throughout learning, Middle: Average number of cost events per episode during training (lower is better). Right: Agent outcome distribution in testing (with sampling turned off). The cautious (Wang (17 = 0.75) weighting shows higher reward and lower cost once trained.
distortion risk measure proposed in Wang (000)). This measure is defined as w(() = (p) 1)), where and h-1 are the standard normal cumulative distribution function and its inverse. While we found this form to be convenient, the "Pow" metric in Dabney et al. (2018a) or any other set of similarly shaped w curves should achieve a similar effect. In experiments using the objective from Tversky & Kahneman (1992), the reference point was taken to be the mean episode reward of the current batch, matching the tendency of humans to change their standards over time. The four weight functions and their resulting coefficients in (9) are shown in Figure []
Plots of the total rewards (including penalties) in training, average cost events per episode in training, and outcome distributions in testing are shown for one environment in Figure for two additional environments in Appendix AA.5] The trends were fairly consistent over the three environments evaluated in this manner. While no explicit effort was made to handle cost (agents were given only
the sum of positive rewards and penalties), the cautious and aggressive weightings consistently accumulated relatively low and high costs, respectively. The cautious (Wang(1) 0.75)) agent typically also generated the highest positive and total rewards after 10 millions steps of training.
To generate the histograms in Figures 2 5 8
and 1 as well as the numbers in Table the trained agents were deployed on a set of 5000 test episodes 1000 for each of the 5 networks learned using different random seeds in training. The resulting distributions therefore include contributions from both aleatoric and epistemic uncertainty. Sampling was turned off, allowing the agents to choose their perceived optimal action at each time step. In this context the benefit of emphasizing the lower part of the outcome distribution (i.e., cautious weighting) became more pronounced, in part because the methods that emphasize poor outcomes tended to maintain higher policy entropy (Appendix AA.5)
4.2 CAUTIOUS WEIGHTINGS
To further explore the apparent benefits of cautious weightings in Safety Gym, we trained a scries of variably cautious agents by tuning 7 in the risk-averse weight function proposed by Wang (2000). Histograms of their episode rewards in testing are given in Appendix[A.55 [A.5]and summarized in Table D In these environments, agent performance both in terms of improving the lower end of the reward distribution and on average was seen to generally improve with increasing until the range y E [0.75, 1.25], subsequently degrading. Additional comparisons were made with PPO (Schulman et al. 2017b), which unsurprisingly was found to closely track performance of the "Uniform" agent. We found that naively incorporating cautious weightings into PPO improved its performance (row PPO + Wang(0.75) in Table Tb though not to the level of the full C3PO method with 1) = 0.75.
We then pursued a set of longer runs to compare C3PO with the cautious objective from Wang (0000) to both unconstrained and constrained benchmarks. Here we did not tune 7) keeping it fixed at 0.75 for all experiments. Comparisons with unconstrained methods for three environments are given in Figures and and for the remaining three environments in Appendix A.6 In addition to PPO, we compared performance with Trust Region Policy Optimization (TRPO; Schulman et al. (2)77)) as configured in Ray et al. (2019). Since this TRPO configuration generally outperformed the PPO configuration in Ray et al. (2019) from which we derived the hyperparameters for C3PO, we would expect C3PO be disadvantage compared to TRPO. However, found C3PO had the highest
7

---

Under review as a conference paper at ICLR 2022
PointButton2
CarGoal2
CarButton2
Mean
Std
Q=0.5
Q=.05
Mean
Std
Q=1.5
Q=005
Mean
Std
Q=0.5
Q=005
Uniform
22.5
7.1
22.8
10.7
18.1
6.6
18.0
7.4
12.4
9.1
14.0
-6.1
CPT Value
16.5
16.3
13.9
7.7
14.6
-0.5
10.7
11.2
-11.0
Wang (-0.75)
17.5
6.1
17.6
7.6
13.5
6.9
14.0
0.9
6.4
10.7
8.9
-15.5
Wang (0.5)
23.3
6.3
23.3
13.1
18.5
6.0
18.8
8.2
12.9
9.0
14.4
-5.6
Wang (0.75)
24.2
6.7
24.4
13.7
19.0
6.4
19.3
8.2
14.3
9.5
15.8
.4.4
Wang (1.0)
24.7
24.9
15.1
20.3
21.3
11.4
11.0
13.8
-12.2
Wang (1.25)
25.4
6.1
25.4
15.9
17.3
6.7
17.8
5.6
12.7
10.4
14.8
-8.3
Wang (1.50)
23.6
5.9
23.7
14.1
16.6
7.5
17.2
3.1
10.1
12.0
13.3
-17.9
Wang (1.75)
23.4
6.2
23.5
13.5
12.5
8.0
13.0
-0.6
7.3
13.0
11.1
-22.8
PPO
19.0
18.9
15.8
15.8
9.5
PPO Wang(0.75)
20.4
8.4
21.7
2.6
17.7
6.7
18.0
6.3
12.1
9.5
13.6
-6.8
Table 1: Testing statistics for episode rewards achieved by agents trained over 10 million steps with
different distributional objectives. Q 0.5 is the median and = 0.05 refers to the location of the 0.05 quantile. Blue bold-face represents the best performance for a given environment; in all cases
these occur for moderately cautious weightings (1 E [0.75, 1.25)..
PointGoal2 Epiodd Rowards
Carlucton2 Episode Rewards
PoinoPust2 Episoce Rewards
1N
Unitur
Moi Eaoode Rewurd
10
CIPO oo-
Unelam
U-fore InO Teno
Moon
0
Troining Experiencs
Triniing Experiences
Tichning
Figure 3: Average episode reward (including penalty) over training for different learning approaches
in three different environments. C3PO with Wang (1 0.75) weighting outperforms others.
average reward (including penalty) in five of the six environments and lowest average cost in five of the six environments. In addition, agents that used the cautious weightings tended to have more stable and repeatable training, as evidenced by the tight distribution of their learning curves. This tightness was found to reflect a lack of negative outlier episodes and potentially lower epistemic uncertainty throughout training. Finally, note that the use of a nonzero penalty for cost events resulted in significantly lower incurred costs than were observed with unconstrained agents trained without a penalty (Ray et al. 2019). PPO and TRPO were seen reach similar cost levels without penalty; these levels are indicated by red dashed lines in the cost figures.
Comparisons with versions of PPO and TRPO that use Lagrangian constraints (PPO-Lagrangian and TRPO-Lagrangian; |aa et al. (2019) to match the cost level of C3PO are shown in Figure 6 and Appendix A.7 We see that, given the same level of cost incurred per episode, agents trained using C3PO consistently achieve higher levels of reward than those trained with PPO-Lagrangian and TRPO--Lgrangging sensitive method. Additional comparisons were generated with Constrained Policy Optimization (CPO: Achiam et al. (2017)), but are not shown in Figure 6 because they failed to maintain the cost levels of the other methods. For completeness, they are given in Appendix A.7
As above, training is seen to be more stable and repeatable using our risk-
PaineGoot2 Eppoode Costs
Carlbntonn spd Costs
PoircPush2 Episode Costs
SPO
Mogn goodk
390
100
CIRD
TePO
(150
15
25
Trainin Expereeces
Trikiin Expeionee
Tickiing Experences
Figure 4: Average number of penalty events per episode (lower is better) over training for different learning approaches in three different environments. The horizontal lines reflect the cost levels reached by both PPO and TRPO training with zero penalty in Ray al. (2019).
8

---

Under review as a conference paper at ICLR 2022
PrintGeall Rmorrd bxtion
Carlurtoe2 Rrcard Didrisations
PoinfPak2 Reward Dionbations
FPO
2.55
PPO
0.30
T1F0
TRPO
T3PC
(1R0 -
1.04
CHR (y -110))
Potaaday
8.10
0.30
-10
Eskodo Rovard
Reward
Figure 5: Testing reward distributions (including penalty; sampling turned off) for long training runs of three Safety Gym environments.
PointGool2 2 bpisode Poskive Rowords
Car Bactoon2 Epbode Hosiivve Rewards
Point-lsh2 tykcode Positive Hewards
Lagregen
T&PO
Moan Roote Breant
(3PD (20)
12 55
Meon
Expelences
Traniig Experiencos
Tichiing Excelences
Figure 6: Comparison of positive contributions to episode reward during training for our approach (yellow) and Lagrangian methods configured to have the same cost level.
5 DISCUSSION
The analysis above allows for sample-based policy gradient estimates of a broad class of distributional objectives. Variance reduction measures were shown to enable efficient optimization based these estimates (Appendix (A.4). However, it was not seen to be the case that given distributional objective could be most effectively optimized directly. Instead, the best results were generally obtained through moderate emphasis on improving negative training outcomes (Table I.
To understand this behavior, consider the interplay of optimization and exploration in the training of cautious and aggressive agents. Cautious weightings continually emphasize the lower part of the outcome distribution, pushing that part of the distribution upward and adjusting behavior the most where it is most necessary. Once part of the state space where the agent is deficient is adequately addressed, different part of the state space takes place. Policy entropy remains high because of the emphasis on problematic situations, ensuring adequate exploration. This trend continues with increasing 7), until the point where the agent begins to ignore high quality training outcomes too much. Conversely, aggressive weightings continually emphasize the best outcomes in the distribution. When an already strong outcome is given increased attention, it is likely to stay at the top. Hence agents trained with aggressive weightings tend to become myopic, obsessing over fraction of the state space while neglecting the rest of it. They tend to explore inadequately and ironically fail to attain better top-end performance than more cautious weightings.
Given the consistent performance gains observed using our method, we propose that it represents useful option for improving the performance and stability of on-policy learners. This should be particularly true in the presence of a meaningful trade-off between positive and negative reward terms and when there is significant stochasticity. While our approach does add an additional hyperparameter one value for that hyperparameter was seen to provide gains across all
the shaping constant environments tested. While our approach does not provide for direct choice of cost limit as constrained methods do, it is simpler to implement and was consistently seen to be more performant for the cost level reached. It is also likely possible to use cautious weightings in conjunction with constrained RL, though this has not yet been investigated.
CONCLUSIONS
In this work, we proposed a risk-sensitive learning algorithm based on a policy gradient estimate for broad class of distributional objectives. When configured to emphasize improvement in scenarios where the agent performs poorly, we found our method to compare favorably with existing unconstrained and constrained on-policy learners.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Josh Achiam. Openai spinning up: Proof for don't let the past distract you, http: http:: /stp://i/pinnnngu openai . ccm en wwtest/spiiniiguuuee cattgguppeettttaeeeeeeeeeeee extra http:eaaaapoooooil____. html 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In
Proceedings the 34th International Conference on Machine Learning (ICML), pp- 22-31, 2017.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. urXiv:1606.06565 2016.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pp. 449-458, 2017.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar A. Duéñez- Guzmán. Lyapunov-based safe policy optimization for continuous control. CoRR, abs/1901.10031, 2019. Uttp::::// org. abs 1901. 10031
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 1096-1105, 2018a.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), pp. 2892-2992099 2901, 2018b.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2021.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econo- metrica, 47(2):263-291) 1979.
Dickson Leavens. Diversification of investments. Trusts and Estates, 80:469-473, 1945.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv:2004.14544 June 2020.
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, May 2019.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
L.A. Prashanth, Cheng Jie, Michael C. Fu, Steven Marcus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In Proceedings of the 33nd International Conference on Machine Learning (ICML), pp. 1406-1415, 2016
John W. Pratt. Risk aversion in the small and the large. Econometrica, 32: 122-136, 1964.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. httpps://cdl opena wtt.://////cepp/eeeeeeeecea short. pddf 2019.
R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization conditional value-at-risk Journal of Risk, 2:21-41 2000.
10

---

Under review as a conference paper at ICLR 2022
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference Learning Representations (ICLR), 2016.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 1rXiv:1707.066: 2017b.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. In Proceedingy of the 28th International Conference Neural Information Processing Systems (NeurIPS), pp. 1468-1476, 2015.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty, Journal of Risk and Uncertainty, 5(4):297-323, 1992.
Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance, 67(1):15, March 2000.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-256, May 1992.
Congbin Wu and Yuanlie Lin. Minimizing risk models in Markov decision processes with policies depending on target values. Journal of Mathematical Analysis and Applications, 231:47-67, 1999.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 21024-21033 2020.
Junyu Zhang, Amrit Singh Bedi, Mengdi Wang. and Alec Koppel. Cautious reinforcement learning distributional risk in the dual domain. IEFE Journal on Selected Areas in Information Theory, 2(2):61 1 126 2021.
Han Zhong, Ethan X. Fang, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive deep RL: Variance- constrained actor-critic provably finds globally optimal policy. arXiv:2012.14098, 2020.
A APPENDIX
A.I EVALUATION CROSS-TRAJECTORY TERMS POLICY GRADIENT
In this section, we first show that the cross-trajectory terms in our policy gradient estimate (6) have an expectation value of We then argue that their removal leads to a policy gradient estimate with reduced variance.
Lemma 1. Cross-trajectory terms of the form [(T,, Sj., u(r(TT)) log ma (1)r))) where 8 + j. do not contribute to the gradient estimate (6) in expectation.
Proof. First, note that
Bnn-m log To (ay,)))).)
= Errppo())
u(r((:))
Vo Vol
11

---

Under review as a conference paper at ICLR 2022
Then consider the innermost expectation:
f p(sj.er a5 To, V, log no
Bry)po(t) Vo log To (a)
=
(Is /
To(aj,1lsy.) Vo log To (a (sd
J p(s).//TT/7
(oTo(a Iday
p(s),ulTe Vg
To(aj, 1 dd.y)day.
Jay
p(s (V) O.
To why removal of cross-trajectory terms leads to reduced variance, consider that the full expression (6) may be written as the sum of terms of the form [(Ti, 8j,a) u(r(T:)) log mjja 8j,))- Its variance is the of the total variance from terms where the total variance from terms where i * j. and term proportional to the covariance of these two totals. However, because each term in the covariance contains at least one trajectory that differs from the rest, the above reasoning may be applied to argue that the covariance is O. Hence, the removal of the cross- cross-trajecteo trajectory terms lowers the variance of the policy gradient estimate by the variance of the cross-trajectory terms.
A.2 INTRODUCTION OF STATIC AND STATE-DEPENDER BASELINES
Lemma 2. A static baseline of the utility may be added to the policy gradient estimate introduction of bias.
9 without
Proof. The additional term is 0 in expectation as
(T)
w
w
Ve log To (ar,
72
T
w
w
Vo log To (a).)))))))) ds
72
n
Bi.
i
p(st,e(TO) / To(ar, log To (a(a (s. dda.
w
w
p(si,lTTOVV / To(an, (s )da,
w
w
n
n
i
w
w'
(( = 0.
The contribution of the weight terms (2) + (((= may be pulled out of the integral between the first and second line because of its independence on both state and action. This term is fixed for given trajectory by the rank of its reward amongst the rewards accumulated on all trajectories in the current batch.
In our variance reduction experiments (Appendix A.4 the "Base" agent uses b equal to the mean of full-episode utility in the current batch.
As described in Section 3.3 may further adjust the policy gradient estimate through introduction of ser-step utilities. this case, we may justify the use of state-dependent baseline through the following.
12

---

Under review as a conference paper at ICLR 2022
Lemma 3. A state nete-dependent baseline (o(s..)) may be added to the policy gradient estimate without introduction of bias, if per-step utilities are assumed.
Proof. The additional term 0 in expectation as
(r()
w
+
w
Vo log (ogg
w
+
w
log (a..
(( [ Te(ai,((s..) Ve log To (ar, (ar. s1., ddy
w
w
w
To(an,,ss, )da,
a..
w
+
p(s TO (s;
(s)
(Voldds =0 0.
The rationale for pulling the w' terms out of the integral is the same as in Lemma
Finally, we note that the ability to pull the contribution of the weight terms (w' + w (== to the front of Equation [ allows us formulate advantage estimates based on per-step utility. Bootstrap estimates of the value function (s and Generalized Advantage Estimation as in |Schulman et al. (2016) can be conducted exactly as they are standard on-policy learning, if rewards are replaced by per-step utilities.
A.3 ADDITIONAL INFORMATION ON SAFETY GYM
As mentioned in Section 4 we chose to evaluate our approach using the OpenAl Safety Gym (Ray et al. 2019). The choice was governed by our desire to test in conditions with clear cost-benefit trade-ofis, significant stochasticity, adequate complexity, and available benchmarks. While our methods are not limited to particular task types or observation/action spaces, we found Safety Gym to be suitable for exploring their potential.
The six environments chosen were the most obstacle-rich of the publicly available environments that used the "Point" and "Car" robots. The Point robot is constrained to the 2D plane and has two control dimensions: one for moving forward/backward and one for turning. The Car robot also has two control dimensions, corresponding to independently actuated parallel wheels. It has freely rotating wheel and, while it is not constrained to the 2D plane, typically remains in it. While we expect our did not experiment with it because of
results to extend to the remaining default robot, "Doggo", the order of magnitude longer training times it exhibited in Ray et al. (2019).
Several types of obstacles and tasks were present in the environments we evaluated. In all cases, the robot is given a fixed amount of time (1000 steps) to complete the prescribed task as many times as possible and is motivated by both sparse and dense reward contributions. In the "Goal" environments, candomly-assigne
the robot must navigate to a series assigned soon as a goal is reached. In the "Button" environments, the robot must reach and press a sequence of goal buttons while avoiding other buttons. In the "Push" task, the robot must push a box series of goal positions. The set of obstacles are different for each task; among the three environments there are a total of five different constraint elements (hazards, vases, incorrect buttons, pillars, and gremlins), each with different dynamics. See Ray et (0199) for further details.
goal positions, with a new target being
A.4 EMPIRICAL PERFORMANCE OF VARIANCE REDUCTION MEASURES
To gauge the impact of the variance reduction techniques outlined in Section 3.3 we evaluated their performance in maximizing the value function of Cumulative Prospect Theory Tversky & Kahneman 1992). As mentioned in Section 3.1 this function has two integrals of the form (
13

---

Under review as a conference paper at ICLR 2022
PaineButto22 Varonce Reduction
Co Gool2 Varance Reduction
CarButton2 Vanance Reduction
646
10
-10
Dua
-15
20
Traiiin Experfencet
-17
Tralning Experiences
Figure 7: Impact of variance reduction measures on optimization of the CPT value function. Here "Base" refers to the risk-sensitive policy gradient estimate (TO), "UTG" adds utility-to-go and neural network bascline (TT), "GAE" incorporates generalized advantage estimation, and "TR" implements trust regions via clipping. Shading represents the variation over five random seeds.
J(0) / (r(T)) d dr(r) w (P((((((t)))) dr(())
(15)
ut (r(T)) dr(r)
d
(-wt(1-Pote dr(r)
In Tversky & Kahneman 1992), the utility functions computed relative reference point and reflect the tendency of humans to be more risk-averse in the presence of gains than in the presence of losses. The weight functions (wt } model inclination emphasize the best and worst possible outcomes in our socision-making
More specifically, in these experiments we used the piecewise utility functions ut (r) H(r (r)(r-ro) and u = >H (ro -r with static reference ro 10, a 0.88, and À = 2.25.
The weight function w(p) (1-p()-)))) +
used, where 0.61 for To and r 0.69 for r 2 Tg. Four methods were evaluated, incorporating progressive amounts of variance reduction:
p²
Base: Risk-sensitive policy gradient with a static baseline (10)
UTG: Base with utility-to-go and neural network baseline 111
GAF: UTG with generalized advantage estimation ( (13 without clipping)
TR: GAE with trust regions
13 with clipping
(44
As shown in Figure 7 the incorporation of these techniques increased the sample efficiency of the CPT value optimization significantly. Consequently, we used the full complement (TR) in all other experiments.
A.S DIFFERING OBJECTIVES: ADDITIONAL RESULTS
Below include results for all environments for the experiments described in Section 4.1
14

---

Under review as a conference paper at ICLR 2022
CorGcal2 Episode Rewards
CarGoo Epioode Costs
Carorall Outcome Dver butices
Moin Eniecs Renurd
1.6
CPT
Meig On 121)
Paag O1 -AT3)
75
L.2
25
040
Trainin Experienoes
Traknng
PoineButton? Epeode Rewards
3uton3 Epsode Costs
PreeSarteed Outrome Datrisation
199
0.0
CPT
Moan Eavece Swoodd
1.55
13
1.0
CPT
1.02
Mann LTT]
Warg - 6:55
m 0000
de 1030)
3.0
0.5
0- Training Experiencos
0.4
L4
00
0.2
3.4 Trakkin Experiences
5.6
68
1.0
Estode Rowd
Figure 8: Impact of different distributional objectives in remaining two environments of initial trials. The shading reflects the standard deviation associated with running over 5 random seeds. Left: Net reward (positive reward minus penalty) throughout learning. Middle: Average number of cost events per episode during training (lower is better). Right: Agent outcome distribution in testing (i.e., with sampling turned off).
CarGrol2 Dinouee Dianbetions
PrintButtonn Ounover Ditribations
Carfurro12 Outoome Dieslee
1.67
E.63
Kheg 124
1.02
L.22
1.11
0:11
1.00
0.30
0.00
Esxodd (favard
speade Sowar
ta scde 1tevard
Figure 9: Agent outcome distributions across trials run over increasingly cautious (n) increasing) objectives. Distributions correspond to results shown in Table
In addition, we note the trend of policy entropies with different distributional objectives. In general, more cautious weightings maintain higher entropy that these plots represent an upper bound because they do not account for action clipping by the environment; see Ray
for longer than more aggressive weightings. Note
(2019) for details.
CarBu Policy Eroopy
CorGoatl Policy Estroyy
Pointurmtn Policy Encropy
1.0
3.2
Paky Erener
2.6
2.4
Werg C.70)
0.700 "We Ww (11
3.71
Wendi-
02
Traking Expereers
Triking Experienee
Traiing Exprrinces
Figure 10: Policy entropy progression during training for three environments. Shading reflects the observed variation over 5 random seeds.
15

---

Under review as a conference paper at ICLR 2022
A.6 ADDITIONAL COMPARISONS WITH UNCONSTRAINED METHODS
Below are plots of average episode reward and average number of episode cost events throughout training for the remainder of the environments which we conducted long (Section 4.2 Also included are histograms of testing performance for those runs.
CorGeatl Epicode Rewards
PointButton2 Episode Rewards
CarPesh2 Epicdd Rewa ds
Moan Eaiece Rooord
Moan Eases Spcan
Moin 0
PPO ming
-1
CIP (10))
TESPO
Traniin Experiecee
Trakiing Licelince
Traiking Experioncss
Figure 11: Average episode reward (including penalty) over training for different unconstrained
learning approaches in remaining three environments.
CorGeell Episade Costs
PaintButtord Epeode Costs
CarPush2 Epicode Cort
200
3.4
120
Hea Ejerds Con
Cano
PRO
FPO
1991
card (77)
CSPO (y 9.75 gene
Training Experiences
Trakiing Experiences
Experiances
Figure 12: Average number of cost events per episode (lower is better) training for different unconstrained learning approaches in remaining three environments. As above, the "zero-penalty" line refers the level reached by PPO and TRPO trained with no penalty in the reward (Ray et al. 2019
CarGeo22 Hrward Ditribatio
Nentiuutton Reward Detribttion
CarPheed Reword Detribation
2.03
Unbone
0.31
Unbore
0.30
Cyttmm
2.55
1.35
Py Deelt
8.04
0.31
0.15
2.03
0,22
1.1
181
0.30
-99
20
1.0
-10
Exioodl Rowand
Episnde Ronart
Eatoree Rowi
Figure 13: Testing reward distributions (including penalty: sampling turned off) for long training runs in the remaining three Safety Gym environments. In five of the six environments, C3PO with 1 = 0.75 provides tangible benefit. smaller 17 is likely required to improve performance on CarPush2.
A.7 ADDITIONAL COMPARISONS WITH CONSTRAINED METHODS
As mentioned Section 4.2 we compared the performance of C3PO with constrained methods by setting the cost limit of the constrained methods match the cost level attained by C3PO. Here provide
the positive reward plots for the remaining three Safety Gym environments studied,
the cost plots for each of the six environments, and
16

---

Under review as a conference paper at ICLR 2022
all plots for Constrained Policy Optimization (CPO; Achiam et al. (2017)
The intent of the cost plots of Figure 15 is to show rough consistency between the cost levels of our approach and Lagrangian methods configured to have the same cost limit. This is verified, but other trends should be noted. First, while the Lagrangian-based methods typically follow the cost constraint well, they cannot satisfy it in each batch. Second, our approach tends to have comparable or lower cost rates throughout training. This safe exploration metric, defined in Ray et al. (2019). refers to the average cost per episode over all of training up to given point.
Results related to Constrained Policy Optimization (CPO: Achiam et al. (2017)) are included here but not in the main text because, consistent with (Ray et al. 2019), we were not able to configure CPO to respect the cost levels of the other constrained methods. Here we show the cost levels reached by CPO compared with C3PO (Figure [6] as well as comparison of the average episode rewards of the two (Figure For the latter, we employed the penalty scaling used by C3PO to enable a fair comparison.
CarGool2 Speccee Postive Rewa de
PoinButtonl Ezitdde Postive Revarte
Carlushl Epizode Pootive Recards
15
Mon Fagiie Powedd
18
12
Maes Fastir
8
PO Lugaang Moon 1211
PVD ThrDLagenn agg
C2RO ...00
0
Training Experiencos
Takking Eapel encEs
Traiiing Experieroo
Figure 14: Comparison of positive contributions episode reward during training for our approach (yellow) and I Lagrangian methods configured to have the same cost level. The plots for the other three environments are shown in Figure 6
PointGos12 Epecde Costs
CorButton! Epicode Corts
PaintPusht Epiodd Costs
2000
ProLagaggn
PrO-Lagaing
175
CPP0
Meat Sptok Core
Zaropraitty
Zerogeralty
300
Man
3t
50
CiFD fe- en) Berspualty
39
Trakiig Experinnce
Traving Experiences
Troking Experiences
CarGeell Episode
PaintBurene
CorFurh2
2006
145
roo-agaagay
177
0:00
rini (s - 12)
Her Edeade Cat
Cost
zamppeartty
235
Epoode
800
60
Lagarn CaPo 1171
Poollapegee CIFD (20)
20
Traiiitz Experiences
Trakiin Experiecos
Tralnic Experincce
Figure 15: Comparison of cost incurred (lower better) during training for our approach and Lagrangian methods configured to have the same cost level. As intended, cost levels are consistently matched between the methods. As above, the "zero-penalyy line refers to the level reached by PPO and TRPO trained with no penalty in the reward (Ray al. 2019)
17

---

Under review as a conference paper at ICLR 2022
PointGoat? Eplsode Costs
CorBucton? Episode Costs
PaintPudh Epicade Costs
2010
Mex Eperd Cent
340
240
Hawle yoth
Mou 830
300
CHO
Trakning Experience
Traking Experiences
Traking Experionce
CorGeol2 Episode Costs
PuionS Episode Costs
CarFush2 Epicode Costs
140
120
CPPO (s- 0.200
poantty
Hea Ejeck Con
393
CPo
LPC CIPO is k.70 conoeenee by
CSPd (y 670) dero-or alte
2
Trokning Expeeences
Trakiin Experiences
Trakning Expelences
Figure 16: Comparison of cost incurred (lower is better) during training for our method and Con- strained Policy Optimization (Achiam et al. 2017) configured to have a matching cost limit. Results are consistent with Ray al. (2019). As above, the "zero-penalty" line refers to the level reached by unconstrained PPO and TRPO trained with no penalty in the reward (Ray et al. (2019).
PainCool2 Epiaods Rowards
CarButtond Epiinde Ravards
PuintPuht Epindd Rewards
Hon Spiokk Roar
Mer Splodd
%
-10
Exprraeoos
CarGoal2 Episode Rewards
Pointiuttnn2 Lpioode Reverds
CarPush22 Episode
CPQ
CPD
Caro 1.71
Hean Epiodd Roord
T5
Mass Epoode Roard
to
Mov
(90
LSPO I4
4
Trannin Exprriec
Trsning Exseninccs
Traning Exprisoo
Figure 17: Comparison of average episode reward (including penalty) for C3PO and CPO.
A.8 SUPPLEMENTARY MATERIALS
The code used to produce these results is included in our Supplementary Materials.
18


Reviews:
[{"review_id": "w62l2_scnYo", "reviewer": "Reviewer_SaK7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths: \n\n- The idea of using CPT in context to risk-sensitive RL seems promising. I also believe the approach is novel to the best of my knowledge. \n\n- The paper is written clearly and with sufficient empirical rigour in the experiments conducted.\n\n### Concerns:\n\n- I think the clarity of some parts can be improved. A major component of the work is Eq 4, where the second transformation is not clear. Is it a property of the CDF or is it derived from the approach of Prashanth et al (2016.)? I think this is an integral component of the work and it should be clarified in more detail. \n\n- The authors claim that the expression in Eqn 11 has a lower variance than expression in Eqn 6 but there is no accompanying proof. A formal explanation of this claim will help to strengthen the work.\n\n- I also don’t understand why risk-sensitive DRL and constrained DRL approaches were excluded from the baselines. The comparison with PPO/TRPO demonstrates that the proposed approach can work better than the unconstrained methods. However, it is not clear what is the benefit of the proposed approach compared to the approaches in constrained RL literature. Some experiments with simple approaches like Lagrangian-PPO, CPO (which are both benchmarked and included with OpenAI gym) can help to make this distinction clear. Another possible method that can be included as a baseline is Cvar based objective (Chow et al 2015).\n\n\n\n### Minor comments:\n\nI think the notation for denoting CDF $P_{\\theta}$ is a bit confusing as it gives the impression of parameterized transition model. I would encourage the authors to explore different notations for this quantity.\n", "summary_of_the_paper": "The paper presents an alternate approach for distributional DRL via proposing an objective inspired from Cumulative Prospect Theory (CPT, Tversky and Kahneman, 1992). They use this distributional objective in conjunction with policy gradient methodology to propose a distribution policy gradient method for risk-sensitive RL. Under their approach, the distribution of the returns is optimized to maximize some chosen function of its CDF. They experiment with different such possible distributional objectives (risk profiles) on the OpenAI Safety Gym environments and show that their approach performs better than PPO. ", "summary_of_the_review": "I think the paper presents a really interesting approach to distributional DRL in context to risk-sensitive RL. Although there are some issues with the clarity, my biggest concern is that advantage of the proposed approach is not clear when compared to the existing literature on risk-sensitive and constrained RL. Although the paper could use more empirical strengthening, nevertheless, I think the will be useful to the community and hence recommend weak acceptance. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "USrbGj-kPG", "reviewer": "Reviewer_VVvx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The generalization to consider the entire CDF with arbitrary weightings appears to be a novel contribution to the literature. Furthermore, the authors also propose a simple to instantiate procedure for these weightings and a sample based algorithms that just sorts the trajectory outcomes. \n\nOther comments:\n* How robust are these results to the more common deep RL benchmarks outside of the safety gym? \n* Once the utility is assumed to additively decompose across each state, action is this not equivalent to simply defining the utility as the reward?\n* The variance reduction is basically the same as the usual policy gradient (and follow directly from the fact that the expectation of the gradient of log prob is 0).\n* In Algorithm 1, where is $\\hat{u}$ coming in? Is this just the sum over t of $u(s_{i,t})$? There seems to be some confusion over the dependence on both state and action or only state for the utility -- same issue between (14) and (15).\n* A $\\frac{1}{N}$ factor appears to be inconsistent across Eqs (11), (12) and (14)\n* It's not clear how important the particular choice of -0.025 for the negative reward is for the results or how it was chosen. In particular, this seems like it would have a complex relationship with the $\\eta$ weighting. \n* Figure 5 captions says 6 environments, but there is only three in the figure.", "summary_of_the_paper": "This paper considers a generalization of the policy gradient method to optimize for arbitrary utility functions with weightings that depend on the entire CDF (rather than the expected reward). This generalization has two aspects: (1) a utility function on top of the trajectory reward and (2) a weighting function for the CDF of the trajectory reward with respect to which the expectation is performed.  The paper derives an expression for the policy gradient and also generalizes the standard variance reduction baselines. Inspired by the PPO loss, the authors then propose a clipped version of the policy gradient calling it C3PO and evaluate this on some benchmarks from the OpenAI Safety Gym, where it is found that the conservative weightings can offer improvements over the standard formulation.", "summary_of_the_review": "The main technical contribution is a novel generalization of the policy gradient expression to consider CDF weightings and a sample based algorithm that weights sorted trajectories to compute the gradient. The variance reduction extensions are fairly trivial. The empirical evaluations are limited to the open AI safety gym and are conducted on a PPO inspired version of the policy gradient result. These look promising, but the role of an arbitrarily chosen reward modification makes it hard to interpret how much the results depend on this  and how robust they are.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UJPovbcsm7S", "reviewer": "Reviewer_ugec", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strenghts\nThe paper aims at solving an interesting problem, which is the optimization of a CPT inspired objective with a policy search approach. The authors derive a policy gradient for their objective and propose and estimator for it. They derive a PPO-like algorithm, which is then employed in the experiments.\nThe experimental analysis seems to be fair and sound: the experimental setting is described in a clear way, the experiments are repeated to take into account for the randomness in the weight initialization and the stochasticity of the performance is correctly represented in the plot with shaded areas, and by showing quantiles in the tables. The author included also the distribution of the returns, which helps understanding the impact of the risk-averse optimization.\n\n\n## Weaknesses\nWhile the author correctly described the state-of-the-art in the fields, some connections with related works are still not completely clear. For instance, it is not clear how much the proposed objective differs from distortion risk-measures (Wang, (1996), Dabney, (2018)). Furthermore, the author says that their estimation approach is similar to the one used in (LA Prashant, 2016), but it is not clear to which extent the developed techniques are different from the ones proposed there. It would help the reader to explicitly describe these connections with prior works, in order to ease the understanding of the authors contribution.\nFrom a methodological viewpoint, the policy gradient derivation lacks a clear separation between the exact gradient formula and its estimator. Moreover, the authors do not show that expression (6) represents an unbiased estimate of the gradient, as they claim. Results in (LA Prashant, 2016) seems to suggest that the estimate would be instead only consistent, hence, asymptotically unbiased.\nFor what concerns the experimental analysis, the authors focus mainly on evaluating their results under the average reward objective. This is not fair in principle, since the proposed approach is actually optimizing a different objective. It would be interesting also to see how the algorithm performs w.r.t. the objective it is optimizing, including its learning curves. Finally authors suggest that the positive effect due to risk-averse optimization is similar to what happens with prioritized replay. However, while the latter technique gives an in higher weight to errors in value function estimation (high TD-errors), risk-averse optimization gives an higher weight to actual lower returns, thus, their objectives are substantally different.", "summary_of_the_paper": "The article propose a policy gradient method for optimizing a CDF based criterion, inspired by CPT. \nBy varying the weighting function inside the objective, it is possible to change the risk-aversion of the agent.\nThe authors derives the policy gradient for the aforementioned objective and propose an estimation technique for it.\nThen, they propose an algorithm which extends PPO for optimizing their objective. Empirical analysis is carried on to evaluate the approach on some modified Safety Gym environments, in which a fixed negative rewards corresponded to adverse events.\nThe authors evaluate different objectives obtained by employing a Wang weighting function, with different values of the parameter $\\eta$. They show that optimizing a cautious (or risk-averse) objective allow to obtain better results in terms of average reward w.r.t. optimizing an aggressive (or risk-seeking) one. By further exploring the parameter space, the author demonstrate that some risk-averse values of the parameters allow to outperform also the risk-neutral version of PPO w.r.t. the average reward objective.", "summary_of_the_review": "While the paper analyses an interesting problem and its experimental analysis is sound, it is not clear how novel its derivation is. Moreover, the author did not provide a clear picture on the policy gradient method they propose, for which further proofs should be included. The main claim of the authors, that some degree of risk-aversion may help risk-neutral optimization, is not novel from an empirical point of view, but it is still largely unexplained from the theoretical one, for which, unfortunately, no contribution is given in this work.\nDue to a balance between the aforemetioned weaknesses and strenghts I suggest a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gr2--ai0Ysw", "reviewer": "Reviewer_apyr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\nThe authors consider an interesting problem. I believe that risk awareness is important in RL. The utilization of a different risk measure, other than the one already explored, can be beneficial to obtain better risk awareness of the learning agent. \n\nWEAKNESSES.\n\nThe paper is difficult to follow. Mathematical notation lacks precision (for example it is unclear whether the authors consider a finite horizon, or an episodic or a continuing problem in Equation 1). Most of all, concepts related to risk measures should be better explained. When one looks at the \"CPT objective\" at the beginning of page 3, it is unclear which roles play the utility function and the weighting. I do understand that probably the authors are very familiar with the underlying theory, but most people in our field (Rl) might be not. Would be nice to give an intuition.\nFurther, many passages seem unnecessary, or they are poorly explained. \n\nExample: why do the authors bother giving us two different forms of $\\nabla_\\theta P_\\theta$ in Equations 4 and 5? And why the same happen in 7? Is there any utility in showing these two different versions of the same quantity?\n\nAlso, it is not clear to me whether Equation 6 is an unbiased estimator of Equation 3. Can the authors give an intuitive explanation of why $\\omega'(i/N)$, and can they show the unbiasedness of such estimator? \n\nSection 3.3 starts with \"unfortunately ... (6) suffers from high variance\". Is this due to the log-likelihood derivative (as in classic REINFORCE) or there is an additional motivation caused by the new objective?\n\nDerivation in (9) is known by most of the people in our field. The authors can just cite classic works on baseline subtraction. \nFurther, starting from Equation 9, and in many equations that follow, $\\mathrm{d}$ is missing in the integrals. \n\nI am not convinced that \n\n$$\n\\left(\\omega '\\left(\\frac{i}{n}\\right) + \\omega '\\left(\\frac{i-1}{n}\\right)\\right)\n$$\n\ncan be pulled out of the expectation $\\mathbb{E_{\\tau_i}}$. Can the authors clarify the passages in 13?\n\nWhy the approach in (14) should minimize the variance? While subtracting the value function $V$ (or utility function, here), is a standard technique, remains unclear whether with the influence of \n\n$$\n\\left(\\omega'\\left(\\frac{i}{n}\\right) + \\omega'\\left(\\frac{i-1}{n}\\right)\\right)\n$$\n\nstill makes $V$ a proper baseline subtraction.\n\nFurther, the usage of the utility function $u$ is unclear. In Equation 3, the utility takes as input a scalar function (the reward), i.e., $u(r(\\tau))$. It is unclear how the per-step utility $u(\\mathbf{s}, \\mathbf{a})$ in Equation 14 is defined, and it is also unclear how $u(\\mathbf{s})$ in Equation 15 is defined. \n\nCan the authors explain the usage of the euclidean norm in Equation 15? The value function and the utility are scalars, why do we need that? Did the author mean to write something like\n\n$$\n\\mathcal{L}(\\phi) = \\sum_{i,t} \\mathbb{E}\\left[V_\\phi(\\mathbf{s}_{i, t}) - \\sum_{t'=t}^{T_i} u(\\mathbf{s}_{i,t'})\\right]?\n$$\n\nIt would be nice to see the derivation of the value function and the advantage in this new setting of the risk measures, for example introducing a risk-aware Bellman equation. It is otherwise difficult to understand why the advantage \"is computed with the standard GAE except with per-step utilities in places of rewards\". The correctness of this approach might be trivial for the authors, but not for me. \n\nIn Section 4, could the authors clarify what ar the robots \"Point\" and \"Car\", and what are the tasks \"Goal\", \"Push\" and \"Button\"? It would be nice to know what are these tasks about, and the dimensionality of action and state spaces.\n\nOne thing that I think was really not clear, is what should I expect from the experiments. What should be the idea of using the CPT objective instead of other measures? How this should help?\n\nI have seen that making the agent risk-aware can lead to better performances, as highlighted in your experiments, but shouldn't this effect appear also with other risk measures (like VaR or CVaR)? Could the author clarify this point? I think that a comparison with other risk measures would have been beneficial. \n\nFurther, uncertainty comes both as an aleatoric component (stochasticity of the environment) and epistemic (trust in my own model). Looks like that this work is focused on the aleatoric component (which is fine). Do the authors plan to extend their work also to the epistemic uncertainty?\n\n", "summary_of_the_paper": "Risk objectives have long been investigated in reinforcement learning (RL). Most of the focus has been on classic risk measures, like exponential utility, value-at-risk (VaR), conditional value-at-risk (CVaR), leaving out, however, the cumulative prospect theory (CPT) developed by Tversky and Nobel Prize Kahneman in 1992, which has not yet been considered.\n\nThe advantage of CPT is to better model human decision-making, still allowing a wide class of risk measures, based on the utility $u$ and weighting $\\omega$.\n\nHence, the authors consider a new risk-aware objective. Following some derivation, they compute a sample-based estimation of the gradient of this new objective w.r.t. the policy parameters.\n\nThe authors propose a PPO-like algorithm (called C3PO), which incorporates the new, risk-aware, gradient estimator.\n\nThey perform an empirical analysis on some tasks of \"Safety Gym\", showing that proper risk-awareness helps increase the performance of classic PPO.", "summary_of_the_review": "STRENGHT:\n\nThe authors propose a new policy gradient algorithm based on a risk measure that has not been considered in RL so far. The considered risk-sensitivity measure comes from the cumulative prospect theory and aims both to mimic human decision making, and to generalizes a large class of other risk measures.\n\nI believe that risk-aware RL is important, as intelligent beings rarely optimize for the \"average\" case, but they often act optimistically or pessimistically, usually giving higher weights to rare events.\n\nWEAKNESSES:\n\nThe paper lacks a proper background, clarity, and precision. The mathematical notation should be improved, and while some passages could be skipped, since they are widely known by most of the audience, some passages instead remain obscure and need further clarification. I think that some passages are wrong, and I am waiting for the authors' response.\n\nThe main objective of the paper remains unclear. What is the benefit of utilizing this method in RL? Why are other risk measures (like VaR or CVaR) not enough? \n\nThe authors did not compare their risk measures with others, which I think was instead necessary. \n\nUPDATE\n---------\n\nAs the author improved the clarity of their submission and clarified some of my doubts (for example explaining the lack of comparison with VaR & CVaR), I am raising the score from 3 to 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new approach for risk sensitive RL by using an objective that depends on the full distribution and can apply a weight to the resulting trajectory. The reviewers thought that focusing on more general and expressive objectives for RL is well motivated. However, they had a number of concerns of the current paper state, including its clarity in a number of sections and its relation to other work in risk-sensitive RL. The authors provided thoughtful responses but some concerns lingered around the prior concerns.
Decision: Reject