An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient
Authors: Yudong Luo
Guiliang Liu
Pascal Poupart
Yangchen Pan
Venue: NeurIPS 2023 poster (2023)
Keywords: risk-averse RL
mean-variance RL
TL;DR: We propose an alternative risk measure, Gini deviation, as a substitute for variance, and derive a policy gradient algorithm to minimize it.

Abstract: Restricting the variance of a policy’s return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient
Yangchen Pan University of Waterloo, "The Chinese University of Hong Kong, Shenzhen, Univeesiity of Oxford, Veector Institute yudong wwo@wateeroo. ca, liuguiliang@cuii edu. en, ppoupart@wwaterloo.. ca, yangchen, paneeng. ox ac. uk
Yudong Luol-4 Guiliang Liu?, Pascal Poupart
Abstract
Restricting the variance of policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as proxy. We thoroughly examine the limitations of these variance-basee methods, such sensitivity numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive policy gradient algorithm to minimize it. Empirical evaluation domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance- aariance-based risk measures and achieves high return with low risk in terms variance and Gini deviation when others fail to learn reasonable policy.
Introduction
The demand for avoiding risks in practical applications has inspired risk-averse reinforcement learning (RARL). For example, we want to avoid collisions in autonomous driving [ or avoid huge financial losses in portfolio management [2] In addition to conventional RI,, which finds policies to maximize the expected return 3, RARI also considers the control of risk.
Many risk measures have been studied for RARL, for instance, exponential utility functions [ value at risk (VaR) (5). conditional value at risk (CVaR) (6) 7 and variance [8, (.. this paper, we mainly focus the variance-related risk measures given their popularity, variance has advantages in interpretability and computation [C M.]. Such paradigm referred to as mean-variance RL. Traditional mean-vane variance RL methods consider the variance of the total return random variable. Usually, the total return variance treated as constraint to the RL problem, i.e., it lower than some threshold g 9 12 Recently, [13] proposed seward-volatility risk measure, which considers the variance of the per-step reward random variable. [13] shows that the per-step reward variance is an upper bound of the total return variance and can better capture the short-term risk. [14] further simplifies [13's method by introducing Fenchel duality.
Directly optimizing total return variance is challenging. It either necessitates double sampling [8] or calls for other techniques to avoid double sampling for faster learning [8] 9 [2]. As for the reward-volatility risk measure, [[3] uses complicated trust region optimization due to the modified reward's ' policy-dependent issue. [4] overcomes this issue by modifying the reward according to Fenchel duality. However, this reward modification strategy can possibly hinder policy learning by changing a "good" reward to a "bad" one, which we discuss detail in this work.
To overcome the limitations of variance-based risk measures, propose to new risk measure: Gini deviation (GD). We first review the background of mean-variance RL. Partieularly, we explain the limitations of both total return variance and per-step reward variance risk measures. We then introduce
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GD as a dispersion measure for random variables and highlight its properties for utilizing it as a risk measure in policy gradient methods. Since computing the gradient using the original definition of GD is challenging, we derive the policy gradient algorithm from its quantile representation to minimize it. To demonstrate the effectiveness of our method in overcoming the limitations of variance-based risk measures, we modify several domains (Guarded Maze 77]. Lunar Lander [55 Mujoco (16] where risk-aversion can be clearly verified. We show that our method can learn risk-averse policy with high return and low risk in terms of variance and GD, when others fail to learn reasonable policy.
2 Mean-Variane
Reinforcement Learning
In standard RL settings, agent-environment interactions are modeled as a Markov decision process (MDP), represented as tuple (S, R, P, Mo:") [17] S and A denote state and action spaces. P(-|s. a) defines the transition. R is the state and action dependent reward variable, Mu is the initial state distribution, and y E (0, 1] is the discount factor. An agent follows its policy т S A - +00). The return at time step is defined as G. = >i= 7 R(SS+ Ar41). Thus, Go is the random variable indicating the total return starting from the initial state following
Mean-varian variance RL aims to maximize T[Go) and additionally minimize its variance V V.G. 12 Generally, there are two ways to define variance- risk. The first one defines the variance based on the Monte Carlo total return Go. The second defines the variance on the per-step reward R. We review these methods and their limitations in the following subsections. We will refer to R, Ta and 0 interchangeably throughout the paper when the context is clear.
2.1 Total Return Variance
Methods proposed by g consider the problem
max TE, Go], s.t. V[Go] S
where & indicates the user's tolerance of the variance. Using the Lagrangian relaxation procedure [8] we can transform it to the following unconstrained optimization problem: max, E[Go] - AV[ hyper-parameter.
where is a trade-off hyper- parameter. Note that the mean- variance objective is in general NP-
hard [99 to optimize. The main reason is that although variance satisfies a Bellman equation, it lacks the monotonicity of dynamic programming (20].
Double Sampling in total return variance. We first show how to solve unconstrained mean- mean-variance RL via vanilla stochastic gradient. Suppose the policy is parameterized by 6, define J(0) = E, [G.] and M == E. [(Ero 7 R(S,, A= then V[Ga] = M() - J²())- The unconstrained mean- variance objective is equivalent to (+) J(Ø) - x(M(0) - -²(0)), whose gradient is
(o,) voJ AVo(((e)) (²()))
(2)
= (O.) -
(3)
The unbiased estimates for (o- and M)) can be estimated by approximating the expectations over trajectories by using a single set of trajectories, i.e., VoJ = E. (0) and VoM =
E. [R[,,, (0)], where R, is the return of trajectory and w, (f) -0)=30 E. V log (a (st. In contrast, computing an unbiased estimate for aO) requires two distinct sets of trajectories to estimate (0) and (0) separately, which is known as double sampling.
Some work claims that double sampling cannot be implemented without having access to a generative model of the environment that allows users to sample at least two next states [12]. This is, however, not an issue in our setting where we allow sampling multiple trajectories. As long as we get enough trajectories, estimating J [() V (P) is possible.
Remark.
Still, different methods were proposed to avoid this double sampling for faster learning. Specif- ically, [8 considers the setting = and considers an unconstrained problem: maxØ (0) =
E[Ga] - \V[V[ - E). where A > 0 is a tunable hyper-parameter, and penalty function g(x) = (max(0, xx) This method produces faster estimates for E[Ga] and V[Gol and a slower updating for at each episode, which yields two-time scale algorithm. 9 consid-
ers the setting < 1 and converts Formula [ into an unconstrained saddle-point problem: max, ming A) = -E + X(V[Gn) E). where À is the dual variable. This approach uses perturbation method and smoothed function method to compute the gradient of value functions with respect to policy parameters. [22 considers the setting 7 = 1, and introduces Fenchel duality x² = max, (2.y y²2 to avoid the term ( v in the gradient. The original problem is then transformed into maxe,y La La(0, y) 2y (E[Go) + +x) y² -EGG)) where y is the dual variable.

---

Limitations of Total Return Variance. The presence of the square term R in the mean-variame (0)) makes the gradient estimate sensitive the numerical scale of the return, empirically verified later. This issue is inherent in all methods that require computing Vo [ Users can not simply scale the reward by a small factor to reduce the magnitude of R2, since when scaling reward by factor E[Go] is scaled by but V[Go] is scaled by e². Consequently, scaling the reward may lead to different optimal policies being obtained.
gradient V,M((B)
2.2 Per-step Reward Variance
A recent perspective uses per-step reward variance V[R] as proxy for V[Go). The probability mass function of R Pr R = 2) E.,a dz (s, a) ().))( where is the indicator function, and
(=(8, = (1 - 7) E 7 Pr((S = 8, A, = ar P) is the normalized discounted state-action
distribution. Then we have E[R (1 -1-17 -- and V[Go] S (I-7)²
viRl
(see Lemma of [3] Thus,
13
considers the following objective
(x) B[M] l B[R X(R B[R))*]
(4)
This objective can be cast as a risk- problem in the original MDP, but with a new reward function f(ss, u) r(x,a) a(r(s,a) - (1 7) (G,o)) However, this n(s, a) is nonstationary
due to the occurrence of E[Go]. so standard risk-ncutral RL algorithms cannot be directly applied. Instead, this method uses trust region optimization [21 to solve.
14 introduces Fenchel duality to Equation 4 The transformed objective is (A( B[R] XE[ + À max, (2E max,(22[[Rjg RRly y²), which equals to max,,y Jx(T,, y) E... as
Ar(s, a)² 2xr(s, a)y) - Any.. The dual variable y and policy ι are updated iteratively. In each inner loop k; y has analytical solution yk++1 Ex,o dzy (s, a)r(s, a) = (1 - FF, [Go] since is quadratic for y. After y is updated, learning π risk-neutral problem the original MDP, but with a new modified reward
f(s,a) = r(s, a) Ar(s, a)² + a)ye44
(5)
Since f(8,a) is now stationary, any risk-neutral RL algorithms can be applied for policy updating.
Limitations of Per-step Reward Variance. 1) V[R] is not an appropriate surrogate for V[Go] due to fundamentally different implications. Consider a simple example. Suppose the policy, the transition dynamics and the rewards are all deterministic, then V[Ga] = 0 while [A] is usually equal. In this case, shifting specific step reward by
nonzero unless all the per-step rewards constant will not affect V[G,) and should not alter the optimal risk-averse policy. However, such shift can lead to a big difference for V[R] and may result in invalid policy we demonstrated in later example. 2) Reward modification hinders policy learning, Since the reward modifications. in [13] (Equation 4) and [14] (Equation 5 share the same issue, here we take Equation 5 an example. This modification is likely to convert a positive reward to a much smaller or even negative value due to the square term, i.c. -Ar(s, aa². In addition, the beginning of the learning phase, when the policy performance is not good, is likely to be negative in some environments (since y relates to B[Gol). Thus, the third term 2Ar(s, a)y decreases the reward value even more. This prevents the agent to visit the good (i.e., rewarding) state even if that state does not contribute any risk. These two limitations raise a great challenge to subtly choose the value for À and design the reward for the environment.
Empirical demonstration of the limitations. Consider a maze problem (a modified version of Guarded Maze (7)) in Figure [ Starting from the bottom left corner, the agent aims to reach the green goal state. The gray color corresponds to walls. The rewards for all states are deter- ministic (i.e., 1) except for the red state whose reward is a categorical distribution with mean 11 The reward for visiting the goal is positive constant value. To reach the goal, risk-neutral agent prefers the path at the bottom that goes through the red state, but V[Gu] will be nonzero. A risk- risk-averse agent prefers the white path in the figure even though E[G,] slightly lower, but V[Ga] 0. Per-step reward variance methods aim to use V[R] as a proxy of V[Go]. For the risk-averse policy leading to the white path, ideally, increasing the goal reward by a constant will not
Figure I:
A modified
Guarded Maze 77 Red state returns an uncertain reward (details in text).
effect V[Gn). but will make big difference to V[R]. For instance, when the goal reward 10, V[R]- = 10. When goal reward 20, V(R) R 36.4, which is much more risk-averse Next, consider the reward modification (Equation 5) for the goal
3

---

reward when it is 20. The square term in Equation is - 400X. It is very easy to make the goal reward negative even for small X, e.g., 0.1. We do find this reward modification prevents the agent from reaching the goal in our experiments.
3 Gini Deviation as an Alternative of Variance
To avoid the limitations of Gl and V[R] we have discussed, in this paper, we propose to use Gini deviation an alternative of variance. Also, since GD has similar definition and similar properties as variance, it serves as a more reasonable proxy of V[Go] compared to V[R].
3.1 Gini Deviation: Definition and Properties
GD [22] also known as Gini mean difference or mean absolute difference, is defined as follows. For random variable X. let X1 and X2 be two i.i.d. copies of X, i..., X1 and X₂ are independent and follow the same distribution as X. Then GD given by
[
F X2]
(6)
Variance can be defined in a similar way as V[X] = [ - X2
Given samples {xt ]= from Xı and { from X2. The unbiased empirical estimations for GD 1 2n2 Ez (== xx3]?
and variance are ID 2n E Σ | - =31 and V[x]
Both risk profiles aim to measure the variability of a random variable and share similar properties [2] For example, they are both location invariant, and can be presented as weighted sum of order statistics. [23 argues that the GD is superior to the variance as a measure of variability for distributions far from Gaussian. We refer readers this paper for full overview. Here we highlight two properties of [X] to help interpret it. Let M denote the set of real random variables and let MP, E [1, 20) denote the set of random variables whose probability measures have finite p-th moment, then
V[X] 2 v3 D[X] for all X E M
. ID[cX] = cD[ for all c > 0 and X E.
The first property is known as Glasser's inequality (24). which shows ID [X[] is a lower bound of V[X] X has finite second moment. The second one is known as positive homogeneity in coherent measures of variability (25), and is also clear from the definition of GD in Equation 6 In RL, considering X is the return variable, this means GD is less sensitive to the reward scale compared to variance, i.e., scaling the return will scale [[X] linearly, but quadratically for V[X]. We also provide an intuition of the relation between GD and variance from the perspective of convex order, as shown in Appendix 7 Note also that while variance and GD are both measures of variability, GD a coheren! measure of variability (5).. Appendix provides a discussion the properties of coherent measures of variability. while explaining the differences with coherent measures of risk such as conditional value at risk (CVaR).
3.2 Signed Choquet Integral for Gini Deviation
This section introduces the concept of signed Choquet integral, which provides an alternative defini- tion of GD and makes muddient-bssed optimization convenient. Note that with the original definition (Equation 6). it can be intractable to compute the gradient w.r.t. the parameters of random variable's density function through its GD.
The Choquet integral [26] was first used in statistical mechanics and potential theory and was later applied to decision making as way of measuring the expected utility (27). The signed Choquet integral belongs to the Choquet integral family and is defined as:
Definition 1 (28). Equation 1). A signed Choquet integral by, X R. L is defined as
xx) = [- (1(Pp(x 2 =)) I(1)) due 1. M(Pr(X x))de
(7)
where _ is the set of bounded random variables in a probability space, is the distortion funcitoon and h E H such that 7 (h: [0, R, h()) 0, h is of bounded variation).
This integral has become the building block of law-invariant risk measures after the work of 29 30 One reason for why signed Choquet integral is of interest to the risk research community is that it
'Law-inwe invariant property is one of the popular "financially reasonable" axioms. If functional returns the same value for two random variables with the same distribution. then the functional is called aw-iivariiant

---

is not necessarily monotone. Since most practical measures of variability are not monotone, e.g.. variance, standard deviation, or deviation measures [T]] is possible represent these measures in terms of 4 by choosing specific distortion function h.
Lemma (128). Section 2.6). Gini deviation is signed Choquet integral with a concave h given by 1(()= = -a a, 1].
This Lemma provides an alternative definition for GD, i... D[X] = [ (Pr(X 2 x)) h(a) = -a 2 +a. However, this integral is still not easy compute. Here turn to its quantile representation for easy calculation.
Lemma 2 ((28), Lemma 3). (X) has a quantile representation. If Fx is continuous, then tn (x) - 5 Fx a d))))))). where Fxt is the quantile function (inverse CDF) ofX.
Combining Lemma and
TD XJ can be computed alternatively as
/. ()-a((((()))
[[X] t (X)
Fxx (a) 1)da
With this quantile representation of GD, problem in the next section. It should be noted that variance cannot be directly defined by a - quantile representation, but complicated related representation: V[X] SuPheи { (x)(X)
derive policy gradient method for new learning
} |1 where | 1 So (h'(p)) if h is continuous, and ||h'| co if it is not continuous (see Example 2.2 of (32 Hence, such representation of the conventional variance measure not readily usable for optimization.
Policy Gradient for Mean-Gini Deviation
In this section, we consider a new learning problem by replacing the variance with GD. Specifically, we consider the following objective
max E[Go] \D
(9)
where À is the trade-off parameter. To maximize this objective, we may update the policy towards the gradient ascent direction. Computing the gradient for the first term has been widely studied in resk-neutral RL (3). Computing the gradient for the second term may be difficult at the first glance from its original definition, however, it becomes possible via its quantile representation (Equation B).
4.1
Gini Deviation Gradient Formula
We first give general gradient calculation for GD of a random variable Z, whose distribution function parameterized by 0. In RL, we can interpret 0 as the policy parameters, and Z as the return under that policy, i.e., Go. Denote the Probability Density Function (PDF) of Z as fz 0).
Given a confidence level a E (0, 1), the levevel quantile of is denoted as qa (Z:0). and given by
Ja Z FZ,
inf Pr(Ze S 2) a
(10)
For technical convenience, we make the following assumptions, which are also realistic in RL.
Assumption 1. 7 is continuous random variable, and bounded in range [-b, b) for all
Assumption 2. a
0e, 9a (Z; 0) exists and is bounded for all 6, where 0g is the i-th element of 0.
Assumption 3. 0f2(=:0) fzz 0) exists and is bounded for all 0, z. 0, is the i-th element of t. 80,
Since Z is continuous. the second assumption is satisfied whenever assumptions are common in likelihood-raid methods, e.g., see (33] Relaxing these assumptions is possible but would complicate the presentation.
00; [2; 0) is bounded. These
0
Proposition 1. Let Assumptions
7
2 hold. Then
[Vo loloo f2(;)) J. (t)
VoD Zo]
(11)
Proof. By Equation g the gradient of [D[2]] = (Zo) (h((a) = -a² a, E i0,
VoD Zo] = V (Zo) / (2a 1) (a)aa = I (2a (2a- (2)))))oo(Z;;)dde
(12)
This requires to calculate the gradient for any a-level quantile of Ze. i.c. Vz Based on the
5

---

assumptions and the definition of the a-level quantile, we have 1_ fz(z;0)ds derivative and using the Leibniz rule we obtain
Taking
9n (2:9)
[z 0)zz = /_ Vof2(=;0)dz Vow
(2.0)
0 V
(Z:
(13)
Rearranging the term, we get Voga (Z;0) Sh ging back to Equation 12 gives an intermediate version of V [zl)
(2
V Vol2(z;0)dz (f:
Plug-
V.D[Ke] L. (2a (2a-1)
(ul(2:0)
Vefz(z;0)d= [fz (2:
da
(14)
By switching the integral order of Equation 14 and applying Va log 1 Vor, we get the final gradient formula Equation [1] The full calculation is in Appendix 8.1
4.2 Gini Deviation Policy Gradient via Sampling
In typical application, Z in Section 4.1
would correspond to the performance of system, e.g., the
total return Go in RI. Note that in order to compute Equation 11 one needs access to Vo log [z 0): the sensitivity of the system performance to the parameters 0. Usually, the system performance is a complicated function and calculating its probability distribution is intractable. However, in RI., the performance a function of trajectories. The sensitivity of the trajectory distribution is often easy to compute. This naturally suggests sampling based algorithm for gradient estimation.
Now consider Equation
I| in the context of RL, i.e., Z = Go and 0 is the policy parameter.
veD[Ga] Eg [V log fco (g: f"" (2Fc. (1) -1dd
(15)
To sample from the total return variable Go. we need to sample trajectory T from the environment by executing To and then compute its corresponding return R2
+ 7T2 + + > -rrrr. where re is the per-step reward at time t, and T is the trajectory length. The probability of the sampled return can be calculated as fGo (R,; 0) = (o(5)) 1I:- l 11
The gradient of
mo(sa) IT= [mo[ag] (oo(aa):))((=::) (()t+ 18,, (8),,))),, since the difference in transition probability does not alter the policy gradient. It is well known that Ve log P(r Ef Ve log (e(a)(s)).
its log-likelihood is the same that of P(r
For the integral part of Equation [5] it requires the knowledge of the CDF of Go- In practice, this means we should obtain the full value distribution of Go. which is usually not easy. One common approach to acquire an empirical CDF or quantile function (inverse CDF) is get the quantile samples of distribution and then apply some reparameterization mechanism. For instance, reparameterizatioo is widely used in distributional RL for quantile function estimation. The quantile function has been parameterized as a step function (34] 355. a piece-wise linear function (36), or other higher order spline functions (37. In this paper, use the step function parameterization given its simplicity. To do so, suppose we have trajectory samples {Tr from the environment and their corresponding returns (R. the returns are sorted in ascending order such that Rr, S R. S ... S RT,, then each R., regarded as a quantile value of Go corresponding to the quantile level a; ] [== + 1 i.c. we assume qo, (Go; 0)- R., This strategy is also common in distributional RL, c.g., see 134 38] The largest return Rr,, regarded as the upper bound in Equation 15
Thus, given ordered trajectory samples { 1= an empirical estimation for GD policy gradient is (a detailed example is given in Appendix 8.2)
2j (Rrg4. Ro,,) (R... R..) Σ j=
n-1
Va log (3(..,))))))) where 1/s
(16)
1=0
The sampled trajectories can be used to estimate the gradient for B[ in the meantime, e.g., the well known vanilla policy gradient (VPG), which has the form E, [R, -T- TT V log To (arls:) It og), which
T- is more often used as E, E Va log To (agls))n* ( where dt Er= - is known to have lower variance. Usually 9t is further subtracted by a value function to improve stability, called REINFORCE with baseline. Apart from VPG, another choice to maximize G is using PPO [39]
4.3 Incorporating Importance Sampling
For on-policy policy gradient, samples are abandoned once the policy is updated, which is expensive for our gradient calculation since we are required to sample n trajectories each time. To improve

---

the sample efficiency to certain degree, we incorporate importance sampling (IS) to reuse samples for multiple updates in each loop. For each T;. the IS ratio is p. IIr (T(0,, (aal,))///(allllsi. (s.e). where i is the old policy parameter when {T) are sampled. Suppose the policy gradient for maximizing B[Go is REINFORCE baseline. With IS, the empirical mean-GD policy gradient is
ΣΣ P
Va log (oggo V(s..)) +
PiT
V log (anee)))))) (17)
n1
where g is the sum of rewards-to-go as defined above. (ss,,)) is the value function. The first part can also be replaced by PPO-Clip policy gradient. Then we have
n IE
V min To(analose) Ac,,, [(e, A..)) + (g(ar,)(s,1)
POR
Vo log (o(ar))ss,) (18)
1=0
where Ai, is the advantage estimate, and f() is the clip function in PPO with being the clip range. i.c. f(e, A..) = clip( al(a))
Ta(as,alEL))" c, c)
The extreme IS values Pi will introduce high variance to the policy gradient. To stabilize learning. one strategy is that in each training loop, we only select T; whose P lies in [1- 5, 1 + 6]. where 8 controls the range. The updating is terminated if the chosen sample size is lower than some threshold, e.g.. 3 8.n, 3 E (0. 1). Another strategy is to directly clip p; by a constant value Ç. i.c., P. = min(p {), e.g., see 40 In experiments,
use the first strategy for Equation [7 and the second for Equation 18 We leave other techniques for variance reduction of IS for future study. The full
algorithm that combines GD with REINFORCE and PPO is in Appendix
5 Experiments
Our experiments were designed to serve two main purposes. First, we investigate whether the GD policy gradient approach could successfully discover risk-averse policies in scenarios where variance-based methods tend to fail. To accomplish this, we manipulated reward choices to assess the ability of the GD policy gradient to navigate risk-averse behavior. Second, we sought to verify the effectiveness of our algorithm in identifying risk-averse policies that have practical significance in both discrete and continuous domains. We aimed demonstrate its ability generate meaningful risk-averse policies that are applicable and valuable in practical settings.
Baselines. We compare our method with the original mean-variance policy gradient (Equation 2 denoted as MVO). Tamar's method [8] (denoted as Tamar), MVP [12), and MVPI [14. Specifically ly,
MVO requires multiple trajectories compute o.))), We use 2 trajectories to estimate (0)
and another 2 to estimate (O), where n is the sample size. MVPI is a general framework for
policy iteration whose inner risk-neutral RL solver is not specified. For the environment with discrete actions, we build MVPI on top of Q-Learning or DQN 41. For continuous action environments, MVPI is built on top of TD3 22 in 14. We use REINFORCE represent the REINFORCE with baseline method. We use MG as a shorthand of mean-GD to represent our method. In each domain, we ensure each method's 's policy or value nets have the same neural network architecture
For policy updating, MVO and MG collect n episodes before updating the policy. In contrast, Tamar and MVP update the policy after each episode. Non-tabular MVPI updates the policy each environment step. In hyperparameter search, we use the parameter search range in MVPI [4] as reference, making reasonable refinements find an optimal parameter setting. Please refer to Appendix [0 for any missing implementation details. Code is available
5.1 Tabular case: Modified Guarded Maze Problem
This domain is a modified Guarded Maze (7] that previously described in Section 2.2 The original Guarded Maze is asymmetric with two openings to reach the top path (in contrast to a single opening for the bottom path). In addition, paths via the top tend to be longer than paths via the bottom. We modified the maze to be more symmetric in order to reduce preferences arising from certain exploration strategies that might be biased towards shorter paths or greater openings, which may confound risk aversion. Every movement before reaching the goal receives a reward of -1 except moving to the red state, where the reward is sampled from {-15, -1, 13) with probability {0.4. 0.2, 0.4} (mean is -1) respectively. The maximum episode length is 100. MVO and MG collect n 50 episodes before updating the policy, Agents are tested for 10 episodes per evaluation.
7

---

Mazo
Moza, geal m22
Mazo. goal 4
Optinall risccversioon rale
0.8
Optnaal noc aversoo rabe
MVD
NUT
0.6
MVD
MVO
Retum
Tamar
Tamar
Tamar
MVP
0.4
MVP
0.4
MVP
MG
MG
MG
0.2
100
0.0
0.0
25
00
100
125
100
175
200
25
75
100
125
100
176
200
26
60
75
100
100
175
200
Episodes/1kk)
Episodes(11 (1R)
Episodes[1K)
Figure 2: (a) Policy evaluation return and (b,c) optimal risk-aversion rate training episodes in Maze. Curves are averaged over 10 seeds with shaded regions indicating standard error.. For optimal risk-aversion rate, higher is better.
The failure of variance-based baselines under simple reward manipulation. We first set the goal reward 20. Here, we report the optimal risk-aversion rate achieved during training. Specifically, we measure the percentage of episodes that obtained the optimal risk-averse path, represented by the white color path in Figure ! out of all completed episodes up the current stage of training.
Notice that MVO performs well this domain when using double sampling to estimate its gradient. Then increase the goal reward 40. This manipulation does affect the return variance of the optimal risk-averse policy, since the reward is deterministic, However, the performances of MVO, Tamar, MVP all decrease, since they more sensitive the numerical scale of the return (due to the E[G term introduced by variance). MVPI is a policy iteration method in this problem, whose learning curve is not intuitive show. It finds the optimal risk-averse path when the goal reward is 20, but it fails when the goal reward is 40, An analysis for MVPI is given in Appendix 10.2.2 We compare the sensitivity of different methods with respect À in Appendix 10.2.4
Remark. Scaling rewards by a small factor is not an appropriate approach to make algorithms less sensitive to the numerical scale for both total return variance and per-step reward variance. since it changes the original mean- mean-variane variance objective both cases.
5.2 Discrete control: LunarLander
This domain is taken from OpenAl Gym Box2D environ- ments [5]. We refer readers to its official documents for the full description. Originally, the agent is awarded 100 if it comes to rest. We divide the ground into two parts by the middle line of the landing pad, as shown in Figure 10 in Appendix. If the agent lands in the right area, an additional noisy reward sam- pled from from.A((.. (O, 1) times 90 is given. A risk-averse agent should learn to land at the left side as much as possible. We include baseline to demonstrate the risk-aversion of our algorithm. REINFORCE, MVO and MG collect = 30 episodes before updating their policies. Agents are tested for 10 episodes per evaluation.
LunarLander
Return
REINFORCE
MVG
-000
Tambe
AAKR
-400
MG
MVPI
REINFORCE
-800
120
0.8
REINFORCE
MvP
Landing loft rate
0.8
Até
We report the rate at which different methods land on the left in FigureBIb (we omit the failed methods), i.e, the percentage of episodes successfully landing the left per evaluation. MVO, Tamar, and MVP do not learn reasonable policies in this domain according their performances in Figure 3 a). MVP learns to land in the middle of the learning phase, but soon after fails to land. Since successfully landing results in a large return (suc- cess reward is 100). the return square term (EE(G)) introduced by variance makes MVP unstable. MVPI also fails land since V[R] is sensitive to the numerical scale of rewards. In this domain, the success reward much larger than other reward values. Furthermore, reward modification in MVPI turns large success rewards into negative values, which prevents the agent from landing on the ground. MG achieves comparable return with REINFORCE, but clearly learns risk-averse policy by landing more on the left.
0.4
0.2
(b)
0.0
20
120
Episodes(1k]
Figure 3: (a) Policy evaluation return and (b) left-landing rate (i.c., risk-averse landing rate) v.s. training episodes in LunarLander. Curves are averaged over 10 seeds with shaded regions indicating stan- dard errors. For landing left rate, higher is better.

---

InvertedPendulum
HalfCheetah
Swimmer
500 (8)
(c)
PPO
(6)
MVO
400
REINFORCE
1900
600
PPO
Return
300
Tamar
1000
MVP
MG
Return
000
Tamar
200
MVP
400
MVP
030
MG
300
FiG
100
0
-000
20
40
80
30
100
120
15
Eplsodes(1k)
(p)sodes(1k)
Epissdes(11))
0.7
Inverteddendullum
HallCheelah
Swimmer
0.6
REINFORCE
(d)
PPO
0.6 (f)
PPO
MVO
MVO
MVQ
Timar
Tamar
0.4
MVR
MvP
X-0.5 0..5 Rate
0.4
MVP
MG
MG
MG
0.3
0.2
0.2
0.1
0.1
0.0
0.0
20
40
Epleodes(10k)
80
100
122
(p))
15
30
10
Episodes(1k)
1.5
20
30
Figure 4: (a,c,e) Policy evaluation return and (b.d.f) location visiting rate v.s, training episodes in Mujoco of episode-based methods. Curves are averaged over 10 seeds with shaded regions indicating standard errors. For location visiting rate, lower is better.
5.3 Continuous control:
Mujoco
Mujoco 16 is a collection of robotics environments with continuous states and actions in OpenAI Gym [1S] Here, we selected three domains (InvertedPendullmm HalfCheetah, and Swimmer) that are conveniently modifiable, where we are free modify the rewards to construct risky regions in the environment (Through empirical testing, risk-neutral learning failed when similar noise was introduced to other Mujoco domains. Consequently, identifying the cause for the failure of risk-averse algorithms on other domains became challenging). Motivated by and following [43] 44), we define risky region based on the X-position. For instance, if X-position > 0.01 in InvertedPendulum. X- X-position < -3 in HalfCheetah, and X-position >> 0.5 Swimmer, an additional noisy reward sampled from (00, 1) times 10 is given. Location information is appended to the agent's observation. risk-averse agent should reduce the time it visits the noisy region in an episode. We also include the risk-neutral algorithms baselines to highlight the risk-aversion degree of different methods.
All the risk-averse policy gradient algorithms still use VPG to maximize the expected return in InvertedPendlulum (thus the risk-neutral baseline is REINFORCE). Using VPG is also how these methods are originally derived. However, VPG is not good more complex Mujoco domains, e.g., 445. In HalfCheetah and Swimmer, combine those algorithms with PPO-style policy gradient to maximize the expected return. Minimizing the risk term remains the same as their original forms. MVPI is an off-policy time-differenee method in Mujoco. We train it with 1с6 steps instead of as many episodes as other methods. MVO and MG sample n = 30 episodes in InvertedPendulum and 10 in HalfCheetah and Swimmer before updating policies. Agents are tested for 20 episodes per evaluation. The percentage of time steps visiting the noisy region in an episode is shown in Compared with other return variance methods, MG achieves higher return while maintaining lower visiting rate. Comparing MVPI and TD3 against episode- based algorithms like MG is not straightforward within the same figure due the difference in parameter update frequency. MVPI and TD3 update parameters at each environment time step. We shown their learning curves in MVPI also learns risk-averse policies in all three domains according its learning curves.
Figure
We further design two domains using HalfCheetah and Swimmer. The randomness of the noisy reward linearly decreases when agent's 's forward distance grows. To maximize the expected return and minimize risk, the agent has to move forward as far as possible. The results are shown in Figures 18 19 in Appendix. In these two cases, only MG shows clear tendency of moving forward, which suggests our method is less sensitive to reward choices compared with methods using V[R).
The return variance and GD during learning in the above environments are also reported in Ap- pendix TO In general, when other return variance based methods can find the risk-averse policy, MG maintains lower or comparable return randomness when measured by both variance and GD. When

---

InverledPendllum
HellChastah
Swimmer
500 (6)
2900 07)
TD3
1000
400
2000
800
Retum
300
Reum
1900
Rotum
600
200
1000
100
TD3
800
TDS MVPI
Steps(10k)
80
Sieps(10k)
20
Stops(10K)
02
100
InvertodPenduluum
HeilCheeahh
Swimmer
0.6
[2)
TD3
(5)
TD3
0.7
MVPI
0.
MVPI
0.4
0.5
0.6
X00.01
0.3
0.4
0.0
MVPI
02
0.1
0.2
0.1
0.0
0.0
Sleps 10k)
00
40
(10ps((100)
05
100
Steps/10k)
100
Figure 5: (a,c,e) Policy evaluation return and (b,d.f) location visiting rate v.s. training episodes in Mujoco TD3 and MVPI. Curves are averaged over seeds with shaded regions indicating standard errors. For location visiting rate, lower is better.
other methods fail learn a reasonably good higher return and lower risk policy compared with risk-neutral methods. MVPI has the advantage to achieve low return randomness location based risky domains, since minimizing V[R) naturally avoids the agent from visiting the noisy region. But it fails in distance-based risky domains.
policy, MG consistently finds a notably
6 Conclusion and Future Work
This paper proposes to use a new risk measure, Gini deviation, as substitute for variance in mean- variance RL. is motivated to overcome the limitations of the existing total return variance and per-step reward variance methods, e.g., sensitivity to numerical scale and hindering of policy learning, gradient formula presented and a sampling- minimize such risk. We empirically show that our method can succeed when the variance-basee methods will fail learn risk-averse or a reasonable policy. This new risk measure may inspire new line of research in RARL. First, one may study the practical impact of using GD and variance risk measures. Second, hybrid risk measuee may be adopted in real-world applications leverage the advantages of various risk measures.
policy gradient estimator is proposed to
Our mean-GD policy gradient requires sampling multiple trajectories for one parameter update, making it less sample efficient compared to algorithms that can perform updates per environment step or per episode. As result, one potential avenue for future work is to enhance the sample efficiency of our algorithm. This can be achieved by more effectively utilizing off off-policy data by adapting the algorithm to compatible with online, incremental learning.
Limitations and future work.
Acknowledgments and Disclosure of Funding
We thank Ruodu Wang from University of Waterloo and Han Wang from University of Alberta for valuable discussions and insights. Resources used in this work were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, companies sponsoring the Vector Institute https://vectorinss
ai/partners/ and the Natural Sciences and Engineering Council of Canada. Yudong Luo is also supported by David R. Cheriton Graduate Scholarship, President's Graduate Scholarship, and an Ontario Graduate Scholarship. Guiliang Liu's research was in part supported by the Start-up Fund UDF01002911 I of the Chinese University of Hong Kong, Shenzhen. Yangchen Pan acknowledges funding from the Turing AI World Leading Fellow.
References
[1] Mohammad Naghshvar, Ahmed Sadek, and Auke Wiggers. Risk-averse behavior planning for autonomous driving under uncertainty, arXiv preprint arXiv:1812.0125: 2018.
10

---

[2] Tomas Björk, Agatha Murgoci, and Xun Yu Zhou. Mean-variamee portfolio optimization with state-dependent risk aversion. Mathematical Finance: An International Journal of Mathematics, Statistics and Financial Economics, 24(1):1-24, 2014.
[3] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[4] Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2):294-311, 2002.
[5] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-612) 2017.
[6] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization mdps. Advances in neural information processing systems, 27, 2014.
[7] Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk- averse reinforcement learning, Advances in Neural Information Processing Systems, 35:32639- 32652. 2022.
[8] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. International Conference on Machine Learning, 2012.
[9] Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. Advances in neural information processing systems, 26, 2013.
[10] Harry M Markowitz and Peter Todd. Mean-variance analysis in portfolio choice and capital markets, volume 66. John Wiley & Sons, 2000.
[11] Duan Li and Wan-Lung Ng. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation. Mathematical finance, 10(3):387-406, 2000.
[12] Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, and Daesub Yoon. A block coordinate ascent algorithm for mean-variant optimization. Advances Neural Information Processing Systems, 31, 2018.
[13] Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatility reduction. International Joint Conference on Artificial Intelligence, 2020.
[14] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk- averse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 00905-10913. 2021.
[15] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
[16] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[17] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[18] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):33-333, 1997.
[19] Shie Mannor and John Tsitsiklis. Mean-varte Mean-variance optimization in markov decision processes. International Conference Machine Learning, 2011.
[20] Matthew Sobel. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794-80) 1982
11

---

[21] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[22] Corrado Gini. Variabilità mutabilità: contributo allo studio delle distribuzioni delle relazioni Mattistiche.(ase. I.J. Tipogr. di P. Cuppini. 1912.
[23] Shlomo Yitzhaki al. Gini's mean difference: A superior measure of variability for non-normal distributions. Metron, 61(2)2885-316, 2003.
[24] Gerald J Glasser. Variance formulas for the mean difference and coefficient of concentration. Journal of the American Statistical Association, 57(299):648 654, 1962.
[25] Edward Furman, Ruodu Wang, and Ričardas Zitikis. Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks. Journal of Banking & Finance, 83:70-84, 2017.
[26] Gustave Choquet. Theory of capacities. In Annales de l'institut Fourier, volume pages 131-295, 1954.
[27] Michel Grabisch. The application of fuzzy integrals in multicriteria decision making, European journal of operational research, 89(3):445-456, 1996.
[28] Ruodu Wang, Yunran Wei, and Gordon E Willmot. Characterization. robustmess, and aggregation of signed choquet integrals. Mathematics of Operations Research, 45(3):993-1015, 2020.
[29] Shigeo Kusuoka. On law invariant coherent risk measures. In Advances in mathematical economics, pages 83-95. Springer, 2001.
[30] Bogdan Grechuk, Anton Molyboha, and Michael Zabarankin. Maximum entropy principle with general deviation measures. Mathematics of Operations Research, 34(2):445 467, 2009.
[31] R Tyrrell Rockafellar, Stan Uryasev, and Michael Zabarankin. Generalized deviations in risk analysis. Finance and Stochastics, 10(1):51-74, 2006.
[32] Fangda Liu, Jun Cai, Christiane Lemieux, and Ruodu Wang. Convex risk functionals: Repre- sentation and applications. Insurance: Mathematics and Economics, 90:66-79, 2020.
[33] Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the cvar via sampling. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
[34] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[35] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105, 105. PMIR, 2018
[36] Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3455-3461, 8 2021.
[37] Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforcement learning with monotonic splines. In International Conference on Learning Representations, 2022.
[38] Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit distributional reinforcement learning. Advances in Neural Information Processing Systems, 33:7 13.7133-7147, 2020.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
12

---

[40]
Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candee Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
[41] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[42] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[43] Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforce- ment learning. In International Conference on Machine Learning, pages 7390-7399, PMLR, 2021.
[44] Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations, 2023.
[45] OpanAL. Performance in each mujoco environment, httpss//eppinn.ngggooppeaaai com/ en aa//attest//spmmmuggg//eeee html
[46] Arjun K Gupta and Mohammad AS Aziz. Convex ordering of random variables and its applications econometrics and actuarial science. European Journal of Pure and Applied Mathematics, 3(5):779-785, 2010.
[47] Michael Rothschild and Joseph E Stiglitz. Increasing risk: 1. definition. In Uncertainty in Economics, pages 99-121. Elsevier, 1978.
[48] Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical finance, finance 9(3):203-222 1999.
[49] Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. Advances in Neural Informa- tion Processing Systems, 32, 2019.
[50] Aravind Rajeswaran, Sarvject Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In International Conference on Learning Representations, 2016.
[51] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: cvar optimization approach. Advances in neural information processing systems, 28, 2015.
[52] Yichuan Charlic Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint arXiv:1911.03618, 2019.
[53] Qisong Yang, Thiago D Simão. Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safety-conss canttreconstrained reinforcement learning. In Proceedings of the AAAI Conference Artificial Intellgence, volume 35, pages 10639-10646,, 2021.
13

---

Supplementary Information
7 Convex Order, Gini Deviation, and Variance
Convex order describes dominance terms of variability and is widely used in actuarial science. Definition (166), Consider two random variables X and Y, X is called convex order smaller than Y, succinetly X Sex Y, if B (X)] (Y)) for all convex function +(), assuming that both expectations exist.
In convex order X Sex Y, Y is also called mean wrean-preservin spread of X 47 which intuitively means that more spread-out (and hence more random) than X. Thus, it often desirable for measure of variability to be monotone with respect to convex order 225 Both variance and GD, as measure of variability, are consistent with convex order, i.e.,
. If X Sex Y, then V[X] I V(Y] for all X,Y Y E M If Sox Y, then X S DJY] for all X,y E EM
Proof. It is immediate that X Sex Y implies E[X] = E[y]. If we take convex function (xx) x² we can get the order of variance V[x] V[Y]. For the proof of GD, please refer to the following Lemma. Recall that GD can be expressed in the form of signed Choquet integral with a concave function h.
Lemma 3 Theorrm 2). Convex order consistency of a signed Choquet integral is equivalent
its distortion function h being concave, i.e., X Sex Y if and only if the signed Choquet integral 4% (X) < (Y) (Y)for all concave functions
s h E H.
GD Gradient Formula Calculation
8.1 General GD Gradient Formula
Proposition Let Assumptions
7 2 hold. Then
[oD] T2 Vo log fz(;;)) f (2Fz, (f) - -1
(11)
Consider random variable Z, whose distribution funetion is controlled by 9. Recall that qa qa (Z:6) represents the a-level quantile of Zg. According to Equation [4 the gradient of GD is
VaDiZa] VoDu(Ze) = f(" (2a - 1) [_
(2:0) (2:0)
vofz(z;0)d= f2 (g. (2;(22))); da
To make the integral over a clearer, we rewrite qa (2:0) as Fz. (a), where Fz, is the CDF.
V Dio S( (2aa-
l F2 (a)
Vofz f)d
fz (F)) da
Switching the integral order, we get
VolD Sr [', PZo(e) (2a- 1)Vo1))
1
fz Zo (a): 0) dodz
(19)
= =- S Vofz(z:0) S F2)
(2a-1) - 1) fz fz(17 (a): dadz
Denote FZ. (a), then Fz, (t). Here, further change the inner integral from da to dF (t), i.c., da = dFz. (t) = (z(t;0)dt The integral range for is now from F7, (Fz, (2)) =
to Zoo (1) = b.
VoDlZal = - /" b
Vol2(=00) L. (2Fz. 1) [z(1;0) dF'z (t) dz
(20)
Volz(z;0) L (2Fz. -1)dt dz
14

---

04
48
R4, R1g Rey R4,
R.
R
Figure An example of parameterizing (inverse) CDF given six quantiles. The function is highlighted in the bold line of orange color.
Applying Vo Vog((( 1 [Vox to Volz(z;0), have
VoD|Zo| = 1" fz( f2(2;0) 12(2:0)Voloo f (2Fz. - d) dz
(21)
[ (2Fz, 1)dt
E
8.2
GD Policy Gradient via Sampling
Following Section
4.2 we consider estimating the following equation via sampling.
1' (2FG. - 1)dt]
VoD(Go) = Ggg.o Vo log Ja.(:))))
Here sampling from Gg corresponds to sampling trajectory with its return R. from the envi- ronment. As discussed the main paper, for trajectory Ti, Vo log Sc, (R:;00 0) is estimated as
Et Va log Ne (as,r (8,,))
ia The CDF function FG, is parameterized by step function given its quantiles {R, -i= which satisfy R2 S R,, I I R,,, An example of the step function is shown in Figure 6 With this parameterization, the integral over CDF can be regarded as the area below the step function.
Thus, for each т;, the integral over CDF approximated (R,, is treated b)
Reo
n-1
(1)d Rs
2 x 1
(R(T)+1) - R(T;))
(22)
Aggregating all the calculations together yields Equation 16
Mean-GD Policy Gradient Algorithm
We consider maximizing E[Go] XD [Go] in this paper. Maximizing the first term, i.e., E[Gu] has been widely studied in risk-neutral RL. For on on-policy policy gradient, we can use vanilla policy gradient (VPG), e.g., REINFORCE with baseline,
advanced techniques like PPO 39
To make fair comparison with other risk- avere policy gradient methods, we first initialize mean-Gi policy gradient with VPG. To improve sample efficiency, we incorporate IS for multiple updates. Taking advantage of having n samples, we can select those trajectories whose ratio the range
[] [1 8, + d] for calculation to reduce gradient variance. In more complex continuous domains, e.g., Mujoco, we combine GD policy gradient with PPO since VPG is not good at Mujoco [45) However, in this case, policy may have significant difference per update, where the former IS selection strategy can no longer be applied. Then we directly clip the IS ratio by constant, though is biased 44).. The full algorithms summarized in Algorithm and [ We omit the parameters for PPO Algorithm for simplicity. We still report the PPO parameter settings in Section TO
15

---

Algorithm 1 Mean-Gini Deviation Policy Gradient (with REINFORCE baseline)
Input: Iterations number K, sample size n, inner update number M, policy learning rate Ogg, value learning rate ag. importance sampling range 6, inner termination parameter B3, trade-off parameter
X.
Initialize policy He parameter 0, value Vo parameter @ for k = to K do
Sample n trajectories {ri)i= by æ0, compute return {R(T)))=
Compute rewards-to-go for each state in Tj: Ii.t for m
M do
Compute importance sampling ratio for each trajectory {pi)i=
Select D = [ whose Ps E [1 6, 8) Sort trajectories such that R(Ti) S S R((p)))
if (D) < n 3 then
break
end if
mean grad = 0, Loni_grad- =0
for 1 to [D] do
mean mnean_grad += pl ET-'ve Ve log ne (a (9,,) V(((s,)))
Update Vi by mean-squared error T Ei=o (Vo(sie) 91.)² with learning rate ola end for for i 1 to [D] do
gini gradgrad-
Pi Yi -t= V log ne (ai (si,)) where
-T
= Σ |P| - [D]
(Rz+1 - R2) - (R|p) Ri)
end for Update गy by ( D mean grad - gini grad) with learning rate ag (Equation 17
end for end for
10
Experiments Details
10.1
General Descriptions of Different Methods
Among the methods compared in this paper, Tamar 8 and MVP 12] are on-policy policy gradient methods. MVO and MG policy gradient methods, but sample trajectories and use IS update. Non-tabular MVPI [4] is an off-policy time-difference method.
Policy gradeint methods. MVO, Tamar, MVP, originally derived based VPG. Since VPG is known to have a poor performance in Mujoco, we also combined these mean-varianc methods with PPO. Thus these merinvariance methods and our mean-GD method have different instantiations to maximize the expected return different domains:
With VPG: in Maze, LunarLander, InvertedPendullum
With PPO: in HalfCheetah, Swimmer.
When the neutral policy gradient is VPG, for MVO and MG, is REINFORCE with baseline; for Tamar and MVP, we strictly follow their papers to implement the algorithm, where no value function is used. MVO and MG collect trajectories and use the IS strategy Algorithm [T to update policies. Tamar and MVP do not need IS, and update policies at the end of each episode.
When the risk-neutral policy gradient PPO, augment PPO with the variance GD policy gradient from the original methods. MVO and MG collect n trajectories and use the IS strategy in Algorithm 2 to compute the gradient for the risk term. Tamar and MVP still update policies once at the end of each episode.
MVPI. We implemented three versions of MVPI in different domains:
Tabular: in Maze, MVPI is a policy iteration method (Algorithm in 14
With DQN: in LunarLanderr, since this environment has discrete action space.
With TD3: InvertedPendulum, HalfCheetah, Swimmer, since these environments have continuous action space.
16

---

Algorithm 2 Mean-Gini Deviation Policy Gradient (with PPO)
Input: Iterations number K, sample size n, inner update number M, policy learning rate Og, value learning rate ag. importance sampling clip bound trade-off parameter X.
Initialize policy parameter 0, value V₂ parameter Φ.
for to K do
Sample n trajectories {T. by Te, compute return {R(T:))== Compute rewards-to-ge go for each state Tú: Ii.t
Compute advantages for each itateacction in T;: A(si,t, ai,z) based on current V for m M do
Compute importance sampling ratio for each trajectory {pili= and pi min b)
Sort trajectories such that R(TI) S S R(TT)
mean_grad = 0, gini ahni_gaad= 0 for to n do
mean graidgrraada += PPO-Clip actor grad
Update Vo by mean-squared error + Eioo (Vo(siz) gi,)) with learning rate ou end for
for 1 to n 1 do
T-1 gini grad += PP: Et=o Vo log No Te(0 where
7/
Σド= a (Rz+1 R₂) (Rn, R.)
2j
end for
Update Ty by ( (NT mean grad [n-1)T gini grad) with learning rate ag (Equation 18
end for
end for
We summarize the components required in different methods in Table
Table 1: Model components in different methods
Policy fune
Value fune
Additional training variables
MVO-VPG
MVO-PPO
Tamar-VPR
V
J.V (mean, variance)
Tamar-PPO
J.V (mean.variance)
MVP-VPG
V
(dual variable)
MVP-PPO
y (dual variable)
MG-VPG
MG-PPO
MVPI-Q-Learning
MVPI-DQN
MVPI-TD3
10.2 Modified Guarded Maze Problem
The maze consists of a x 6 grid. The agent visit every free cell without wall. The agent can take four actions (up. down, left, right). The maximum episode length is 100.
Policy function. For methods requiring policy function, i.e., MVO, Tamar, MVP, MG, the policy is represented as
mo (a(s))
(23)
EB co(s,b)-
where d(s a) is the state-action feature vector. Here we use one-hot encoding to represent d(s, a). Thus, the dimension of o(s, 6 x 6 x 4. The derivative of the logarithm is
Ve log To (als) = Φ(s, a) Es ((s, b)
(24)
Value function. For methods requiring a value function, i.e., REINFORCE baseline used in MVO and MG, and Q-learning MVPI, the value function represented (s) = <(s)) w or (,, a) ((s, a) w. Similarly, ((s) is one-hot encoding.
17

---

Optimizer. The policy and value loss are optimized by stochastic gradient descent (SGD).
10.2.1 Learning Parameters
We set discount factor = 0.999.
policy learning rate is le-5 (55e-,, 1e-5, 5e-6), value function learning rate is 100 times policy learning rate. À = 1.0 e(0.6, 0.8, 1.0, 1.2). Sample size n = 50. Maximum inner update number M 10. IS ratio range 5 = 0.5. Inner termination ratio 8 = 0.6.
MVO:
Tamar: policy learning rate le-5 E le-5, 5e-6), J. V learning rate is 100 times policy learning rate. Threshold b = 50 (10, 50, 100), À 0.1 ( 0.2, 0.4).
MVP: policy learning rate is le-5e Ie-5, 5c-6), y learning rate the same. À 0.1 E 0.2, 0.4).
MG: policy learning rate is le-46 Se-5), value function learning rate is 100 times policy learning rate. X = 1.2 E [0.8. 1.0, 1.2). Sample size n = 50. Maximum inner update number M = 10. IS ratio range 5 0.5. Inner termination ratio 3 0.6.
MVPI: Q function learning rate (S-3E le-3, 5e-4), ) 0.2
10.2.2 Analysis for MVPI in Maze (MVPI-Q-Lecrrning)
MVPI-Q-LaR -rPl-Q-Learnin finds the optimal risk-averse path when goal reward is but fails when goal reward is 40. Since it is not intuitive to report the learning curve for policy iteration method where its reward is modified each iteration, we give an analysis here.
The value of dual variable in Equation is (1 (0--)))B(c B[oo given the current policy. Recall that the maximum episode length 100. At the beginning, when the Q function randomly initialized (i.e.. it is a random policy), B[Go] Σ9 0.909' (-1) 0.0952, the goal reward after modification is r'ooll 20 0.2 20 + 2 0.2 x 20 x y ~ 60.7. For the red state, its original reward is sampled from -15 1, 133. After the reward modification, becomes sampling from (- 59.4, 1.16. -2..2.. Thus the expected reward of the red state is now Tred = 0.4 x (-59.4) + 0.2 x (-1.16) + 0.4 x (-21.2) = -32.472. Given the maximum episode length 100, the optimal policy is still the white path in Figure T (Because the expected return for the white path is Ei 0.999' (-1) + (-997)) As 700 The expected return for a random walk is E 0.999* )) = 25.2. The expected return for the shortest path going through the red state is even lower than the white path since the reward of the red state after modification is pretty negative: -2..472.
95.2. Thus y (1 0.999) x (-95.2)
However, when goal reward is 40, after modification, the goal reward becomes rgoal 40 0.2 x 402 + x 0.2 x 40 X y s 281.5. In this case, the optimal policy has to avoid the goal state since leads to a even lower return.
10.2.3 Return Variance and Gini Deviation in Maze
report the return's 's variance and GD during learning for different methods, as shown in Figure 7 and Tamar [8] is unable reach the goal in both settings. MVO fails to reach the goal when the return magnitude increases. MVP [2]'s 's optimal rvsk-aversion rate is much lower than MG. MG can learn risk averse policy in both settings with lower variance and GD, which suggests it is less sensitive to the return numerical scale.
Maze, goal 20, Retur
Maze, gool 20, Return Variance
Maze, goal 2-2 Retur Gini Deviation
MVO
MVD
Tamair (failed)
Temar (falle)
MVO
WVR W
MVR
Tamar
MG
MG
MVP
-60
40
-80
20
100
0
25
50
75
100
125
150
175
200
0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
(psodee((1k
Episodee/1kk
(pssodes(1k)
Figure 7: Expected return, return variance and Gini deviation different methods in Maze when
goal reward is 20. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
18

---

Maze, goal Retum
goal_r440, Variance
Maze, roal_r400, Return Gini Deviation
40
MVO (failed)
30
MVD (falle)
60
Taimar (failed)
25
Tamar (falled)
MVP
MVP
MVO
Ma
2.0
ABG
Tamar
1.5
-60
MG
20
100
75
200
100 Episodes] 1lk)
175
160
(1)
(pisodes(1k)
Figure 8: Expected return, return variance and Gini deviation of different methods in Maze when goal reward is 40. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
10.2.4 Sensitivity to Trade-off Parameter
We report the learning curves total return based methods with different À when goal reward is 20 in Figure 9
The learning parameters are the same as shown in Section 10.2.1.
10.3 LunarLande Discrete
The agent's goal land the lander on the ground without crashing- The state dimension is 8. The action dimension is 4. The detailed reward information is available at this webpage [ whole ground into left and right parts by the middle line of the landing pad as shown in Figure I0 the agent lands in the right part, an additional noisy reward signal sampled from N(0, 1) tims 90 is given. We set the maximum episode length to 1000. Note that the original reward for successfully landing is 100, thus the numerical scale of both return and reward is relatively large in this domain.
We divide the
Policy function. The policy is categorical distribution in REINFORCE, MVO, Tamar, MVP and MG, modeled as neural network with two hidden layers. The hidden size is 128. Activation is ReLUU Softmax function is applied to the output to generate categorical probabilities.
Value function. The value function REINFORCE, MVO, MG, and Q function in MVPI-DQN is a neural network with hidden layers. The hidden size is 128. Activation is ReLU.
Optimizer. The optimizer for policy and value functions is Adam.
10.3.1 Learning Parameters
Discount factor is 7 0.999
REINFORCE (with baseline): the policy learning rate is 7e-4 E (7e-4, 3e-4, 7e-5), value function learning rate 10 times policy learning rate.
MVO: policy learning rate is 7e-5 E (7e-4, 3e-4, 7e-5), value function learning rate is 10 times policy learning rate. x = 0.4 E (0.4, 0.6, 0.8). Sample size n = 30. Maximum inner update number M = 10. IS ratio range d 0.5. Inner termination ratio 3 0.6.
Tamar: policy learning rate 7c-5 E (7c-4, 3c-4, 7e-5). J, V learning rate is 100 times the policy learning rate. Threshold b = 50 e 100). X = 0.2 E (0.2, 0.4, 0.6).
MVP: policy learning rate is 7e-5 E (7c-4, 3c-4, 7 J. y learning rate is the same. À 0.2 {0.2. 0.4, 0.6).
MG: policy learning rate is 7e-4 E [7e-4, 3e-4, 7e-5), value function learning rate is 10 times policy learning rate. = 0.6 E (0.4, 0.6, 0.8). Sample size n = 30. Maximum inner update number M = 10. IS ratio range å = 0.5. Inner termination ratio 3 0.6.
MVPI: Q function learning rate is 7e-4 E (7e-4, 3e-4, 7e-5), 0.2 (0.2, 0.4, 0.6). Batch size is 64.
10.3.2 Return Variance and Gini Deviation LunarLander
The return's 's variance and GD of different methods during training is shown in Figure II All the risk-averse methods, apart from ours, fail to learn a reasonable policy in this domain. Our method achieves a comparable return, but with lower variance and GD compared with fisk-neutral method.
http:///wwwggiillleeee/eoooomooe lander/
19

---

goal r-20.0, MVD
goa 220.0, MVO
Oplimal risk-aversion rate
0.8
-20
Return
-40
0.4
lame8
lam=0.5
lam=1.0
lam=110 00
-80
lam=2.0
0.2
lam=2.0
-100
lam=4.0 4.0
0.0
lam=4.0 0
25
75
100
125
150
175
200
25
50
75
100
125
180
175 200
Episodes(1k)
Episodes(1k)
goal__=200 (000__2000,, Tamar
m0.00,
-80
lam=0.05
lam=0.05
lam
lam=0.1
lam=0.2
Oplmal tak-averson
0.150
lamm0..2
Return
lam=0 4
0.125
lam=0.4
-90
-95
0.050
0.025
100
25
100
125
150
175
200
Episodes(1k)
Episodes(1k)
goal 2=20.0 g0..0r MVP
r00.0, MVP
Optimal risk aversion rale
0.4
lam=0.05 0.05
lam+0.1 1.1
lam=0 2
Return
lam=0.4
-40
lam=0.0g
0.2
-60
lam=0.1
lam=0.2
0.1
-80
lam=0
Episodes(1k)
Episodes(1k)
_oal_r=00 D, MG
F0
goal r=20.0, MG
Optimal riek- malc-aversion rate
0.8
lam=0
lam=1-2
-20
lam=0 8
0.6
am==166
Return
lam=1.2
lam=2
lam=1.6
0.4
-60
lam=2.0
-80
100
0.0
25
50
75
100
125
150
175
200
25
50
100 Episodes(1K)
75
125
150
175
200
Eplaodes(1k)
Figure 9: Policy evaluation return and optimal risk-aversion rate v.s. training episodes in Maze (goal reward is 20) for MVO, Tamar, MVP, and MG with different X. The reasonable À range varies in different methods. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
20

---

Figure 10: Divide the ground of LunarLander into left and right parts by the middle (red) line. If landing in the right area, an additional reward sampled from A ( 1) times 90 given.
LunarLander, Return Variance
LunarLander, Return Gini Devistion
200000
150000
REINFORCE
175
REINFORCE
100000
NYO (failee)
(falled)
Tamar (falled)
MVP (failed)
125
MVP (falled)
30000
MG
MVPI (elld)
75
15000
25
20
RU
100
120
40
80
80
100
120
(plsodes(1k)
Figure 11: Return variance and Gini deviation of different methods LunarLander,, Curves are averaged over 10 seeds with shaded regions indicating standard errors.
10.4 InvertedPendulum
(The description of the Mujoco environments can be found at this webpage +
The agent's goal is to balance a pole on a cart. The state dimension (X (X-position already contained). The action dimension is 1. At each step. the environment provides reward of 1. the agent reaches the region X-coordinate > 0.01, an additional noisy reward signal sampled from N (0, 1) times 10 is given. To avoid the initial random speed forcing the agent to the X- coordinate > 0.01 region, we decrease the initial randomness for the speed from U-00.01, 0.01) to U(-0.0005, 0.0005), where U() represents the uniform distribution. The game ends if angle between the pole and the cart is greater than 0.2 radian or a maximum episode length 500 is reached.
Policy function. The policy is normal distribution in REINFORCE, and VPG based methods (MVÓ, Tamar, MVP, and MG), modeled as neural network with two hidden layers. The hidden size 128. Activation is ReLU. Tanh is applied to the output to scale it to (-1, 1). The output times the maximum absolute value of the action serves as the mean of the normal distribution. The logarithm of standard deviation is independent trainable parameter.
The policy deterministic function TD3 and MVPI, modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Tanh is applied to the output to scale it to (-1,1). The output times the maximum absolute value of the action the true action executed the environment.
Value function. The value function in REINFORCE, VPG based MVO, VPG based MG, TD3, and MVPI is a neural network with two hidden layers. The hidden size is 128. Activation is ReL.U.
Optimizer. Optimizer for both policy and value function is Adam.
10.4.1 Learning Parameters
Discount factor 7 = 0.999.
REINFORCE (with baseline): policy learning rate is le-4 E ( [le-4, 5e-5, 5e-4,, value function learning rate 10 times policy learning rate.
21

---

MVO: policy learning rate is le-5 E f1e-4, 5e-5, le-5), value function learning rate is 10 times policy learning rate. À 0.6 E (0.2, 0.4, 0.6). Sample size n 30. Maximum inner update number M = 10. IS ratio range å = 0.5. Inner termination ratio / = 0.6.
Tamar: policy learning rate is 1e-5 E (1e-4, 5e-5, 1e-5). ]. J. learning rate is 100 times policy learning rate. Threshold b = 50 E (10.50,100). À = 0.2 E {0.2, 0.4, 0.6).
MVP: policy learning rate is le-5 E (le-4, 5c-4, le-5). y learning rate is the same. À 0.2 E {0.2, 0.4, 0.6].
MG: policy learning rate is le-4 E { [1e44, 5e-5, le-4), value function learning rate is 10 times policy learning rate. 1.0 E (0.6, 1.0, 1.4). Sample size 30. Maximum inner update number M = 10. IS ratio range ā = 0.5. Inner termination ratio 33 0.6.
MVPI: Policy and value function learning rate 3e-4 E (3e-4, 7e-5, le-5), À 0.2 E (0.2, 0.4, 0.6). Batch size is 256.
TD3: Policy and value function learning rate 3e-4 [3e-4, 7e-5, le-5). Batch size is 256.
10.4.2 Return Variance and Gini Deviation InvertedPendulum
The return varaince and GD of different methods are shown in Figure 12 and 13
InverledPendullm. Return Variance
InvertedPendulum, Return Gini Deviation
REINFORCE
REINFORCE
30000
MVO (failed)
MUD (falled) Tamer (falled) MVP (failed
25000
Tariaar (failed)
WW
MVP (hanee)
20000
MG
MG
15000
40
10000
20
20
40
80
80
100
120
20
80
80
100
120
Episodess(18) 1k)
Episodes(11
Figure 12: Return variance and Gini deviation of policy gradient methods in InvertedPendulum. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
InvartadPandullum, Return Variance
InvertedPendullm, Return Giri Devistion
25000
TD3
70
TD3
M/PI
60
MVP
50
10000
90
5000
Steps(10k)
Steps((10k)
Figure 13: Return variance and Gini deviation of TD3 and MVPI in InvertedPendullm. Curves are averaged over 10 seeds with shaded regions indicating standard errors. 10.5 HalfCheetah
The agent controls robot with two legs. The state dimension is 18 (add X- X-position). The action dimension is 6. The reward is determined by the speed between the current and the previous time step and penalty over the magnitude of the input action (Originally, only speed toward right is positive, we make the speed positive in both direction so that agent is free to move left or right). the agent reaches the region X-coordinate (O 1) times 10 is given. The game ends when a maximum episode length 500 is reached.
-3, additional noisy reward signal sampled from
Policy function. The policy is normal distribution in PPO, and PPO based methods (MVO, Tamar, MVP, and MG). The architecture is the same as in InvertedPendulum. Hidden size is 256.
The policy of TD3 and MVPI is the same as in InvertedPendullm. Hidden size is 256.
Value function. The value function in PPO, PPO based methods (MVO, Tamar, MVP, and MG). TD3 and MVPI is neural network with two hidden layers. The hidden size 256. Activation is ReLU. Optimizer. Optimizer for policy and value is Adam.
22

---

10.5,1
Learning Parameters
Discount factor is 7 = 0.99.
Common parameters of PPO and PPO based methods. GAE parameter: 0.95, Entropy coef: 0.01, Critic coef: 0.5, Clip e: 0.2, Grad norm: 0.5.
PPO. policy learning rate 3e-4 E (3e-4, 7e-5, le-5), value function learning rate is the same. Inner update number M
MVO. policy learning rate 7c-5 E (3e-4, 7c-5, 1e-5J, value function learning rate is the same. Sample size n 10. Inner update number 5.
Tamar. policy learning rate 7e-5 E (3e-4, 7e-5, le-5), value function learning rate is the same. J, V learning rate is 100 times policy learning rate. Threshold b 50 E (10,50,100). À 0.2 E (0.2, 0.4, 0.6).
MVP. policy learning rate 7e-5 E (3e-4, 7e-5, le-5), value function and y learning rate is the same. 0.2 (0.2, 0.4, 0.6).
MG. policy learning rate 3e-4 [3e-4, 7e-5, Le-5), value function learning rate is the same. Sample size n 10. Inner update number .M 5.
TD3. policy learning rate 3e-4 E (3e-4, 7e-5, le-5), value function learning rate is the same. Batch size is 256.
MVPL policy learning rate 3e-4 E (3e-4, 7e-5, le-5), value function learning rate is the same. À 0.2 E (0.2, 0.4, 0.6). Batch size is 256.
10.5.2 Return Variance and Gini Deviation in HalfCheetah
See Figures
and
HallCheelah, Return Variance
HalCCheeiah, Retum Gini Devialion
PPO
PPO
MVO (falled)
MVD (failed)
Tamar (failed)
Tamar (falle)
150000
MVP (failed]
MVP (failed)
60000
20
Episodias(1k)
(pisodes[1k)
Figure 14: Return variance and Gini deviation of on-policy methods in HalfCheetah. Curves averaged over 10 seeds with shaded regions indicating standard errors HalfCheetah, Retum Varianse
HaifChestah, Retum Gni Deviation
80000
TD3
TD3
70000
MVP1
120
MVPI
60000
100
60
20000
40
80
100
0
80
100
Steps(10k)
Steps[10K) (10)
Figure 15: Return variance and Gini deviation of off-policy methods in HalfCheetah. Curves are averaged over 10 seeds with shaded regions indicating standard errors 10.6 Swimmer
The agent controls a robot with two rotors (connecting three segments) and learns how to move. The state dimension is 10 (add XY-positions). The action dimension is The reward is determined by the speed between the current and the previous time step and a penalty over the magnitude of the input action (Originally, only speed toward right positive, we make speed positive both direction so that agent is free to move left right). If agent reaches the region X-coordinate > 0.5. an additional noisy reward signal sampled from (0, 1) times 10 given. The game ends when maximum episode length 500 is reached.
23

---

The neural network architectures and learning parameters are the same as in HalfCheetah.
10.6.1
Return Variance and Gini Deviation in Swimmer
See Figures
16
17
Swimmer, Return Varlance
Swimmer, Retum Gini Deviation
PPO
PPO
MVQ
80
MVO
Tainar
Tamar
20000
15000
MG
MG
10000
40
30
8000
Episodes(1k)
Episodes(11)
Figure 16: Return variance and Gini deviation of on-policy methods in Swimmer: Curves averaged over 10 seeds with shaded regions indicating standard errors
Swimmer, Retur Varience
Swimmer, Retum Gini Devielion
35000
TD3
100
30000
MVPI
80
25000
TD3
15000
10000
Staps(10K) Ok)
Staps(10K)
Figure 17: Return variance and Gini deviation of off-policy methods in Swimmer. Curves averaged over 10 seeds with shaded regions indicating standard errors
11 Additional Results
design two other senarios in HalfCheetah and Swimmer (marked as HalfCheetahl and Swimmerl in the figure's caption). Here the randomness of the noisy reward linearly decreases over the forward distance (right direction) the agent has covered. To encourage the agent to move forward, only the forward speed is positive. The additional noisy reward is sampled from (O 1) times 10 times 1 X X X-position >> 0. To maximize the expected return and minimize the risk, the agent should move forward far possible.
The learning parameters are the same as Section 10.5
The distance agents covered in these two environments are shown in Figures
18
HalfCheelah1, X-poeition
HalfCheetahl, X-posiion
Tamar
MVP
-2
40
20
20
20
30
40
60
100
Episodes(1k)
Steps/10k)
Figure 18: The distance agents covered in HalfChectahl. Curves are averaged over 10 seeds with shaded regions indicating standard errors.

---

X-posttion
Swimmert, postoo
MMD
-02
MVPI
0.
0.90
MVP
-03
0.25
-0.4
0.00
-025
-050
-0.0
-0.75
Episodes(11)
Steps(11k)
Figure 19: The distance agents covered in Swimmerl. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
12
Additional Related Work and Discussion
12.1 Coherent measure of variability and risk
This paper focuses on the measure of variability, i.e., the dispersion random variable. Both variance and Gini deviation are measures of variability as discussed in Sections 2 and 3. Gini deviation is further known coherent measure of variability.
Another widely adopted risk measure is conditional value at risk (CVaR), which is a coherent risk measure [48]. Coherence is usually important in financial domains. Consider continuous random variable Z with the cumulative distribution function F(z) = P(Z < 2). The value at risk (VaR) or quantile confidence level a E (0, 1] defined VaR. (Z) min F(z) 2 a}. Then the CVaR at confidence level a is defined as
1 CVaR (2) = - a / Va
(25)
Due to the nature of CVaR of only considering the tailed quantiles, does not capture the variability of Z beyond the quantile level a. Thus it is not considered as measure of variability. In [25]. measures of variability and measures of risk are clearly distinguished and their coherent properties are different. We summarize their properties here 25
Consider measure p M (-00, oc]
(A) Law-inariiance X, Y E MM have the same distributions, then p(X) = p(Y)
(A1) Positive homogeneity: p(AX) Xp(X) for all 0, X
(A2) Sub-additivity: p(X Y) p(X) + p(Y) for X,Y e M
B Monotonicity; p(X) S p(Y) when X < SY M
(B2) Translation invariance: p(X m) = (X)) - m for ER
(CI) Standardization; p(m) 0 for all E IR
(C2) Location invariance: p(X m) p(X) for m
Coherent measure of variability: measure of variability coherent it satisfies properties (A), (A1), (A2), (C1) and (C2). An example is Gini deviation.
Coherent risk measure: A risk measure is coherent if it satisfies properties (A1), (A2), (B1) and (B2). An example CVaR.
12.2 CVaR optimization in RL
CVaR focuses on the extreme values of a random variable and is usually adopted in RL domains avoid catastrophic outcomes. Policy gradient an important method to optimize CVaR, e.g.. see [33 49 [7] Since CVaR focuses on the worst returns, policy gradient techniques often ignore 7 proposed cross entropy soft risk optimization to improve sample efficiency by optimizing with respect to all trajectories while maintaining risk aversion. A time difference method is proposed [51] CVaR also used for action selection in distributional RI.
high-return trajectories 33 50
literature, c.g., see 35
52 53
25


Reviews:
[{"review_id": "T3WloRGx9d", "reviewer": "Reviewer_diaH", "summary": "This paper discusses the challenges of mean-variance RARL methods, and proposes the Gini deviation as a substitute to variance penalization.", "questions": "Included in “weaknesses”. ", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and well-motivated (by the disadvantages of return and instantaneous reward variance). The proposed method is evaluated in several experiments with comparator baselines, which seem convincing.", "weaknesses": "It seems the proposed algorithm has to sort its samples in every iteration (L245-248) which seems computationally expensive, and which may be sensitive to the parameters chosen for $n$? \n\nWhile I recognize that this is a largely empirical paper, I suppose the paper could have been made more complete by providing some convergence guarantees for policy gradient, e.g., to a stationary point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review. We address main concerns below. \n\n+ **It seems the proposed algorithm has to sort its samples in every iteration (L245-248) which seems computationally expensive, and which may be sensitive to the parameters chosen for $n$?**\n\nThe sample here corresponds to trajectory's return. In practice, sorting samples in our experiments is in general not very time consuming since the maximum number of trajectory's return to sort is only 50 (in the maze problem in Section 5.1).\n\nRegarding the sensitivity, we did not find it difficult to choose $n$ in the experiments. For instance, $n$=10 in HalfCheetah and Swimmer (in Section 5.3) is reasonably good to achieve a low chance of visiting the high variance region in our experiments."}, {"review_id": "n9uxVaugY9", "reviewer": "Reviewer_FmwV", "summary": "This paper presents a new approach to measure the risk for risk-averse reinforcement learning problems. \nTo be more precise, traditional methods usually use the variance of total return or per-step reward as the measurement of risk, since it is known that the variance is easy to calculate. Though easy to calculate, variance-based methods are observed to be sensitive to numerical scale and hard to optimize the policy. Based on the observation, the authors propose to use Gini deviation as the alternative to variance for measuring the risk. The authors study the properties of the new measurement and present how to estimate Gini deviation using empirical samples. Empirical experiments demonstrate that the new proposed methods can mitigate the limitation of variance-based methods while it can still learn high reward returns, compared with previous methods. ", "questions": "- Can you clarify if we use empirical samples to estimate Gini Deviation, what would be the draw back? For example, what's the variance and bias of the new estimator using empirical samples if it has? \n\n- Would you mind explaining the complexity of the new proposed method? \n\n- Also, is it possible to demonstrate that with the new proposed measurement, the new estimator would be robust to numerical scales? (Maybe no need for empirical experiments, some intuition explanation would also be okay). \n", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well written, and the authors did an excellent job of presenting the limitation of variance methods before introducing Gini Deviation. As a result, readers can get a sense of the motivation for using Gini deviation as an alternative. \n\n- The authors thoroughly introduce the properties of Gini deviation and present how to estimate this quantity using empirical samples. Besides the authors also introduce how to estimate this quantity under the setting of using importance sampling and PPO. And I think through this, authors can get a better understanding how to use the quantity in different variant of policy gradient methods. \n\n- The authors also demonstrate the advantage of Gini Derivation via empirical experiments, for both tabular, discrete, and continuous control experiments. \n\n", "weaknesses": "- Though the authors present that Gini Deviation can be estimated via empirical samples, my primary concern is that it does not have close form and thus its estimation totally depends on the quality of the samples, if we do not have enough samples, then the variance or the bias of the estimation can lead to undesired optimization behavior. \n\n- Since Gini Deivation requires estimating the quantile, it may introduce additional complexity compared with variance-based method. \n\n- The authors claim that variance-based methods might be sensitive to numerical scales, but I have not seen any ablation study to directly compare this with Gini Deivation-based policy gradient method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review. We find the questions in \"Weaknesses\" and \"Questions\" are closely related. We combine the related ones and address main concerns below.\n\n1. **Regarding the draw back, bias and variance of the sample based estimator**\n\nAs discussed in the limitation section (section 6), the draw back of our sample based estimator is that we need a sufficient number of samples for each update due to estimating the CDF. However, we would also like to highlight that using multiple trajectory samples to estimate CDF or inverse CDF is also common in static CVaR related policy gradient methods, e.g., see papers [Optimizing the CVaR via sampling; Efficient risk-averse reinforcement learning].\n\nGenerally, rigorous characterizing the bias and variance of $\\eta_i$ in Equation 16 requires the true CDF, which is unknown. Also the bias and variance of $\\eta_i$ involves the bias and variance of ordered samples, which are also complex. We give an intuitive analysis first and then demonstrate by simutation.\n\nThe bias of $\\eta_i$ comes from the difference between the empirical CDF and the true CDF. This bias will decrease when the sample size increases as the empirical CDF will be closer to the true CDF. \n\nFor variance, we can compare our updating rule with REINFORCE. In REINFORCE, the return term is multiplied to the sum of log pi, while in GD policy gradient, $\\eta_i$ is multiplied to the sum of log pi. Compared with this return term, $\\eta_i$ is calculated based on the difference between ordered returns times a value in (0,1), which has a much smaller numerical scale, thus smaller variability.\n\nWe demonstrate the bias and variance by simulating from a truncated Gaussian. Suppose ordered samples $x_{(1)}, x_{(2)},...,x_{(n)}$ (i.e., the samples are sorted in increasing order) are from a truncated Gaussian in range [-25, 25] with mean 0 and variance $10^2$. For $\\eta_i=\\sum_{i=1}^{n-1} \\frac{2i}{n} (x_{(i+1)} - x_{(i)})-(x_{(n)}-x_{(i)})$, the bias is $\\delta_i = \\eta_i -  \\int_{x_{(i)}}^{25} 2F(t)dt + (25 - x_{(i)})$.\n    \nWe can estimate the bias as $B=\\mathbb{E}[\\frac{1}{n-1}\\sum_{i=1}^{n-1}\\delta_i]$. When $n=10$, $B\\approx -9.32$. When $n=50$, $B\\approx -4.53$. When $n=100$, $B\\approx -2.97$. The bias decreases as the sample size increases.\n\nWe can estimate the variance as $V=\\mathbb{E}[\\frac{1}{n-1}\\sum_{i=1}^{n-1}(\\eta_i - \\bar{\\eta})^2]$, where $\\bar{\\eta}=\\frac{1}{n-1}\\sum_{i=1}^{n-1}\\eta_i$. When $n=10$, $V\\approx 6.96$. When $n=50$, $V\\approx 10.79$. When $n=100$, $V\\approx 11.66$. This variance is much smaller than the variance of the sampling distribution, i.e, 100. Here we can treat $x_{(i)}$ as the return term used in REINFORCE and $\\eta_i$ be the term in GD gradeint. This indicates the variance of $\\eta_i$ in GD gradient is relatively small.\n\n2. **Regarding the complexity of the new proposed method?**\n\nSince we are updating gradients based on episodes, we suppose that computing the gradient for one trajectory is one time unit. Suppose the total number of training trajectories is $N$. Consider the mean-GD gradient built on top of PPO, based on importance sampling, the gradient for each trajectory is updated a fixed number of times $m$. Then the total complexity is $mN$ time units. \n\nWhen mean-GD gradient is built on top of the REINFORCE baseline, the complexity is in general between $N$ to $mN$ time units since the importance sampling update is terminated if the ratio is too extreme.\n\n3. **Regarding the intuition of GD be more robust to numerical scales than varaince**\n\nThanks for this good question. The intuition directly comes from the definitions of Gini deviation and variance as shown in Equation 6 and in Line 149. Thus $\\mathbb{V}[cX]=c^2\\mathbb{V}[X]$, while $\\mathbb{D}[cX]=c\\mathbb{D}[X]$ for $c>0$. We also highlight this property in Line 159. As a result, scaling the reward or the return will scale the Gini deviation linearly, but quadratically for variance.\n\n4. **Regarding compare GD and variance with respect to the sensitivity to numerical scales.**\n\nWe directly compare mean-variance with mean-Gini deviation in Section 5.1.  Following Line 299 in the paper, i.e. the sentence \"The failure of variance-based baselines under simple reward manipulation\", we provide an analysis of our experiments by modifying the numerical scale of the goal reward of the maze environment. This simple manipulation affects the mean-variance methods, but makes little difference to mean-Gini deviation as shown in Figure 2."}, {"review_id": "s8KXmmqOmA", "reviewer": "Reviewer_cQhS", "summary": "The paper proposes to restrict the Gini deviation of the return for risk-averse reinforcement learning (RARL). This is motivated by a discussion of the shortcomings of variance-based RARL methods, such as scale dependence and instability due to large squared returns. A derivation of an expression for the gradient of the Gini deviation of the return serves as a basis for a practical algorithm, mean-GD (MG). It adds a penalty proportional to the Gini deviation to the reinforcement learning objective. The distribution of returns, which is required for the gradient of the Gini deviation of the return, is approximated with a finite number of sampled trajectories. An empirical evaluation of MG in the tabular and function approximation settings shows advantages of MG in learning risk-averse policies over return and reward-based baselines.", "questions": "* It seems to me that the equality in line 234 is incorrect as the probability density of sampling a trajectory is not necessarily the same as the probability density of obtaining its reward. An extreme counter example would be an environment with a reward identical to zero but many possible trajectories. I do not think this poses a problem for the further derivation of MG as the set of trajectories that result in the same reward is independent of the policy parameters. However, it would still make sense to correct this formulation for better readability.\n* As the policy is defined to output values in $[0, 1]$ (line 48), does this mean discrete actions are assumed?\n* I would suggest to better relate the paragraph on the baselines (starting in line 281) to the discussion of prior methods in section 2. With the current formulation, it is not clear to the reader which baselines correspond to which of the methods discussed in section 2. Adding some clarification here would make interpreting the results a lot easier.\n* I would be interested in learning why most of the environments were specifically designed for this paper. Are no standard environments for RARL available that would have been useful?\n* Would it be possible to use the standard deviation instead of the variance to get scale invariance?\n* Is the variance of the estimate for the CDF in equation (16) a problem?\n* Related to the last question: Would it be possible to learn an approximation of the return distribution to reduce variance and get better convergence of the policy? The return distribution is clearly non-stationary but tracking non-stationary functions with estimates is not uncommon in reinforcement learning. Maybe this could help with asymptotic performance.\n* Why is MG performing poorly on HalfCheetah while MVPI seems to do well?", "limitations": "The limitations of the MG algorithm, mainly low sample efficiency, have been addressed in a separate paragraph.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of the Gini deviation as a new risk measure for RARL is motivated well by a concise discussion of the limitations of variance-based methods and is, to the best of my knowledge, new. The shortcomings of existing methods are illustrated with the help of a simple maze environment. The development of methods that are less sensitive to hyperparameters and environment properties such as reward scale is furthermore highly relevant for applications. \n\nThe derivation of a tractable expression for the gradient of the Gini deviation is done carefully and required assumptions are stated clearly. The experimental evaluation furthermore considers a sufficiently broad range of environments (tabular maze over lunar lander to MuJoCo environments) and baselines.", "weaknesses": "While the derivation of equations (15) and (16) are sufficiently clear, I would appreciate some intuition about the nature of the contribution of the Gini deviation penalty to the parameter update. The current version of the text does not really discuss the proposed algorithm but jumps right into the experimental evaluation. I think that for adoption of the algorithm by the community an intuitive understanding would be helpful.\n \nThere is furthermore a gap between the assumptions for the derivation and the actual environments used for the experiments and the illustration. The maze environment, for example, uses a shortest path objective which implies a discrete return. Many real-world environments will have complicated return distributions with atoms etc. It would be interesting to learn whether this poses a problem or if the assumptions are technical in nature and only required for the proof.\n\nThe significance of the experimental results is difficult to judge without further information. While I found the used hyperparameters in the supplementary material, I did not find information on how they were chosen. Was a hyperparameter optimization performed and, if yes, was it done manually, with a grid search or with some other algorithm? Without this information the frequent failures of the baselines are difficult to place. I would furthermore suggest to put vertical lines indicating the performance of MVPI in the figures (to avoid the problem of incompatible x axes). Right now it looks a bit like the most competitive baseline is only shown in the supplementary material where it is hard to compare it to the rest.\n\nRelated to this point: Part of the motivation for the proposed algorithm was to mitigate the need to extensively tune the hyperparameter $\\lambda$ for different environments/reward functions. It would therefore be interesting to see how sensitive MG as well as the baselines are to the hyperparameters controlling the risk-averseness.\n\nTypos: \n\n* In line 42 “We show that our method can learn risk-averse policy […]”, there is an “a” missing.\n* In line 69 the expression for the gradient of $M(\\theta)$ contains an $a$ and a $u(\\theta)$ which have not been introduced. Together with a different expression given in line 88, this looks like a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review. We address main concerns below.\n\n**For Weaknesses**\n\n1.**The current version does not really discuss the proposed algorithm but jumps right into the experiments.**\n\nThe full algorithms are in Section 9 in appendix due to page limit.\n\nThe procedures of the algorithms are 1. collect a group of trajectories 2. calculate the risk-neutral policy gradient 3. sort the trajectories by returns and calculate the Gini deviation policy gradient 4. update the policy by adding them together with a trade-off $\\lambda$.\n\n2.**Regarding the gap between the assumptions and the actual environments.**\n\nThis gap can be easily closed since we can always add a truncated Gaussian noise to the reward to make the return continuous without changing the optimal solution. This assumption for continuity of the return variable is also common in the literature, e.g., see Section 2 of [Optimizing the CVaR via Sampling].\n\n3.**Regarding the hyperparameters.**\n\nSince most of the comparing methods are also compared in the MVPI paper, e.g. Tamar, MVP, MVPI, TD3, we take the parameter searching range in MVPI as a reference and manually search by ourselves. We ensure each algorithm sweeps over the same number of hyperparameter combinations to make a fair comparison. We will add more detials in the revised version.\n\n4.**Right now it looks a bit like the most competitive baseline is only shown in the supplementary material.**\n\nThanks for raising this concern. MVPI is a TD learning method, i.e., update paremters at each environment step. Thus, it is not straightfoward to be compared with episode based methods. In addition, the MVPI plots were in appendix mainly due to the page limit. If the paper is accepted, we will move the figure to the main content using the extra page for accepted papers.\n\n5.**Regading the sensitivity to lambda**\n\nWe show the learning curves of total return based methods, e.g., MVO, Tamar, MVP, MG with different lambdas in the Maze domain in the uploaded PDF file.\n\n**For Questions**\n\n1.**Regarding equation in line 234. An extreme counter example would be an environment with a reward identical to zero but many possible trajectories.**\n\nWe think this one is closely related to the previous question on the continuous assumption of the return variable. The counter example here conflicts with this assumption. \n\n2.**As the policy is defined in [0, 1](line 48), do you assume discrete actions?**\n\nWe apologize for this typo. We do not assume the actions to be discrete. We will correct this in the next version.\n\n3.**Regarding the environment design, and standard environments for RARL?**\n\nThe reason for specific design is that the evaluations in previous mean-variance papers are either done (a) in small domains but without a clear understanding of why a particular type of algorithm can succeed or fail (we did this in Maze) or (b) in large domains but verifying the risk-aversion is not very straightforward (e.g., MVPI adds action noise to Mujoco. It reports the mean-variance score but it is unclear what  a risk averse policy is or whether risk-aversion is achieved). Thus we modify several domains where the risk-aversion can be clearly defined to better evaluate each algorithm.\n\nFor standard environments for RARL, as far as we know, there is no common benchmark in all previous mean-variance papers in Section 2. For example, Tamar's paper used portofolio. MVP's paper used portofolio, american-style option, and optimal stopping. MVPI's paper used Mujoco with noisy action.\n\n4.**Regarding using standard deviation to get scale invariance**\n\nFirst, tough Std is related to the signed Choquet integral, it is a supremum over some signed Choquet integrals, e.g., see Example 3 in the paper [Characterization, Robustness and Aggregation of Signed Choquet Integrals]. The gradient calculation is in general hard.\n\nSecond, directly computing the gradient of Std is possible. $Std = Var^{\\frac{1}{2}}$, then $\\nabla Std = \\frac{1}{2}\\frac{1}{\\sqrt{Var}}\\nabla Var$. This gradient may still suffer from numerical scale issues due to the $\\nabla Var$ term.\n\n5.**Regarding the variance of the estimator**\nTo get an intuitive understanding of the variance, we can compare our updating rule with REINFORCE. In REINFORCE, the return term is multiplied to the sum of log pi.\n\nThe variance of $\\eta_i$ depends on the variance of ordered samples, which may not have a closed form. However, compared with the return in REINFORCE, $\\eta_i$ is calculated based on the difference between ordered returns times a value in (0,1), which has a much smaller numerical scale, thus smaller variability.\n\n6.**Would it be possible to learn an approximation of the return distribution to reduce variance and get better convergence?**\n\nThanks for this suggestion. Learning a return distribution function can leverage distributional RL. However, due to the non time consistency of the risk term, incorporating TD method (distributional RL uses TD learning) is challenging, since optimizing risk at each time step is usually not the same as optimizing the total return risk at the initial state. We postpone this part as our future work.\n\n7.**Why is MG performing poorly on HalfCheetah while MVPI seems to do well?**\n\nWe designed two HalfCheetah domains (the second in Appendix).\n\nOne is adding a noisy reward based on X-location (X<-3). In this domain, MVPI performs good since (a) MVPI is built on top of TD3. MG is built on top of PPO. TD3 learns faster and better than PPO in this domain as shown in Figure 5 of TD3 paper [Addressing Function Approximation Error in Actor-Critic Methods]. (b) MVPI's reward modification strategy naturally prevents it from visiting the X<-3 region.\n\nThe other is adding linearly decayed noisy reward based on the distance the agent has covered. The distance that agents covered is in Figure 19 (Section 11). It is clear that MVPI gets stuck in this domain, i.e., final X location is close to the origin."}, {"review_id": "Q1cH5GsM6t", "reviewer": "Reviewer_oAim", "summary": "This paper considers the problem of risk-sensitive reinforcement learning in the policy optimization setting, under the mean-variance risk criteria. They provide background on the problem and demonstrate the issues with learning an optimal policy due to the estimation of the variance requiring double samples. They demonstrate that previous approaches to this problem which use the immediate reward variance as a proxy are not desirable (Section 2). They introduce the Gini deviation as a surrogate for the variance, and demonstrate that through a Choquet integral characterization, it can be estimated without the double-sampling issue by leveraging quantile distributional RL. They incorporate this into a PPO-style algorithm and compare its performance against others across a range of risk-sensitive benchmarks.", "questions": "- What is the space $\\mathcal{M}^\\infty$ introduced in Definition 1? It is never defined.", "limitations": "The authors discuss limitations of their work, and propose directions for future work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Well presented motivation and contribution.\n- Novel idea, which achieves strong empirical performance.", "weaknesses": "- There is little theoretical support for the algorithm, which would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review. We address main concerns below.\n\n+ **What is the space $M^\\infty$ introduced in Definition 1?**\n\nWe apologize for this typo. It should be $L^\\infty$ to represent the set of bounded random variables, e.g., see original definition in Equation 1 of the paper [Characterization, Robustness and Aggregation of Signed Choquet Integrals]."}, {"review_id": "OmbsMbYF6Q", "reviewer": "Reviewer_Dz1b", "summary": "The paper points to limitations of the mean-variance criterion for risk-averse RL, and of the methods that optimize it. Instead, the paper proposes an alternative risk measure relying on Gini Deviation. The paper shows how to derive the PG for this metric, and demonstrates that it learns risk-averse policies more stably than methods that optimize the mean-variance.", "questions": "See the issues discussed above under Weaknesses.\n\nAdditional question:\nAs specified in the paper, the current method uses multiple trajectories for a single policy gradient. However, quantile regression (as in QRDQN) allows to learn quantile value representation without this limitation. Could such quantile regression be used to overcome the multi-trajectory limitation in mean-GD as well?", "limitations": "Algorithmic limitations are fairly discussed in Section 6.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using Gini Deviation for risk-averse RL is original as far as I know.\n2. The metric is demonstrated as more stable than mean-variance.", "weaknesses": "Risk-aversion in RL is often handled by optimizing risk measures. Many risk measures have been studied, e.g., mean-variance, var, cvar, entropic, distortion measures (some of them are briefly mentioned in the paper). There are studies about both the properties of different measures and how to optimize them. In particular, commonly desired properties of the metric are captured by the notion of coherent risk measures. For example, mean-variance is not a coherent risk measure, but CVaR is, and many works have recently studied it.\n\nA paper that proposes a new risk measure and claims to its benefits cannot ignore these works and the notion of coherence.\n\nConsidering the discussion above:\n1. The presentation of the paper contribution lacks a lot of relevant context, hence the novelty and contribution are difficult to judge.\n2. Relevant literature is hardly covered. There is no organized literature survey, and relevant works are not mentioned.\n3. The experiments mostly demonstrate the weakness of the mean-variance risk measure, which is already known and cannot justify the suggested metric just by itself. Since the paper justification is essentially empirical, this is a quite critical issue.\n\nFinally, the maze benchmark used in Sec 2.2, Sec 5.1 and Fig 1 is taken directly from [Greenberg et al. 2022], with neither crediting its original nor citing the corresponding work. Notice that using benchmarks and code (even if you modify them) without explicitly citing them in both paper and code may be considered Plagiarism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. We address main concerns below.\n\n**For Weaknesses**\n\n1. **Regarding the notion of coherence.**\n\nThanks for pointing out the coherent risk measure. We are aware of this category of risk as mentioned in Section 1. However, as indicated by our title and abstract, we are aiming to address some limitations of variance related risk measures in this paper, instead of comparing with other coherent risk measures.\n\nBoth variance and Gini deviation are measures of variability, while CVaR (a coherent risk measure) is a measure for extreme outcomes. They are targeting different application scenarios and users. \n\nGini deviation is known as a **coherent measure of variability** (Line 161), e.g., see [22](Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks). The properties of coherent measure of variability are different from the properties of coherent measure of risk, e.g., see Section 2.3 of [22].  We will briefly compare them in the revised version.\n\n2. **Regarding lacks a lot of relevant context, and relevant literature is hardly covered.**\n\nSince our paper focuses on the variance related risk measures, i.e., variability. We give a detailed analysis of the main stream mean-variance methods in Section 2. \n\n3. **The experiments mostly demonstrate the weakness of the mean-variance risk measure, which is already known and cannot justify the suggested metric just by itself.**\n\nThanks for raising this concern. We think there may be a misunderstanding.  It is already known that variance is not a coherent risk measure, but this is not the weakness that we demonstrate nor try to resolve.  First we would like to invite the reviewer to read Section 2 of [22] (Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks).  We focus on measures of variability (instead of measures of risk such as CVaR) as formally defined in [22].  Measures of variability are location invariant and consider the entire distribution of outcome instead of the tails.  They include variance and Gini-deviation, but not CVaR.  The confusion stems from the fact that the term \"risk\" is used casually in RL to refer to both measures of risk and variability.  However in this work we demonstrate that variance is not positive homogeneous (Line 161) and then propose to use Gini-deviation as a replacement since it is positive homogeneous and a coherent measure of variability (but not a coherent measure of risk).  We will add text to that effect in the paper.\n\n4. **Regarding the missing citation of [Greenberg et al. 2022]**\n\nWe acknowledge that the maze problem that we described in Figure 1 is a modified version of the Guarded Maze problem in [Greenberg et al. 2022] and that we failed to cite their work in our paper and our code.  We sincerely regret this omission.  While we had no intention of misappropriating their work, we now realize that this mistake may be seen as a form of plagiarism.  We are taking this seriously and we have resolved this issue directly with the authors.  The authors contacted us separately and independently of the NeurIPS reviews to point out the missing citation and work attribution.  We exchanged several emails with the authors and quickly fixed this mistake.  We ran by the authors a revised version of the manuscript where we did the following changes:\n\n1) In Section 1, second paragraph, we added a citation to [Greenberg et al., 2022]\n2) In Section 1, last paragraph, we changed \"we created several domains\" to \"we modified several domains (Guarded Maze [7], Lunar Lander [15], Mujoco [16])\".\n3) In Figure 1: we indicated that the environment is a modified version of Guarded Maze and added a citation to [Greenberge el al., 2022]\n4) In Section 5.1, we added the following explanation.  \"The original Guarded Maze problem is asymmetric with two openings to reach the top path (in contrast to a single opening for the bottom path).  In addition, paths via the top tend to be longer than paths via the bottom. We modified the maze to be more symmetric in order to reduce preferences arising from certain exploration strategies that might be biased towards shorter paths or greater openings, which may confound risk aversion.\"\n5) In the appendix, we added a discussion about additional related work in which we cited [Greenberg et al., 2022]\n\nAfter inspecting those changes, Greenberg wrote: \"Thank you for your recent messages and for revising the manuscript. I am glad to see that our paper could help you in your very interesting work. The attribution in the revised paper that you sent seems entirely proper to me. I will be happy to see you in a conference.\"  Hence, if our paper is accepted, we will incorporate the changes summarized above that have been approved by Greenberg.\n\n\n**For Questions**\n\n1. **Reagrding using QRDQN to learn quantile value representation**\n\nThanks for this good question. QRDQN is a TD learning method for risk neutral decision making. Since most risk terms are not time consistent, i.e., optimizing risk at each time step is usually not the same as optimizing the total return risk at the initial state, incorporating quantile TD learning is thus challenging. We postpone this part to future work."}]

Author rebuttal:
We include a PDF file which contains additional results requested by reviewer cQhS, regarding the sensitivity to $\lambda$

Meta-review: The authors investigated the application of the Gini-deviation (equation 6) risk measure in risk-averse Reinforcement Learning (RL). To do so, additional objective was added to the standard average cumulative reward objective of the RL problem. Further, a policy gradient based algorithm, which minimizes the new objective (equation 9), was implemented and tested. By conducting a thorough experimental study the advantages of the Gini-deviation objective was highlighted in the context of RL, compared to other previously suggested risk-averse measures.

The reviewers were in a consensus for the acceptance of this paper due to the novelty of combining the Gini-deviation risk measure in RL, the derivation of a policy gradient based algorithm for the objective, and the empirical study the authors conducted. Constructive feedback, given by the reviewers during the discussion period, will be combined in the final version of the paper. For these reasons I am pleased to support the acceptance of this work.
Decision: Accept (poster)