Title: An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient
Venue: NeurIPS 2023
Accepted: yes
Keywords: risk-averse RL

mean-variance RL
Abstract: Restricting the variance of a policy’s return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient

Abstract

Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

1 Introduction

The demand for avoiding risks in practical applications has inspired risk-averse reinforcement learning (RARL). For example, we want to avoid collisions in autonomous driving [1], or avoid huge financial losses in portfolio management [2]. In addition to conventional RL, which finds policies to maximize the expected return [3], RARL also considers the control of risk.

Many risk measures have been studied for RARL, for instance, exponential utility functions [4], value at risk (VaR) [5], conditional value at risk (CVaR) [6, 7], and variance [8, 9]. In this paper, we mainly focus on the variance-related risk measures given their popularity, as variance has advantages in interpretability and computation [10, 11]. Such a paradigm is referred to as mean-variance RL. Traditional mean-variance RL methods consider the variance of the total return random variable. Usually, the total return variance is treated as a constraint to the RL problem, i.e., it is lower than some threshold [8, 9, 12]. Recently, [13] proposed a reward-volatility risk measure, which considers the variance of the per-step reward random variable. [13] shows that the per-step reward variance is an upper bound of the total return variance and can better capture the short-term risk. [14] further simplifies [13]'s method by introducing Fenchel duality.

Directly optimizing total return variance is challenging. It either necessitates double sampling [8] or calls for other techniques to avoid double sampling for faster learning [8, 9, 12]. As for the reward-volatility risk measure, [13] uses a complicated trust region optimization due to the modified reward's policy-dependent issue. [14] overcomes this issue by modifying the reward according to Fenchel duality. However, this reward modification strategy can possibly hinder policy learning by changing a 'good' reward to a 'bad' one, which we discuss in detail in this work.

To overcome the limitations of variance-based risk measures, we propose to use a new risk measure: Gini deviation (GD). We first review the background of mean-variance RL. Particularly, we explain the limitations of both total return variance and per-step reward variance risk measures. We then introduce

GD as a dispersion measure for random variables and highlight its properties for utilizing it as a risk measure in policy gradient methods. Since computing the gradient using the original definition of GD is challenging, we derive the policy gradient algorithm from its quantile representation to minimize it. To demonstrate the effectiveness of our method in overcoming the limitations of variance-based risk measures, we modify several domains (Guarded Maze [7], Lunar Lander [15], Mujoco [16]) where risk-aversion can be clearly verified. We show that our method can learn risk-averse policy with high return and low risk in terms of variance and GD, when others fail to learn a reasonable policy.

2 Mean-Variance Reinforcement Learning

In standard RL settings, agent-environment interactions are modeled as a Markov decision process (MDP), represented as a tuple ( S , A , R, P, µ 0 , γ ) [17]. S and A denote state and action spaces. P ( ·| s, a ) defines the transition. R is the state and action dependent reward variable, µ 0 is the initial state distribution, and γ ∈ (0 , 1] is the discount factor. An agent follows its policy π : S × A → [0 , + ∞ ) . The return at time step t is defined as G t = ∑ ∞ i =0 γ i R ( S t + i , A t + i ) . Thus, G 0 is the random variable indicating the total return starting from the initial state following π .

Mean-variance RL aims to maximize E [ G 0 ] and additionally minimize its variance V [ G 0 ] [8, 9, 12]. Generally, there are two ways to define a variance-based risk. The first one defines the variance based on the Monte Carlo total return G 0 . The second defines the variance on the per-step reward R . We review these methods and their limitations in the following subsections. We will refer to π , π θ and θ interchangeably throughout the paper when the context is clear.

2.1 Total Return Variance

Methods proposed by [8, 9, 12] consider the problem

where ξ indicates the user's tolerance of the variance. Using the Lagrangian relaxation procedure [18], we can transform it to the following unconstrained optimization problem: max π E [ G 0 ] -λ V [ G 0 ] , where λ is a trade-off hyper-parameter. Note that the mean-variance objective is in general NPhard [19] to optimize. The main reason is that although variance satisfies a Bellman equation, it lacks the monotonicity of dynamic programming [20].

Double Sampling in total return variance. We first show how to solve unconstrained mean-variance RL via vanilla stochastic gradient. Suppose the policy is parameterized by θ , define J ( θ ) = E π [ G 0 ] and M ( θ ) := E π [ ( ∑ ∞ t =0 γ t R ( S t , A t )) 2 ] , then V [ G 0 ] = M ( θ ) -J 2 ( θ ) . The unconstrained meanvariance objective is equivalent to J λ ( θ ) = J ( θ ) -λ ( M ( θ ) -J 2 ( θ ) ) , whose gradient is

The unbiased estimates for ∇ θ J ( θ ) and ∇ θ M ( θ ) can be estimated by approximating the expectations over trajectories by using a single set of trajectories, i.e., ∇ θ J ( θ ) = E τ [ R τ ω τ ( θ )] and ∇ θ M ( θ ) = E τ [ R 2 τ ω τ ( θ )] , where R τ is the return of trajectory τ and ω τ ( θ ) = ∑ t ∇ θ log π θ ( a t | s t ) . In contrast, computing an unbiased estimate for J ( θ ) ∇ θ J ( θ ) requires two distinct sets of trajectories to estimate J ( θ ) and ∇ θ J ( θ ) separately, which is known as double sampling.

Remark. Some work claims that double sampling cannot be implemented without having access to a generative model of the environment that allows users to sample at least two next states [12]. This is, however, not an issue in our setting where we allow sampling multiple trajectories. As long as we get enough trajectories, estimating J ( θ ) ∇ θ J ( θ ) is possible.

Still, different methods were proposed to avoid this double sampling for faster learning. Specifically, [8] considers the setting γ = 1 and considers an unconstrained problem: max θ L 1 ( θ ) = E [ G 0 ] -λg ( V [ G 0 ] -ξ ) , where λ > 0 is a tunable hyper-parameter, and penalty function g ( x ) = (max { 0 , x } ) 2 . This method produces faster estimates for E [ G 0 ] and V [ G 0 ] and a slower updating for θ at each episode, which yields a two-time scale algorithm. [9] considers the setting γ < 1 and converts Formula 1 into an unconstrained saddle-point problem: max λ min θ L 2 ( θ, λ ) = -E [ G 0 ] + λ ( V [ G 0 ] -ξ ) , where λ is the dual variable. This approach uses perturbation method and smoothed function method to compute the gradient of value functions with respect to policy parameters. [12] considers the setting γ = 1 , and introduces Fenchel duality x 2 = max y (2 xy -y 2 ) to avoid the term J ( θ ) ∇ θ J ( θ ) in the gradient. The original problem is then transformed into max θ,y L 3 ( θ, y ) = 2 y ( E [ G 0 ] + 1 2 λ ) -y 2 -E [ G 2 0 ] , where y is the dual variable.

Limitations of Total Return Variance. The presence of the square term R 2 τ in the mean-variance gradient ∇ θ M ( θ ) = E τ [ R 2 τ ω τ ( θ )] (Equation 2) makes the gradient estimate sensitive to the numerical scale of the return, as empirically verified later. This issue is inherent in all methods that require computing ∇ θ E [ G 2 0 ] . Users can not simply scale the reward by a small factor to reduce the magnitude of R 2 τ , since when scaling reward by a factor c , E [ G 0 ] is scaled by c but V [ G 0 ] is scaled by c 2 . Consequently, scaling the reward may lead to different optimal policies being obtained.

2.2 Per-step Reward Variance

A recent perspective uses per-step reward variance V [ R ] as a proxy for V [ G 0 ] . The probability mass function of R is Pr( R = x ) = ∑ s,a d π ( s, a ) I r ( s,a )= x , where I is the indicator function, and d π ( s, a ) = (1 -γ ) ∑ ∞ t =0 γ t Pr( S t = s, A t = a | π, P ) is the normalized discounted state-action distribution. Then we have E [ R ] = (1 -γ ) E [ G 0 ] and V [ G 0 ] ≤ V [ R ] (1 -γ ) 2 (see Lemma 1 of [13]). Thus, [13] considers the following objective

This objective can be cast as a risk-neutral problem in the original MDP, but with a new reward function ˆ r ( s, a ) = r ( s, a ) -λ ( r ( s, a ) -(1 -γ ) E [ G 0 ] ) 2 . However, this ˆ r ( s, a ) is nonstationary (policy-dependent) due to the occurrence of E [ G 0 ] , so standard risk-neutral RL algorithms cannot be directly applied. Instead, this method uses trust region optimization [21] to solve.

[14] introduces Fenchel duality to Equation 4. The transformed objective is ˆ J λ ( π ) = E [ R ] -λ E [ R 2 ] + λ max y (2 E [ R ] y -y 2 ) , which equals to max π,y J λ ( π, y ) = ∑ s,a d π ( s, a ) ( r ( s, a ) -λr ( s, a ) 2 +2 λr ( s, a ) y ) -λy 2 . The dual variable y and policy π are updated iteratively. In each inner loop k , y has analytical solution y k +1 = ∑ s,a d π k ( s, a ) r ( s, a ) = (1 -γ ) E π k [ G 0 ] since it is quadratic for y . After y is updated, learning π is a risk-neutral problem in the original MDP, but with a new modified reward

Since ˆ r ( s, a ) is now stationary, any risk-neutral RL algorithms can be applied for policy updating.

Limitations of Per-step Reward Variance. 1) V [ R ] is not an appropriate surrogate for V [ G 0 ] due to fundamentally different implications. Consider a simple example. Suppose the policy, the transition dynamics and the rewards are all deterministic, then V [ G 0 ] = 0 while V [ R ] is usually nonzero unless all the per-step rewards are equal. In this case, shifting a specific step reward by a constant will not affect V [ G 0 ] and should not alter the optimal risk-averse policy. However, such shift can lead to a big difference for V [ R ] and may result in an invalid policy as we demonstrated in later example. 2) Reward modification hinders policy learning. Since the reward modifications in [13] (Equation 4) and [14] (Equation 5) share the same issue, here we take Equation 5 as an example. This modification is likely to convert a positive reward to a much smaller or even negative value due to the square term, i.e. -λr ( s, a ) 2 . In addition, at the beginning of the learning phase, when the policy performance is not good, y is likely to be negative in some environments (since y relates to E [ G 0 ] ). Thus, the third term 2 λr ( s, a ) y decreases the reward value even more. This prevents the agent to visit the good (i.e., rewarding) state even if that state does not contribute any risk. These two limitations raise a great challenge to subtly choose the value for λ and design the reward for the environment.

Empirical demonstration of the limitations . Consider a maze problem (a modified version of Guarded Maze [7]) in Figure 1. Starting from the bottom left corner, the agent aims to reach the green goal state. The gray color corresponds to walls. The rewards for all states are deterministic (i.e., -1 ) except for the red state whose reward is a categorical distribution with mean -1 . The reward for visiting the goal is a positive constant value. To reach the goal, a risk-neutral agent prefers the path at the bottom that goes through the red state, but V [ G 0 ] will be nonzero. A risk-averse agent prefers the white path in the figure even though E [ G 0 ] is slightly lower, but V [ G 0 ] = 0 . Per-step reward variance methods aim to use V [ R ] as a proxy of V [ G 0 ] . For the risk-averse policy leading to the white path, ideally, increasing the goal reward by a constant will not effect V [ G 0 ] , but will make a big difference to V [ R ] . For instance, when the goal reward is 10 , V [ R ] = 10 . When goal reward is 20 , V [ R ] ≈ 36 . 4 ,

which is much more risk-averse. Next, consider the reward modification (Equation 5) for the goal

reward when it is 20 . The square term in Equation 5 is -400 λ . It is very easy to make the goal reward negative even for small λ , e.g., 0 . 1 . We do find this reward modification prevents the agent from reaching the goal in our experiments.

3 Gini Deviation as an Alternative of Variance

To avoid the limitations of V [ G 0 ] and V [ R ] we have discussed, in this paper, we propose to use Gini deviation as an alternative of variance. Also, since GD has a similar definition and similar properties as variance, it serves as a more reasonable proxy of V [ G 0 ] compared to V [ R ] .

3.1 Gini Deviation: Definition and Properties

GD [22], also known as Gini mean difference or mean absolute difference, is defined as follows. For a random variable X , let X 1 and X 2 be two i.i.d. copies of X , i.e., X 1 and X 2 are independent and follow the same distribution as X . Then GD is given by

Variance can be defined in a similar way as V [ X ] = 1 2 E [( X 1 -X 2 ) 2 ] .

Given samples { x 1 i } n i =1 from X 1 and { x 2 j } n j =1 from X 2 . The unbiased empirical estimations for GD and variance are ˆ D [ X ] = 1 2 n 2 ∑ n i =1 ∑ n j =1 | x 1 i -x 2 j | and ˆ V [ X ] = 1 2 n 2 ∑ n i =1 ∑ n j =1 ( x 1 i -x 2 j ) 2 .

Both risk profiles aim to measure the variability of a random variable and share similar properties [23]. For example, they are both location invariant, and can be presented as a weighted sum of order statistics. [23] argues that the GD is superior to the variance as a measure of variability for distributions far from Gaussian. We refer readers to this paper for a full overview. Here we highlight two properties of D [ X ] to help interpret it. Let M denote the set of real random variables and let M p , p ∈ [1 , ∞ ) denote the set of random variables whose probability measures have finite p -th moment, then

V [ X ] ≥ √ 3 D [ X ] for all X ∈ M 2 .

D [ cX ] = c D [ X ] for all c > 0 and X ∈ M .

The first property is known as Glasser's inequality [24], which shows D [ X ] is a lower bound of V [ X ] if X has finite second moment. The second one is known as positive homogeneity in coherent measures of variability [25], and is also clear from the definition of GD in Equation 6. In RL, considering X is the return variable, this means GD is less sensitive to the reward scale compared to variance, i.e., scaling the return will scale D [ X ] linearly, but quadratically for V [ X ] . We also provide an intuition of the relation between GD and variance from the perspective of convex order, as shown in Appendix 7. Note also that while variance and GD are both measures of variability, GD is a coherent measure of variability [25]. Appendix 12 provides a discussion of the properties of coherent measures of variability , while explaining the differences with coherent measures of risk such as conditional value at risk (CVaR).

3.2 Signed Choquet Integral for Gini Deviation

This section introduces the concept of signed Choquet integral, which provides an alternative definition of GD and makes gradient-based optimization convenient. Note that with the original definition (Equation 6), it can be intractable to compute the gradient w.r.t. the parameters of a random variable's density function through its GD.

The Choquet integral [26] was first used in statistical mechanics and potential theory and was later applied to decision making as a way of measuring the expected utility [27]. The signed Choquet integral belongs to the Choquet integral family and is defined as:

Definition 1 ([28], Equation 1) . A signed Choquet integral Φ h : X → R , X ∈ L ∞ is defined as

where L ∞ is the set of bounded random variables in a probability space, h is the distortion function and h ∈ H such that H = { h : [0 , 1] → R , h (0) = 0 , h is of bounded variation } .

This integral has become the building block of law-invariant risk measures 1 after the work of [29, 30]. One reason for why signed Choquet integral is of interest to the risk research community is that it

is not necessarily monotone. Since most practical measures of variability are not monotone, e.g., variance, standard deviation, or deviation measures in [31], it is possible to represent these measures in terms of Φ h by choosing a specific distortion function h .

Lemma 1 ([28], Section 2.6) . Gini deviation is a signed Choquet integral with a concave h given by h ( α ) = -α 2 + α, α ∈ [0 , 1] .

This Lemma provides an alternative definition for GD, i.e., D [ X ] = ∫ ∞ -∞ h ( Pr( X ≥ x ) ) dx, h ( α ) = -α 2 + α . However, this integral is still not easy to compute. Here we turn to its quantile representation for easy calculation.

Lemma 2 ([28], Lemma 3) . Φ h ( X ) has a quantile representation. If F -1 X is continuous, then Φ h ( X ) = ∫ 0 1 F -1 X (1 -α ) dh ( α ) , where F -1 X is the quantile function (inverse CDF) of X.

Combining Lemma 1 and 2, D [ X ] can be computed alternatively as

With this quantile representation of GD, we can derive a policy gradient method for our new learning problem in the next section. It should be noted that variance cannot be directly defined by a Φ h -like quantile representation, but as a complicated related representation: V [ X ] = sup h ∈H { Φ h ( X ) -1 4 ∥ h ′ ∥ 2 2 } , where ∥ h ′ ∥ 2 2 = ∫ 1 0 ( h ′ ( p )) 2 dp if h is continuous, and ∥ h ′ ∥ 2 2 := ∞ if it is not continuous (see Example 2.2 of [32]). Hence, such representation of the conventional variance measure is not readily usable for optimization.

4 Policy Gradient for Mean-Gini Deviation

In this section, we consider a new learning problem by replacing the variance with GD. Specifically, we consider the following objective

where λ is the trade-off parameter. To maximize this objective, we may update the policy towards the gradient ascent direction. Computing the gradient for the first term has been widely studied in risk-neutral RL [3]. Computing the gradient for the second term may be difficult at the first glance from its original definition, however, it becomes possible via its quantile representation (Equation 8).

4.1 Gini Deviation Gradient Formula

We first give a general gradient calculation for GD of a random variable Z , whose distribution function is parameterized by θ . In RL, we can interpret θ as the policy parameters, and Z as the return under that policy, i.e., G 0 . Denote the Probability Density Function (PDF) of Z as f Z ( z ; θ ) . Given a confidence level α ∈ (0 , 1) , the α -level quantile of Z is denoted as q α ( Z ; θ ) , and given by

For technical convenience, we make the following assumptions, which are also realistic in RL.

Assumption 1. Z is a continuous random variable, and bounded in range [ -b, b ] for all θ .

Assumption 2. ∂ ∂θ i q α ( Z ; θ ) exists and is bounded for all θ , where θ i is the i -th element of θ .

Assumption 3. ∂f Z ( z ; θ ) ∂θ i /f Z ( z ; θ ) exists and is bounded for all θ, z . θ i is the i -th element of θ .

Since Z is continuous, the second assumption is satisfied whenever ∂ ∂θ i f Z ( z ; θ ) is bounded. These assumptions are common in likelihood-ratio methods, e.g., see [33]. Relaxing these assumptions is possible but would complicate the presentation.

Proposition 1. Let Assumptions 1, 2, 3 hold. Then

Proof. By Equation 8, the gradient of D [ Z θ ] = Φ h ( Z θ ) ( h ( α ) = -α 2 + α, α ∈ [0 , 1] ) is

This requires to calculate the gradient for any α -level quantile of Z θ , i.e., ∇ θ q α ( Z ; θ ) . Based on the

assumptions and the definition of the α -level quantile, we have ∫ q α ( Z ; θ ) -b f Z ( z ; θ ) dz = α . Taking a derivative and using the Leibniz rule we obtain

Rearranging the term, we get ∇ θ q α ( Z ; θ ) = -∫ q α ( Z ; θ ) -b ∇ θ f Z ( z ; θ ) dz · [ f Z ( q α ( Z ; θ ); θ )] -1 . Plugging back to Equation 12 gives us an intermediate version of ∇ θ D [ Z θ ] .

By switching the integral order of Equation 14 and applying ∇ θ log( x ) = 1 x ∇ θ x , we get the final gradient formula Equation 11. The full calculation is in Appendix 8.1.

4.2 Gini Deviation Policy Gradient via Sampling

In a typical application, Z in Section 4.1 would correspond to the performance of a system, e.g., the total return G 0 in RL. Note that in order to compute Equation 11, one needs access to ∇ θ log f Z ( z ; θ ) : the sensitivity of the system performance to the parameters θ . Usually, the system performance is a complicated function and calculating its probability distribution is intractable. However, in RL, the performance is a function of trajectories. The sensitivity of the trajectory distribution is often easy to compute. This naturally suggests a sampling based algorithm for gradient estimation.

Now consider Equation 11 in the context of RL, i.e., Z = G 0 and θ is the policy parameter.

To sample from the total return variable G 0 , we need to sample a trajectory τ from the environment by executing π θ and then compute its corresponding return R τ := r 1 + γr 2 + ... + γ T -1 r T , where r t is the per-step reward at time t , and T is the trajectory length. The probability of the sampled return can be calculated as f G 0 ( R τ ; θ ) = µ 0 ( s 0 ) ∏ T -1 t =0 [ π θ ( a t | s t ) p ( r t +1 | s t , a t )] . The gradient of its log-likelihood is the same as that of P ( τ | θ ) = µ 0 ( s 0 ) ∏ T -1 t =0 [ π θ ( a t | s t ) p ( s t +1 | s t , a t )] , since the difference in transition probability does not alter the policy gradient. It is well known that ∇ θ log P ( τ | θ ) = ∑ T -1 t =0 ∇ θ log π θ ( a t | s t ) .

For the integral part of Equation 15, it requires the knowledge of the CDF of G 0 . In practice, this means we should obtain the full value distribution of G 0 , which is usually not easy. One common approach to acquire an empirical CDF or quantile function (inverse CDF) is to get the quantile samples of a distribution and then apply some reparameterization mechanism. For instance, reparameterization is widely used in distributional RL for quantile function estimation. The quantile function has been parameterized as a step function [34, 35], a piece-wise linear function [36], or other higher order spline functions [37]. In this paper, we use the step function parameterization given its simplicity. To do so, suppose we have n trajectory samples { τ i } n i =1 from the environment and their corresponding returns { R τ i } n i =1 , the returns are sorted in ascending order such that R τ 1 ≤ R τ 2 ≤ ... ≤ R τ n , then each R τ i is regarded as a quantile value of G 0 corresponding to the quantile level α i = 1 2 ( i -1 n + i n ) , i.e., we assume q α i ( G 0 ; θ ) = R τ i . This strategy is also common in distributional RL, e.g., see [34, 38]. The largest return R τ n is regarded as the upper bound b in Equation 15.

Thus, given ordered trajectory samples { τ i } n i =1 , an empirical estimation for GD policy gradient is (a detailed example is given in Appendix 8.2)

The sampled trajectories can be used to estimate the gradient for E [ G 0 ] in the meantime, e.g., the well known vanilla policy gradient (VPG), which has the form E τ [ R τ ∑ T -1 t =0 ∇ θ log π θ ( a t | s t )] . It is more often used as E τ [ ∑ T -1 t =0 ∇ θ log π θ ( a t | s t ) γ t g t ] , where g t = ∑ T -1 t ′ = t γ t ′ -t r ( s t ′ , a t ′ ) , which is known to have lower variance. Usually g t is further subtracted by a value function to improve stability, called REINFORCE with baseline. Apart from VPG, another choice to maximize E [ G 0 ] is using PPO [39].

4.3 Incorporating Importance Sampling

For on-policy policy gradient, samples are abandoned once the policy is updated, which is expensive for our gradient calculation since we are required to sample n trajectories each time. To improve

the sample efficiency to a certain degree, we incorporate importance sampling (IS) to reuse samples for multiple updates in each loop. For each τ i , the IS ratio is ρ i = ∏ T -1 t =0 π θ ( a i,t | s i,t ) /π ˆ θ ( a i,t | s i,t ) , where ˆ θ is the old policy parameter when { τ i } n i =1 are sampled. Suppose the policy gradient for maximizing E [ G 0 ] is REINFORCE baseline. With IS, the empirical mean-GD policy gradient is

where g i,t is the sum of rewards-to-go as defined above. V ( s i,t ) is the value function. The first part can also be replaced by PPO-Clip policy gradient. Then we have

where A i,t is the advantage estimate, and f () is the clip function in PPO with ϵ being the clip range, i.e. f ( ϵ, A i,t ) = clip( π θ ( a i,t | s i,t ) π ( a | s ) , 1 -ϵ, 1 + ϵ ) A i,t .

The extreme IS values ρ i will introduce high variance to the policy gradient. To stabilize learning, one strategy is that in each training loop, we only select τ i whose ρ i lies in [1 -δ, 1+ δ ] , where δ controls the range. The updating is terminated if the chosen sample size is lower than some threshold, e.g., β · n, β ∈ (0 , 1) . Another strategy is to directly clip ρ i by a constant value ζ , i.e., ρ i = min( ρ i , ζ ) , e.g., see [40]. In our experiments, we use the first strategy for Equation 17, and the second for Equation 18. We leave other techniques for variance reduction of IS for future study. The full algorithm that combines GD with REINFORCE and PPO is in Appendix 9.

5 Experiments

Our experiments were designed to serve two main purposes. First, we investigate whether the GD policy gradient approach could successfully discover risk-averse policies in scenarios where variance-based methods tend to fail. To accomplish this, we manipulated reward choices to assess the ability of the GD policy gradient to navigate risk-averse behavior. Second, we sought to verify the effectiveness of our algorithm in identifying risk-averse policies that have practical significance in both discrete and continuous domains. We aimed to demonstrate its ability to generate meaningful risk-averse policies that are applicable and valuable in practical settings.

Baselines. We compare our method with the original mean-variance policy gradient (Equation 2, denoted as MVO), Tamar's method [8] (denoted as Tamar), MVP [12], and MVPI [14]. Specifically, MVOrequires multiple trajectories to compute J ( θ ) ∇ θ J ( θ ) . We use n 2 trajectories to estimate J ( θ ) and another n 2 to estimate ∇ θ J ( θ ) , where n is the sample size. MVPI is a general framework for policy iteration whose inner risk-neutral RL solver is not specified. For the environment with discrete actions, we build MVPI on top of Q-Learning or DQN [41]. For continuous action environments, MVPI is built on top of TD3 [42] as in [14]. We use REINFORCE to represent the REINFORCE with baseline method. We use MG as a shorthand of mean-GD to represent our method. In each domain, we ensure each method's policy or value nets have the same neural network architecture.

For policy updating, MVO and MG collect n episodes before updating the policy. In contrast, Tamar and MVP update the policy after each episode. Non-tabular MVPI updates the policy at each environment step. In hyperparameter search, we use the parameter search range in MVPI [14] as a reference, making reasonable refinements to find an optimal parameter setting. Please refer to Appendix 10 for any missing implementation details. Code is available at 2 .

5.1 Tabular case: Modified Guarded Maze Problem

This domain is a modified Guarded Maze [7] that was previously described in Section 2.2. The original Guarded Maze is asymmetric with two openings to reach the top path (in contrast to a single opening for the bottom path). In addition, paths via the top tend to be longer than paths via the bottom. We modified the maze to be more symmetric in order to reduce preferences arising from certain exploration strategies that might be biased towards shorter paths or greater openings, which may confound risk aversion. Every movement before reaching the goal receives a reward of -1 except moving to the red state, where the reward is sampled from {-15 , -1 , 13 } with probability { 0 . 4 , 0 . 2 , 0 . 4 } (mean is -1 ) respectively. The maximum episode length is 100 . MVOand MG collect n = 50 episodes before updating the policy. Agents are tested for 10 episodes per evaluation.

Maze,goal_r=20

Maze,goal_r=20

Maze,goal_r=40

20

(a)

(b)

(c)

Optimal risk-aversion rate

0.8

Optimal risk-aversionrate

0.8

0

-20

MVO

0.6

MVO

0.6

MVO

Return

-40

Tamar

Tamar

Tamar

MVP

0.4

MVP

0.4

MVP

-60

MG

MG

MG

-80

0.2

0.2

-100

0.0

0.0

0

25

50

75

100

125

150

175200

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175200

Episodes(1k)

Episodes(1k)

Episodes(1k)

The failure of variance-based baselines under simple reward manipulation. We first set the goal reward to 20 . Here, we report the optimal risk-aversion rate achieved during training. Specifically, we measure the percentage of episodes that obtained the optimal risk-averse path, represented by the white color path in Figure 1, out of all completed episodes up to the current stage of training.

Notice that MVO performs well in this domain when using double sampling to estimate its gradient. Then we increase the goal reward to 40 . This manipulation does not affect the return variance of the optimal risk-averse policy, since the reward is deterministic. However, the performances of MVO, Tamar, MVP all decrease, since they are more sensitive to the numerical scale of the return (due to the E [ G 2 0 ] term introduced by variance). MVPI is a policy iteration method in this problem, whose learning curve is not intuitive to show. It finds the optimal risk-averse path when the goal reward is 20 , but it fails when the goal reward is 40 . An analysis for MVPI is given in Appendix 10.2.2. We compare the sensitivity of different methods with respect to λ in Appendix 10.2.4.

Remark. Scaling rewards by a small factor is not an appropriate approach to make algorithms less sensitive to the numerical scale for both total return variance and per-step reward variance, since it changes the original mean-variance objective in both cases.

5.2 Discrete control: LunarLander

This domain is taken from OpenAI Gym Box2D environments [15]. We refer readers to its official documents for the full description. Originally, the agent is awarded 100 if it comes to rest. We divide the ground into two parts by the middle line of the landing pad, as shown in Figure 10 in Appendix. If the agent lands in the right area, an additional noisy reward sampled from N (0 , 1) times 90 is given. A risk-averse agent should learn to land at the left side as much as possible. We include REINFORCE as a baseline to demonstrate the risk-aversion of our algorithm. REINFORCE, MVO and MG collect n = 30 episodes before updating their policies. Agents are tested for 10 episodes per evaluation.

We report the rate at which different methods land on the left in Figure 3(b) (we omit the failed methods), i.e, the percentage of episodes successfully landing on the left per evaluation. MVO, Tamar, and MVP do not learn reasonable policies in this domain according to their performances in Figure 3(a). MVP learns to land in the middle of the learning phase, but soon after fails to land. Since successfully landing results in a large return (success reward is 100 ), the return square term ( E [ G 2 0 ] ) introduced by variance makes MVP unstable. MVPI also fails to land since V [ R ] is sensitive to the numerical scale of rewards. In this domain, the success reward is much larger than other reward values. Furthermore, reward modification in MVPI turns large success rewards into negative values, which prevents the agent from landing on the ground. MG achieves a comparable return with REINFORCE, but clearly learns a risk-averse policy by landing more on the left.

LunarLander

200

0

Return

REINFORCE

MVO

-200

Tamar

MVRNN

-400

MG

MVPI

(a)

-600

0

20

40

60

80

100

120

0.8

REINFORCE

MVP

leftrate

0.6

MG

0.2

(b)

0.0

0

20

40

60

80

100

120

Episodes(1k)

InvertedPendulum

HalfCheetah

Swimmer

500 (a)

(c)

PPO

800 (e)

2000

MVO

700

400

REINFORCE

1500

Tamar

600

PPO

Return

300

MVO

Tamar

Return

1000

MG

MVP

Return

500

MVO

Tamar

200

MVP

400

MVP

500

MG

300

MG

100

0

200

0

0

20

40

60

80

100

120

-500

0

5

10

15

20

25

30

100

0

5

10

15

20

25

30

Episodes(1k)

Episodes(1k)

Episodes(1k)

0.7

0.6

(b)

InvertedPendulum

REINFORCE

0.6 (d)

PPO

HalfCheetah

0.6

(f)

Swimmer

PPO

X>0.01 Rate

0.5

0.4

MVO

MVP

Tamar

0.5

MVP

MVO

Tamar

X>0.5 Rate

0.4

0.5

Tamar

MVO

MVP

S

MG

AMG

0.3

MG

0.2

0.2

0.2

0.1

0.1

0.1

0.0

0.0

0.0

0

20

40

Episodes(10k)

60

80

100

120

0

5

10

Episodes(1k)

15

20

25

30

0

5

10

Episodes(1k)

15

20

25

30

5.3 Continuous control: Mujoco

Mujoco [16] is a collection of robotics environments with continuous states and actions in OpenAI Gym [15]. Here, we selected three domains (InvertedPendulum, HalfCheetah, and Swimmer) that are conveniently modifiable, where we are free to modify the rewards to construct risky regions in the environment (Through empirical testing, risk-neutral learning failed when similar noise was introduced to other Mujoco domains. Consequently, identifying the cause for the failure of risk-averse algorithms on other domains became challenging). Motivated by and following [43, 44], we define a risky region based on the X-position. For instance, if X-position > 0 . 01 in InvertedPendulum, X-position < -3 in HalfCheetah, and X-position > 0 . 5 in Swimmer, an additional noisy reward sampled from N (0 , 1) times 10 is given. Location information is appended to the agent's observation. A risk-averse agent should reduce the time it visits the noisy region in an episode. We also include the risk-neutral algorithms as baselines to highlight the risk-aversion degree of different methods.

All the risk-averse policy gradient algorithms still use VPG to maximize the expected return in InvertedPendlulum (thus the risk-neutral baseline is REINFORCE). Using VPG is also how these methods are originally derived. However, VPG is not good at more complex Mujoco domains, e.g., see [45]. In HalfCheetah and Swimmer, we combine those algorithms with PPO-style policy gradient to maximize the expected return. Minimizing the risk term remains the same as their original forms. MVPI is an off-policy time-difference method in Mujoco. We train it with 1e6 steps instead of as many episodes as other methods. MVO and MG sample n = 30 episodes in InvertedPendulum and n = 10 in HalfCheetah and Swimmer before updating policies. Agents are tested for 20 episodes per evaluation. The percentage of time steps visiting the noisy region in an episode is shown in Figure 4(b,d,f). Compared with other return variance methods, MG achieves a higher return while maintaining a lower visiting rate. Comparing MVPI and TD3 against episode-based algorithms like MGis not straightforward within the same figure due to the difference in parameter update frequency. MVPI and TD3 update parameters at each environment time step. We shown their learning curves in Figure 5. MVPI also learns risk-averse policies in all three domains according to its learning curves.

We further design two domains using HalfCheetah and Swimmer. The randomness of the noisy reward linearly decreases when agent's forward distance grows. To maximize the expected return and minimize risk, the agent has to move forward as far as possible. The results are shown in Figures 18,19 in Appendix. In these two cases, only MG shows a clear tendency of moving forward, which suggests our method is less sensitive to reward choices compared with methods using V [ R ] .

The return variance and GD during learning in the above environments are also reported in Appendix 10. In general, when other return variance based methods can find the risk-averse policy, MG maintains a lower or comparable return randomness when measured by both variance and GD. When

InvertedPendulum

HalfCheetah

Swimmer

500 (a)

2500 (c)

TD3

1000

(e)

MVPI

400

2000

800

Return

600

200

400

100

TD3

500

200

TD3

MVPI

MVPI

0

0

20

Steps(10k)

40

60

80

100

0

0

20

Steps(10k)

40

60

80

100

0

20

Steps(10k)

40

60

80

100

InvertedPendulum

HalfCheetah

Swimmer

(b)

TD3

(d)

TD3

0.7 ()

0.5

MVPI

0.8

MVPI

0.6

X>0.01 Rate

0.4

Rate

0.5

0.3

X>0.51

0.4

TD3

0.3

MVPI

0.2

0.2

0.1

0.2

0.1

0.0

0.0

0.0

0

20

Steps(10k)

40

60

80

100

0

20

Steps(10k)

40

60

80

100

0

20

40

60

80

100

Steps(10k)

other methods fail to learn a reasonably good risk-averse policy, MG consistently finds a notably higher return and lower risk policy compared with risk-neutral methods. MVPI has the advantage to achieve low return randomness in location based risky domains, since minimizing V [ R ] naturally avoids the agent from visiting the noisy region. But it fails in distance-based risky domains.

6 Conclusion and Future Work

This paper proposes to use a new risk measure, Gini deviation, as a substitute for variance in meanvariance RL. It is motivated to overcome the limitations of the existing total return variance and per-step reward variance methods, e.g., sensitivity to numerical scale and hindering of policy learning. A gradient formula is presented and a sampling-based policy gradient estimator is proposed to minimize such risk. We empirically show that our method can succeed when the variance-based methods will fail to learn a risk-averse or a reasonable policy. This new risk measure may inspire a new line of research in RARL. First, one may study the practical impact of using GD and variance risk measures. Second, hybrid risk-measure may be adopted in real-world applications to leverage the advantages of various risk measures.

Limitations and future work. Our mean-GD policy gradient requires sampling multiple trajectories for one parameter update, making it less sample efficient compared to algorithms that can perform updates per environment step or per episode. As a result, one potential avenue for future work is to enhance the sample efficiency of our algorithm. This can be achieved by more effectively utilizing off-policy data or by adapting the algorithm to be compatible with online, incremental learning.

Acknowledgments and Disclosure of Funding

We thank Ruodu Wang from University of Waterloo and Han Wang from University of Alberta for valuable discussions and insights. Resources used in this work were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute https://vectorinstitute.ai/partners/ and the Natural Sciences and Engineering Council of Canada. Yudong Luo is also supported by a David R. Cheriton Graduate Scholarship, a President's Graduate Scholarship, and an Ontario Graduate Scholarship. Guiliang Liu's research was in part supported by the Start-up Fund UDF01002911 of the Chinese University of Hong Kong, Shenzhen. Yangchen Pan acknowledges funding from the Turing AI World Leading Fellow.

References

Mohammad Naghshvar, Ahmed K Sadek, and Auke J Wiggers. Risk-averse behavior planning for autonomous driving under uncertainty. arXiv preprint arXiv:1812.01254 , 2018.

Tomas Björk, Agatha Murgoci, and Xun Yu Zhou. Mean-variance portfolio optimization with state-dependent risk aversion. Mathematical Finance: An International Journal of Mathematics, Statistics and Financial Economics , 24(1):1-24, 2014.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of operations research , 27(2):294-311, 2002.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120, 2017.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems , 27, 2014.

Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient riskaverse reinforcement learning. Advances in Neural Information Processing Systems , 35:3263932652, 2022.

Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. International Conference on Machine Learning , 2012.

Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. Advances in neural information processing systems , 26, 2013.

Harry M Markowitz and G Peter Todd. Mean-variance analysis in portfolio choice and capital markets , volume 66. John Wiley & Sons, 2000.

Duan Li and Wan-Lung Ng. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation. Mathematical finance , 10(3):387-406, 2000.

Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, and Daesub Yoon. A block coordinate ascent algorithm for mean-variance optimization. Advances in Neural Information Processing Systems , 31, 2018.

Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatility reduction. International Joint Conference on Artificial Intelligence , 2020.

Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for riskaverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10905-10913, 2021.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

Shie Mannor and John Tsitsiklis. Mean-variance optimization in markov decision processes. International Conference on Machine Learning , 2011.

Matthew J Sobel. The variance of discounted markov decision processes. Journal of Applied Probability , 19(4):794-802, 1982.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Corrado Gini. Variabilità e mutabilità: contributo allo studio delle distribuzioni e delle relazioni statistiche.[Fasc. I.] . Tipogr. di P. Cuppini, 1912.

Shlomo Yitzhaki et al. Gini's mean difference: A superior measure of variability for non-normal distributions. Metron , 61(2):285-316, 2003.

Gerald J Glasser. Variance formulas for the mean difference and coefficient of concentration. Journal of the American Statistical Association , 57(299):648-654, 1962.

Edward Furman, Ruodu Wang, and Riˇ cardas Zitikis. Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks. Journal of Banking & Finance , 83:70-84, 2017.

Gustave Choquet. Theory of capacities. In Annales de l'institut Fourier , volume 5, pages 131-295, 1954.

Michel Grabisch. The application of fuzzy integrals in multicriteria decision making. European journal of operational research , 89(3):445-456, 1996.

Ruodu Wang, Yunran Wei, and Gordon E Willmot. Characterization, robustness, and aggregation of signed choquet integrals. Mathematics of Operations Research , 45(3):993-1015, 2020.

Shigeo Kusuoka. On law invariant coherent risk measures. In Advances in mathematical economics , pages 83-95. Springer, 2001.

Bogdan Grechuk, Anton Molyboha, and Michael Zabarankin. Maximum entropy principle with general deviation measures. Mathematics of Operations Research , 34(2):445-467, 2009.

R Tyrrell Rockafellar, Stan Uryasev, and Michael Zabarankin. Generalized deviations in risk analysis. Finance and Stochastics , 10(1):51-74, 2006.

Fangda Liu, Jun Cai, Christiane Lemieux, and Ruodu Wang. Convex risk functionals: Representation and applications. Insurance: Mathematics and Economics , 90:66-79, 2020.

Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the cvar via sampling. In Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning , pages 1096-1105. PMLR, 2018.

Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 3455-3461, 8 2021.

Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforcement learning with monotonic splines. In International Conference on Learning Representations , 2022.

Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit distributional reinforcement learning. Advances in Neural Information Processing Systems , 33:7135-7147, 2020.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning , pages 7390-7399. PMLR, 2021.

Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations , 2023.

OpanAI. Performance in each mujoco environment. https://spinningup.openai.com/ en/latest/spinningup/bench.html .

Arjun K Gupta and Mohammad AS Aziz. Convex ordering of random variables and its applications in econometrics and actuarial science. European Journal of Pure and Applied Mathematics , 3(5):779-785, 2010.

Michael Rothschild and Joseph E Stiglitz. Increasing risk: I. a definition. In Uncertainty in Economics , pages 99-121. Elsevier, 1978.

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance , 9(3):203-228, 1999.

Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. Advances in Neural Information Processing Systems , 32, 2019.

Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In International Conference on Learning Representations , 2016.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: a cvar optimization approach. Advances in neural information processing systems , 28, 2015.

Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint arXiv:1911.03618 , 2019.

Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10639-10646, 2021.

Supplementary Information

7 Convex Order, Gini Deviation, and Variance

Convex order describes dominance in terms of variability and is widely used in actuarial science.

Definition 2 ([46]) . Consider two random variables X and Y , X is called convex order smaller than Y , succinctly X ≤ cx Y , if E [ ψ ( X )] ≤ E [ ψ ( Y )] , for all convex function ψ () , assuming that both expectations exist.

In convex order X ≤ cx Y , Y is also called a mean-preserving spread of X [47], which intuitively means that Y is more spread-out (and hence more random) than X . Thus, it is often desirable for a measure of variability to be monotone with respect to convex order [25]. Both variance and GD, as a measure of variability, are consistent with convex order, i.e.,

If X ≤ cx Y , then V [ X ] ≤ V [ Y ] for all X,Y ∈ M

If X ≤ cx Y , then D [ X ] ≤ D [ Y ] for all X,Y ∈ M

Proof. It is immediate that X ≤ cx Y implies E [ X ] = E [ Y ] . If we take convex function ψ ( x ) = x 2 , we can get the order of variance V [ X ] ≤ V [ Y ] . For the proof of GD, please refer to the following Lemma. Recall that GD can be expressed in the form of signed Choquet integral with a concave function h .

Lemma 3 ([28],Theorem 2) . Convex order consistency of a signed Choquet integral is equivalent to its distortion function h being concave, i.e., X ≤ cx Y if and only if the signed Choquet integral Φ h ( X ) ≤ Φ h ( Y ) for all concave functions h ∈ H .

8 GD Gradient Formula Calculation

8.1 General GD Gradient Formula

Proposition 1. Let Assumptions 1, 2, 3 hold. Then

Consider a random variable Z , whose distribution function is controlled by θ . Recall that q α ( Z ; θ ) represents the α -level quantile of Z θ . According to Equation 14, the gradient of GD is

To make the integral over α clearer, we rewrite q α ( Z ; θ ) as F -1 Z θ ( α ) , where F Z θ is the CDF.

Switching the integral order, we get

Denote t = F -1 Z θ ( α ) , then α = F Z θ ( t ) . Here, we further change the inner integral from dα to dF Z θ ( t ) , i.e., dα = dF Z θ ( t ) = f Z ( t ; θ ) dt . The integral range for t is now from F -1 Z θ ( F Z θ ( z )) = z to F -1 Z θ (1) = b .

Applying ∇ θ log( x ) = 1 x ∇ θ x to ∇ θ f Z ( z ; θ ) , we have

8.2 GD Policy Gradient via Sampling

Following Section 4.2, we consider estimating the following equation via sampling.

Here sampling from G 0 corresponds to sampling trajectory τ with its return R τ from the environment. As discussed in the main paper, for a trajectory τ i , ∇ θ log f G 0 ( R τ i , θ ) is estimated as ∑ T -1 t =0 ∇ θ log π θ ( a i,t | s i,t ) .

The CDF function F G 0 is parameterized by a step function given its quantiles { R τ i } n i =1 , which satisfy R τ 1 ≤ R τ 2 ≤ ... ≤ R τ n . An example of the step function is shown in Figure 6. With this parameterization, the integral over CDF can be regarded as the area below the step function.

Thus, for each τ i , the integral over CDF is approximated as ( R τ n is treated as b )

Aggregating all the calculations together yields Equation 16.

9 Mean-GD Policy Gradient Algorithm

We consider maximizing E [ G 0 ] -λ D [ G 0 ] in this paper. Maximizing the first term, i.e., E [ G 0 ] has been widely studied in risk-neutral RL. For on-policy policy gradient, we can use vanilla policy gradient (VPG), e.g., REINFORCE with baseline, or more advanced techniques like PPO [39].

To make fair comparison with other risk-averse policy gradient methods, we first initialize mean-GD policy gradient with VPG. To improve sample efficiency, we incorporate IS for multiple updates. Taking advantage of having n samples, we can select those trajectories whose IS ratio is in the range [1 -δ, 1 + δ ] for calculation to reduce gradient variance. In more complex continuous domains, e.g., Mujoco, we combine GD policy gradient with PPO since VPG is not good at Mujoco [45]. However, in this case, policy may have a significant difference per update, where the former IS selection strategy can no longer be applied. Then we directly clip the IS ratio by a constant, though it is biased [40]. The full algorithms are summarized in Algorithm 1 and 2. We omit the parameters for PPO in Algorithm 2 for simplicity. We still report the PPO parameter settings in Section 10.

Algorithm 1 Mean-Gini Deviation Policy Gradient (with REINFORCE baseline)

Input: Iterations number K , sample size n , inner update number M , policy learning rate α θ , value learning rate α ϕ , importance sampling range δ , inner termination parameter β , trade-off parameter

10 Experiments Details

10.1 General Descriptions of Different Methods

Among the methods compared in this paper, Tamar [8] and MVP [12] are on-policy policy gradient methods. MVO and MG are policy gradient methods, but sample n trajectories and use IS to update. Non-tabular MVPI [14] is an off-policy time-difference method.

Policy gradeint methods. MVO, Tamar, MVP, are originally derived based on VPG. Since VPG is known to have a poor performance in Mujoco, we also combined these mean-variance methods with PPO. Thus these mean-variance methods and our mean-GD method have different instantiations to maximize the expected return in different domains:

With VPG: in Maze, LunarLander, InvertedPendulum.

With PPO: in HalfCheetah, Swimmer.

When the risk-neutral policy gradient is VPG, for MVO and MG, it is REINFORCE with baseline; for Tamar and MVP, we strictly follow their papers to implement the algorithm, where no value function is used. MVO and MG collect n trajectories and use the IS strategy in Algorithm 1 to update policies. Tamar and MVP do not need IS, and update policies at the end of each episode.

When the risk-neutral policy gradient is PPO, we augment PPO with the variance or GD policy gradient from the original methods. MVO and MG collect n trajectories and use the IS strategy in Algorithm 2 to compute the gradient for the risk term. Tamar and MVP still update policies once at the end of each episode.

MVPI. We implemented three versions of MVPI in different domains:

Tabular: in Maze, MVPI is a policy iteration method (Algorithm 1 in [14]).

With DQN: in LunarLander, since this environment has discrete action space.

With TD3: in InvertedPendulum, HalfCheetah, Swimmer, since these environments have continuous action space.

Algorithm 2 Mean-Gini Deviation Policy Gradient (with PPO)

Input: Iterations number K , sample size n , inner update number M , policy learning rate α θ , value learning rate α ϕ , importance sampling clip bound ζ , trade-off parameter λ .

Initialize policy π parameter θ , value V parameter ϕ .

θ ϕ for k = 1 to K do Sample n trajectories { τ i } n i =1 by π θ , compute return { R ( τ i ) } n i =1 Compute rewards-to-go for each state in τ i : g i,t Compute advantages for each state-action in τ i : A ( s i,t , a i,t ) based on current V ϕ for m = 1 to M do Compute importance sampling ratio for each trajectory { ρ i } n i =1 , and ρ i = min( ρ i , b ) Sort trajectories such that R ( τ 1 ) ≤ ... ≤ R ( τ n ) mean_grad = 0, gini_grad = 0 for i = 1 to n do mean_grad += PPO-Clip actor grad Update V ϕ by mean-squared error 1 T ∑ T -1 t =0 ( V ϕ ( s i,t ) -g i,t ) 2 with learning rate α ϕ end for for i = 1 to n -1 do gini_grad += -ρ i · η i ∑ T -1 t =0 ∇ θ log π θ ( a i,t | s i,t ) , where η i = ∑ n -1 j = i 2 j n ( R j +1 -R j ) -( R n -R i ) end for Update π θ by ( 1 nT mean_grad -λ ( n -1) T gini_grad ) with learning rate α θ (Equation 18) end for end for

We summarize the components required in different methods in Table 1.

10.2 Modified Guarded Maze Problem

The maze consists of a 6 × 6 grid. The agent can visit every free cell without a wall. The agent can take four actions (up, down, left, right). The maximum episode length is 100.

Policy function. For methods requiring a policy function, i.e., MVO, Tamar, MVP, MG, the policy is represented as

where ϕ ( s, a ) is the state-action feature vector. Here we use one-hot encoding to represent ϕ ( s, a ) . Thus, the dimension of ϕ ( s, a ) is 6 × 6 × 4 . The derivative of the logarithm is

Value function. For methods requiring a value function, i.e., REINFORCE baseline used in MVO and MG, and Q-learning in MVPI, the value function is represented as V ω ( s ) = ϕ ( s ) · ω or Q ω ( s, a ) = ϕ ( s, a ) · ω . Similarly, ϕ ( s ) is a one-hot encoding.

Optimizer. The policy and value loss are optimized by stochastic gradient descent (SGD).

10.2.1 Learning Parameters

We set discount factor γ = 0 . 999 .

MVO : policy learning rate is 1e-5 ∈ {5e-5, 1e-5, 5e-6}, value function learning rate is 100 times policy learning rate. λ = 1 . 0 ∈ {0.6, 0.8, 1.0, 1.2}. Sample size n = 50 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

Tamar : policy learning rate is 1e-5 ∈ {5e-5, 1e-5, 5e-6}, J, V learning rate is 100 times policy learning rate. Threshold b = 50 ∈ {10, 50, 100}, λ = 0 . 1 ∈ {0.1, 0.2, 0.4}.

MVP : policy learning rate is 1e-5 ∈ {5e-5, 1e-5, 5e-6}, y learning rate is the same. λ = 0 . 1 ∈ {0.1, 0.2, 0.4}.

MG : policy learning rate is 1e-4 ∈ {5e-4, 1e-4, 5e-5}, value function learning rate is 100 times policy learning rate. λ = 1 . 2 ∈ {0.8, 1.0, 1.2}. Sample size n = 50 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

MVPI : Q function learning rate 5e-3 ∈ {5e-3, 1e-3, 5e-4}, λ = 0 . 2 ∈ {0.2, 0.4, 0.6}.

10.2.2 Analysis for MVPI in Maze (MVPI-Q-Learning)

MVPI-Q-Learning finds the optimal risk-averse path when goal reward is 20 but fails when goal reward is 40 . Since it is not intuitive to report the learning curve for a policy iteration method where its reward is modified in each iteration, we give an analysis here.

The value of dual variable y in Equation 5 is (1 -γ ) E [ G 0 ] given the current policy. Recall that the maximum episode length is 100 . At the beginning, when the Q function is randomly initialized (i.e., it is a random policy), E [ G 0 ] = ∑ 99 t =0 0 . 999 t ( -1) ≈ -95 . 2 . Thus y = (1 -0 . 999) × ( -95 . 2) = -0 . 0952 , the goal reward after modification is r goal = 20 -0 . 2 × 20 2 +2 × 0 . 2 × 20 × y ≈ -60 . 7 . For the red state, its original reward is sampled from {-15 , -1 , 13 } . After the reward modification, it becomes sampling from {-59 . 4 , -1 . 16 , -21 . 2 } . Thus the expected reward of the red state is now r red = 0 . 4 × ( -59 . 4) + 0 . 2 × ( -1 . 16) + 0 . 4 × ( -21 . 2) = -32 . 472 . Given the maximum episode length is 100 , the optimal policy is still the white path in Figure 1. (Because the expected return for the white path is ∑ 9 t =0 0 . 999 t ( -1) + 0 . 999 10 ( -60 . 7) ≈ -70 . The expected return for a random walk is ∑ 99 t =0 0 . 999 t ( -1) ≈ -95 . 2 . The expected return for the shortest path going through the red state is even lower than the white path since the reward of the red state after modification is pretty negative: -32 . 472 .)

However, when goal reward is 40 , after modification, the goal reward becomes r goal = 40 -0 . 2 × 40 2 +2 × 0 . 2 × 40 × y ≈ -281 . 5 . In this case, the optimal policy has to avoid the goal state since it leads to a even lower return.

10.2.3 Return Variance and Gini Deviation in Maze

We report the return's variance and GD during learning for different methods, as shown in Figure 7 and 8. Tamar [8] is unable to reach the goal in both settings. MVO fails to reach the goal when the return magnitude increases. MVP [12]'s optimal risk-aversion rate is much lower than MG. MG can learn a risk averse policy in both settings with lower variance and GD, which suggests it is less sensitive to the return numerical scale.

Maze,goal_r=20,Return

Maze,goal_r=20,ReturnVariance

Maze,goal_r=20,ReturnGiniDeviation

20

120

MVO

5

MVO

0

100

Tamar(failed)

Tamar(failed)

-20

MVO

80

MAVRm

4

MVR

-40

MVP

Tamar

60

MG

3

MG

-60

MG

40

2

-80

20

-100

0

1

0

25

50

75

100125

150

175200

0

25

50

75

100

125

150

175200

0

25

50

75

100

0125

150

175

200

Episodes(1k)

Episodes(1k)

Episodes(1k)

Maze,goal_r=40,Return

Maze,goal_r=40,ReturnVariance

Maze,goal_r=40,ReturnGiniDeviation

40

70

MVO(failed)

3.0

MVO (failed)

20

60

Tamar(failed)

2.5

Tamar(failed)

0

MVO

50

MVP

MVP

-20

Tamar

40

2.0

MG

-40

MVP

30

1.5

-60

MG

20

1.0

-80

10

0.5

-100

0

0.0

0

25

50

75

100

125

150

175200

0

25

50

75

100

125

150

175

200

0

25

50

75

100125

150

175

200

Episodes(1k)

Episodes(1k)

Episodes(1k)

10.2.4 Sensitivity to Trade-off Parameter

We report the learning curves of total return based methods with different λ when goal reward is 20 in Figure 9. The learning parameters are the same as shown in Section 10.2.1.

10.3 LunarLander Discrete

The agent's goal is to land the lander on the ground without crashing. The state dimension is 8 . The action dimension is 4 . The detailed reward information is available at this webpage 3 . We divide the whole ground into left and right parts by the middle line of the landing pad as shown in Figure 10. If the agent lands in the right part, an additional noisy reward signal sampled from N (0 , 1) tims 90 is given. We set the maximum episode length to 1000 . Note that the original reward for successfully landing is 100 , thus the numerical scale of both return and reward is relatively large in this domain.

Policy function. The policy is a categorical distribution in REINFORCE, MVO, Tamar, MVP and MG, modeled as a neural network with two hidden layers. The hidden size is 128 . Activation is ReLU. Softmax function is applied to the output to generate categorical probabilities.

Value function. The value function in REINFORCE, MVO, MG, and Q function in MVPI-DQN is a neural network with two hidden layers. The hidden size is 128 . Activation is ReLU.

Optimizer. The optimizer for policy and value functions is Adam.

10.3.1 Learning Parameters

Discount factor is γ = 0 . 999

REINFORCE (with baseline): the policy learning rate is 7e-4 ∈ {7e-4, 3e-4, 7e-5}, value function learning rate is 10 times policy learning rate.

MVO : policy learning rate is 7e-5 ∈ {7e-4, 3e-4, 7e-5}, value function learning rate is 10 times policy learning rate. λ = 0.4 ∈ {0.4, 0.6, 0.8}. Sample size n = 30 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

Tamar : policy learning rate is 7e-5 ∈ {7e-4, 3e-4, 7e-5}. J, V learning rate is 100 times the policy learning rate. Threshold b = 50 ∈ {10,50,100}. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MVP : policy learning rate is 7e-5 ∈ {7e-4, 3e-4, 7e-5}. y learning rate is the same. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MG : policy learning rate is 7e-4 ∈ {7e-4, 3e-4, 7e-5}, value function learning rate is 10 times policy learning rate. λ = 0 . 6 ∈ {0.4, 0.6, 0.8}. Sample size n = 30 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

MVPI : Q function learning rate is 7e-4 ∈ {7e-4, 3e-4, 7e-5}, λ = 0 . 2 ∈ {0.2, 0.4, 0.6}. Batch size is 64.

10.3.2 Return Variance and Gini Deviation in LunarLander

The return's variance and GD of different methods during training is shown in Figure 11. All the risk-averse methods, apart from ours, fail to learn a reasonable policy in this domain. Our method achieves a comparable return, but with lower variance and GD compared with risk-neutral method.

Left

Right

LunarLander,ReturnVariance

LunarLander,ReturnGiniDeviation

200000

200

150000

REINFORCE

175

REINFORCE

100000

MVO.(failed)

MVO (failed)

Remaifailed

150

MVP(failed)

125

MVP(failed)

30000/

MG

100

MG

MVPI（failed)

75

failed)

15000

50

25

0

20

4pisodes(1)0

80

100

120

0

0

20

40

60

80

100

120

Episodes(1k)

10.4 InvertedPendulum

(The description of the Mujoco environments can be found at this webpage 4 .)

The agent's goal is to balance a pole on a cart. The state dimension is 4 (X-position is already contained). The action dimension is 1 . At each step, the environment provides a reward of 1 . If the agent reaches the region X-coordinate > 0 . 01 , an additional noisy reward signal sampled from N (0 , 1) times 10 is given. To avoid the initial random speed forcing the agent to the Xcoordinate > 0 . 01 region, we decrease the initial randomness for the speed from U ( -0 . 01 , 0 . 01) to U ( -0 . 0005 , 0 . 0005) , where U () represents the uniform distribution. The game ends if angle between the pole and the cart is greater than 0.2 radian or a maximum episode length 500 is reached.

Policy function. The policy is a normal distribution in REINFORCE, and VPG based methods (MVO, Tamar, MVP, and MG), modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Tanh is applied to the output to scale it to ( -1 , 1) . The output times the maximum absolute value of the action serves as the mean of the normal distribution. The logarithm of standard deviation is an independent trainable parameter.

The policy is a deterministic function in TD3 and MVPI, modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Tanh is applied to the output to scale it to ( -1 , 1) . The output times the maximum absolute value of the action is the true action executed in the environment.

Value function. The value function in REINFORCE, VPG based MVO, VPG based MG, TD3, and MVPI is a neural network with two hidden layers. The hidden size is 128. Activation is ReLU.

Optimizer. Optimizer for both policy and value function is Adam.

10.4.1 Learning Parameters

Discount factor γ = 0 . 999 .

REINFORCE (with baseline): policy learning rate is 1e-4 ∈ {1e-4, 5e-5, 5e-4}, value function learning rate is 10 times policy learning rate.

MVO : policy learning rate is 1e-5 ∈ {1e-4, 5e-5, 1e-5}, value function learning rate is 10 times policy learning rate. λ = 0.6 ∈ {0.2, 0.4, 0.6}. Sample size n = 30 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

Tamar : policy learning rate is 1e-5 ∈ {1e-4, 5e-5, 1e-5}. J, V learning rate is 100 times policy learning rate. Threshold b = 50 ∈ {10,50,100}. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MVP : policy learning rate is 1e-5 ∈ {1e-4, 5e-4, 1e-5}. y learning rate is the same. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MG : policy learning rate is 1e-4 ∈ {1e-4, 5e-5, 1e-4}, value function learning rate is 10 times policy learning rate. λ = 1 . 0 ∈ {0.6, 1.0, 1.4}. Sample size n = 30 . Maximum inner update number M = 10 . IS ratio range δ = 0 . 5 . Inner termination ratio β = 0 . 6 .

MVPI : Policy and value function learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}, λ = 0 . 2 ∈ {0.2, 0.4, 0.6}. Batch size is 256.

TD3 : Policy and value function learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}. Batch size is 256.

10.4.2 Return Variance and Gini Deviation in InvertedPendulum

The return varaince and GD of different methods are shown in Figure 12 and 13.

0

20

40

60

80

100

120

Episodes(1k)

0

5000

10000

15000

20000

25000

30000

35000

REINFORCE

MVO (failed)

MVP (failed)

MG

REINFORCE

MVO (failed)

MVP (failed)

MG

40

60

80

Episodes(1k)

0

20

40

60

80

100

Steps(10k)

0

5000

10000

15000

20000

25000

TD3

MVPI

40

60

Steps(10k)

10.5 HalfCheetah

The agent controls a robot with two legs. The state dimension is 18 (add X-position). The action dimension is 6. The reward is determined by the speed between the current and the previous time step and a penalty over the magnitude of the input action (Originally, only speed toward right is positive, we make the speed positive in both direction so that agent is free to move left or right). If the agent reaches the region X-coordinate < -3 , an additional noisy reward signal sampled from N (0 , 1) times 10 is given. The game ends when a maximum episode length 500 is reached.

Policy function. The policy is a normal distribution in PPO, and PPO based methods (MVO, Tamar, MVP, and MG). The architecture is the same as in InvertedPendulum. Hidden size is 256.

The policy of TD3 and MVPI is the same as in InvertedPendulum. Hidden size is 256.

Value function. The value function in PPO, PPO based methods (MVO, Tamar, MVP, and MG), TD3 and MVPI is a neural network with two hidden layers. The hidden size is 256. Activation is ReLU.

Optimizer. Optimizer for policy and value is Adam.

TD3

MVPI

100

100

100

80

60

40

20

0

70

60

50

40

30

20

10

0

0

0

20

20

80

120

10.5.1 Learning Parameters

Discount factor is γ = 0 . 99 .

Common parameters of PPO and PPO based methods. GAE parameter: 0.95, Entropy coef: 0.01, Critic coef: 0.5, Clip ϵ : 0.2, Grad norm: 0.5.

PPO. policy learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. Inner update number M = 5 .

MVO. policy learning rate 7e-5 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. Sample size n = 10 . Inner update number 5.

Tamar. policy learning rate 7e-5 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. J, V learning rate is 100 times policy learning rate. Threshold b = 50 ∈ {10,50,100}. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MVP. policy learning rate 7e-5 ∈ {3e-4, 7e-5, 1e-5}, value function and y learning rate is the same. λ = 0.2 ∈ {0.2, 0.4, 0.6}.

MG. policy learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. Sample size n = 10 . Inner update number M = 5 .

TD3. policy learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. Batch size is 256.

MVPI. policy learning rate 3e-4 ∈ {3e-4, 7e-5, 1e-5}, value function learning rate is the same. λ = 0.2 ∈ {0.2, 0.4, 0.6}. Batch size is 256.

10.5.2 Return Variance and Gini Deviation in HalfCheetah

See Figures 14 and 15.

0

5

10

15

20

25

30

Episodes(1k)

0

50000

100000

150000

200000

PPO

MVO (failed)

MVP (failed)

MG

0

5

10

15

20

25

30

Episodes(1k)

50

100

150

200

PPO

MVO (failed)

MVP (failed)

MG

0

20

40

60

80

100

Steps(10k)

0

10000

20000

30000

40000

50000

60000

70000

80000

TD3

MVPI

0

20

40

60

80

100

Steps(10k)

0

20

40

60

80

100

120

140

TD3

MVPI

10.6 Swimmer

The agent controls a robot with two rotors (connecting three segments) and learns how to move. The state dimension is 10 (add XY-positions). The action dimension is 2. The reward is determined by the speed between the current and the previous time step and a penalty over the magnitude of the input action (Originally, only speed toward right is positive, we make the speed positive in both direction so that agent is free to move left or right). If agent reaches the region X-coordinate > 0 . 5 , an additional noisy reward signal sampled from N (0 , 1) times 10 is given. The game ends when a maximum episode length 500 is reached.

The neural network architectures and learning parameters are the same as in HalfCheetah.

10.6.1 Return Variance and Gini Deviation in Swimmer

See Figures 16 and 17.

0

5

10

15

20

25

30

Episodes(1k)

0

5000

10000

15000

20000

25000

30000

PPO

MVO

Tamar

MVP

MG

0

5

10

15

20

25

30

Episodes(1k)

10

20

30

40

50

60

70

80

90

PPO

MVO

Tamar

MVP

MG

0

20

40

60

80

100

Steps(10k)

0

5000

10000

15000

20000

25000

30000

35000

TD3

MVPI

0

20

40

60

80

100

Steps(10k)

20

40

60

80

100

TD3

MVPI

11 Additional Results

We design two other senarios in HalfCheetah and Swimmer (marked as HalfCheetah1 and Swimmer1 in the figure's caption). Here the randomness of the noisy reward linearly decreases over the forward distance (right direction) the agent has covered. To encourage the agent to move forward, only the forward speed is positive. The additional noisy reward is sampled from N (0 , 1) times 10 times 1 -X 20 if X -position > 0 . To maximize the expected return and minimize the risk, the agent should move forward as far as possible.

The learning parameters are the same as Section 10.5.

The distance agents covered in these two environments are shown in Figures 18 19.

12 Additional Related Work and Discussion

12.1 Coherent measure of variability and risk

This paper focuses on the measure of variability , i.e., the dispersion of a random variable. Both variance and Gini deviation are measures of variability as discussed in Sections 2 and 3. Gini deviation is further known as a coherent measure of variability.

Another widely adopted risk measure is conditional value at risk (CVaR), which is a coherent risk measure [48]. Coherence is usually important in financial domains. Consider a continuous random variable Z with the cumulative distribution function F ( z ) = P ( Z ≤ z ) . The value at risk (VaR) or quantile at confidence level α ∈ (0 , 1] is defined as VaR α ( Z ) = min { z | F ( z ) ≥ α } . Then the CVaR at confidence level α is defined as

Due to the nature of CVaR of only considering the tailed quantiles, it does not capture the variability of Z beyond the quantile level α . Thus it is not considered as a measure of variability. In [25], measures of variability and measures of risk are clearly distinguished and their coherent properties are different. We summarize their properties here [25].

Consider a measure ρ : M→ ( -∞ , ∞ ]

(A) Law-invariance : if X,Y ∈ M have the same distributions, then ρ ( X ) = ρ ( Y )

(A1) Positive homogeneity : ρ ( λX ) = λρ ( X ) for all λ > 0 , X ∈ M

(A2) Sub-additivity : ρ ( X + Y ) ≤ ρ ( X ) + ρ ( Y ) for X,Y ∈ M

(B1) Monotonicity : ρ ( X ) ≤ ρ ( Y ) when X ≤ Y , X,Y ∈ M

(B2) Translation invariance : ρ ( X -m ) = ρ ( X ) -m for all m ∈ R , X ∈ M

(C1) Standardization : ρ ( m ) = 0 for all m ∈ R

(C2) Location invariance : ρ ( X -m ) = ρ ( X ) for all m ∈ R , X ∈ M

Coherent measure of variability : A measure of variability is coherent if it satisfies properties (A), (A1), (A2), (C1) and (C2). An example is Gini deviation.

Coherent risk measure : A risk measure is coherent if it satisfies properties (A1), (A2), (B1) and (B2). An example is CVaR.

12.2 CVaR optimization in RL

CVaR focuses on the extreme values of a random variable and is usually adopted in RL domains to avoid catastrophic outcomes. Policy gradient is an important method to optimize CVaR, e.g., see [33][49][7]. Since CVaR focuses on the worst returns, policy gradient techniques often ignore high-return trajectories [33, 50]. [7] proposed cross entropy soft risk optimization to improve sample efficiency by optimizing with respect to all trajectories while maintaining risk aversion. A time difference method is proposed in [51]. CVaR is also used for action selection in distributional RL literature, e.g., see [35][52][53].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points to limitations of the mean-variance criterion for risk-averse RL, and of the methods that optimize it. Instead, the paper proposes an alternative risk measure relying on Gini Deviation. The paper shows how to derive the PG for this metric, and demonstrates that it learns risk-averse policies more stably than methods that optimize the mean-variance.\n\nSTRENGTHS:\n1. The idea of using Gini Deviation for risk-averse RL is original as far as I know.\n2. The metric is demonstrated as more stable than mean-variance.\n\nWEAKNESSES:\nRisk-aversion in RL is often handled by optimizing risk measures. Many risk measures have been studied, e.g., mean-variance, var, cvar, entropic, distortion measures (some of them are briefly mentioned in the paper). There are studies about both the properties of different measures and how to optimize them. In particular, commonly desired properties of the metric are captured by the notion of coherent risk measures. For example, mean-variance is not a coherent risk measure, but CVaR is, and many works have recently studied it.\n\nA paper that proposes a new risk measure and claims to its benefits cannot ignore these works and the notion of coherence.\n\nConsidering the discussion above:\n1. The presentation of the paper contribution lacks a lot of relevant context, hence the novelty and contribution are difficult to judge.\n2. Relevant literature is hardly covered. There is no organized literature survey, and relevant works are not mentioned.\n3. The experiments mostly demonstrate the weakness of the mean-variance risk measure, which is already known and cannot justify the suggested metric just by itself. Since the paper justification is essentially empirical, this is a quite critical issue.\n\nFinally, the maze benchmark used in Sec 2.2, Sec 5.1 and Fig 1 is taken directly from [Greenberg et al. 2022], with neither crediting its original nor citing the corresponding work. Notice that using benchmarks and code (even if you modify them) without explicitly citing them in both paper and code may be considered Plagiarism.\n\nQUESTIONS:\nSee the issues discussed above under Weaknesses.\n\nAdditional question:\nAs specified in the paper, the current method uses multiple trajectories for a single policy gradient. However, quantile regression (as in QRDQN) allows to learn quantile value representation without this limitation. Could such quantile regression be used to overcome the multi-trajectory limitation in mean-GD as well?\n\nLIMITATIONS:\nAlgorithmic limitations are fairly discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper considers the problem of risk-sensitive reinforcement learning in the policy optimization setting, under the mean-variance risk criteria. They provide background on the problem and demonstrate the issues with learning an optimal policy due to the estimation of the variance requiring double samples. They demonstrate that previous approaches to this problem which use the immediate reward variance as a proxy are not desirable (Section 2). They introduce the Gini deviation as a surrogate for the variance, and demonstrate that through a Choquet integral characterization, it can be estimated without the double-sampling issue by leveraging quantile distributional RL. They incorporate this into a PPO-style algorithm and compare its performance against others across a range of risk-sensitive benchmarks.

STRENGTHS:
- Well presented motivation and contribution.
- Novel idea, which achieves strong empirical performance.

WEAKNESSES:
- There is little theoretical support for the algorithm, which would strengthen the contribution.

QUESTIONS:
- What is the space $\mathcal{M}^\infty$ introduced in Definition 1? It is never defined.

LIMITATIONS:
The authors discuss limitations of their work, and propose directions for future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to restrict the Gini deviation of the return for risk-averse reinforcement learning (RARL). This is motivated by a discussion of the shortcomings of variance-based RARL methods, such as scale dependence and instability due to large squared returns. A derivation of an expression for the gradient of the Gini deviation of the return serves as a basis for a practical algorithm, mean-GD (MG). It adds a penalty proportional to the Gini deviation to the reinforcement learning objective. The distribution of returns, which is required for the gradient of the Gini deviation of the return, is approximated with a finite number of sampled trajectories. An empirical evaluation of MG in the tabular and function approximation settings shows advantages of MG in learning risk-averse policies over return and reward-based baselines.\n\nSTRENGTHS:\nThe use of the Gini deviation as a new risk measure for RARL is motivated well by a concise discussion of the limitations of variance-based methods and is, to the best of my knowledge, new. The shortcomings of existing methods are illustrated with the help of a simple maze environment. The development of methods that are less sensitive to hyperparameters and environment properties such as reward scale is furthermore highly relevant for applications. \n\nThe derivation of a tractable expression for the gradient of the Gini deviation is done carefully and required assumptions are stated clearly. The experimental evaluation furthermore considers a sufficiently broad range of environments (tabular maze over lunar lander to MuJoCo environments) and baselines.\n\nWEAKNESSES:\nWhile the derivation of equations (15) and (16) are sufficiently clear, I would appreciate some intuition about the nature of the contribution of the Gini deviation penalty to the parameter update. The current version of the text does not really discuss the proposed algorithm but jumps right into the experimental evaluation. I think that for adoption of the algorithm by the community an intuitive understanding would be helpful.\n \nThere is furthermore a gap between the assumptions for the derivation and the actual environments used for the experiments and the illustration. The maze environment, for example, uses a shortest path objective which implies a discrete return. Many real-world environments will have complicated return distributions with atoms etc. It would be interesting to learn whether this poses a problem or if the assumptions are technical in nature and only required for the proof.\n\nThe significance of the experimental results is difficult to judge without further information. While I found the used hyperparameters in the supplementary material, I did not find information on how they were chosen. Was a hyperparameter optimization performed and, if yes, was it done manually, with a grid search or with some other algorithm? Without this information the frequent failures of the baselines are difficult to place. I would furthermore suggest to put vertical lines indicating the performance of MVPI in the figures (to avoid the problem of incompatible x axes). Right now it looks a bit like the most competitive baseline is only shown in the supplementary material where it is hard to compare it to the rest.\n\nRelated to this point: Part of the motivation for the proposed algorithm was to mitigate the need to extensively tune the hyperparameter $\\lambda$ for different environments/reward functions. It would therefore be interesting to see how sensitive MG as well as the baselines are to the hyperparameters controlling the risk-averseness.\n\nTypos: \n\n* In line 42 “We show that our method can learn risk-averse policy […]”, there is an “a” missing.\n* In line 69 the expression for the gradient of $M(\\theta)$ contains an $a$ and a $u(\\theta)$ which have not been introduced. Together with a different expression given in line 88, this looks like a typo.\n\nQUESTIONS:\n* It seems to me that the equality in line 234 is incorrect as the probability density of sampling a trajectory is not necessarily the same as the probability density of obtaining its reward. An extreme counter example would be an environment with a reward identical to zero but many possible trajectories. I do not think this poses a problem for the further derivation of MG as the set of trajectories that result in the same reward is independent of the policy parameters. However, it would still make sense to correct this formulation for better readability.\n* As the policy is defined to output values in $[0, 1]$ (line 48), does this mean discrete actions are assumed?\n* I would suggest to better relate the paragraph on the baselines (starting in line 281) to the discussion of prior methods in section 2. With the current formulation, it is not clear to the reader which baselines correspond to which of the methods discussed in section 2. Adding some clarification here would make interpreting the results a lot easier.\n* I would be interested in learning why most of the environments were specifically designed for this paper. Are no standard environments for RARL available that would have been useful?\n* Would it be possible to use the standard deviation instead of the variance to get scale invariance?\n* Is the variance of the estimate for the CDF in equation (16) a problem?\n* Related to the last question: Would it be possible to learn an approximation of the return distribution to reduce variance and get better convergence of the policy? The return distribution is clearly non-stationary but tracking non-stationary functions with estimates is not uncommon in reinforcement learning. Maybe this could help with asymptotic performance.\n* Why is MG performing poorly on HalfCheetah while MVPI seems to do well?\n\nLIMITATIONS:\nThe limitations of the MG algorithm, mainly low sample efficiency, have been addressed in a separate paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach to measure the risk for risk-averse reinforcement learning problems. \nTo be more precise, traditional methods usually use the variance of total return or per-step reward as the measurement of risk, since it is known that the variance is easy to calculate. Though easy to calculate, variance-based methods are observed to be sensitive to numerical scale and hard to optimize the policy. Based on the observation, the authors propose to use Gini deviation as the alternative to variance for measuring the risk. The authors study the properties of the new measurement and present how to estimate Gini deviation using empirical samples. Empirical experiments demonstrate that the new proposed methods can mitigate the limitation of variance-based methods while it can still learn high reward returns, compared with previous methods.\n\nSTRENGTHS:\n- The paper is well written, and the authors did an excellent job of presenting the limitation of variance methods before introducing Gini Deviation. As a result, readers can get a sense of the motivation for using Gini deviation as an alternative. \n\n- The authors thoroughly introduce the properties of Gini deviation and present how to estimate this quantity using empirical samples. Besides the authors also introduce how to estimate this quantity under the setting of using importance sampling and PPO. And I think through this, authors can get a better understanding how to use the quantity in different variant of policy gradient methods. \n\n- The authors also demonstrate the advantage of Gini Derivation via empirical experiments, for both tabular, discrete, and continuous control experiments.\n\nWEAKNESSES:\n- Though the authors present that Gini Deviation can be estimated via empirical samples, my primary concern is that it does not have close form and thus its estimation totally depends on the quality of the samples, if we do not have enough samples, then the variance or the bias of the estimation can lead to undesired optimization behavior. \n\n- Since Gini Deivation requires estimating the quantile, it may introduce additional complexity compared with variance-based method. \n\n- The authors claim that variance-based methods might be sensitive to numerical scales, but I have not seen any ablation study to directly compare this with Gini Deivation-based policy gradient method.\n\nQUESTIONS:\n- Can you clarify if we use empirical samples to estimate Gini Deviation, what would be the draw back? For example, what's the variance and bias of the new estimator using empirical samples if it has? \n\n- Would you mind explaining the complexity of the new proposed method? \n\n- Also, is it possible to demonstrate that with the new proposed measurement, the new estimator would be robust to numerical scales? (Maybe no need for empirical experiments, some intuition explanation would also be okay).\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

comments: SUMMARY:
This paper discusses the challenges of mean-variance RARL methods, and proposes the Gini deviation as a substitute to variance penalization.

STRENGTHS:
The paper is clearly written and well-motivated (by the disadvantages of return and instantaneous reward variance). The proposed method is evaluated in several experiments with comparator baselines, which seem convincing.

WEAKNESSES:
It seems the proposed algorithm has to sort its samples in every iteration (L245-248) which seems computationally expensive, and which may be sensitive to the parameters chosen for $n$? 

While I recognize that this is a largely empirical paper, I suppose the paper could have been made more complete by providing some convergence guarantees for policy gradient, e.g., to a stationary point.

QUESTIONS:
Included in “weaknesses”.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11