SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \mathbb {E} _ {z \sim Z _ {\theta}} \left[ \nabla_ {\theta} \log f _ {Z} (z; \theta) \int_ {z} ^ {b} (2 F _ {Z _ {\theta}} (t) - 1) d t \right] \tag {11}
$$
Proof. By Equation 8, the gradient of  $\mathbb{D}[Z_{\theta}] = \Phi_h(Z_{\theta})$  ( $h(\alpha) = -\alpha^2 + \alpha, \alpha \in [0,1]$ ) is
$$
\nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = \nabla_ {\theta} \Phi_ {h} (Z _ {\theta}) = \int_ {0} ^ {1} (2 \alpha - 1) \nabla_ {\theta} F _ {Z _ {\theta}} ^ {- 1} (\alpha) d \alpha = \int_ {0} ^ {1} (2 \alpha - 1) \nabla_ {\theta} q _ {\alpha} (Z; \theta) d \alpha . \tag {12}
$$
This requires to calculate the gradient for any  $\alpha$ -level quantile of  $Z_{\theta}$ , i.e.,  $\nabla_{\theta}q_{\alpha}(Z;\theta)$ . Based on the
assumptions and the definition of the  $\alpha$ -level quantile, we have  $\int_{-b}^{q_{\alpha}(Z;\theta)}f_Z(z;\theta)dz = \alpha$ . Taking a derivative and using the Leibniz rule we obtain
$$
0 = \nabla_ {\theta} \int_ {- b} ^ {q _ {\alpha} (Z; \theta)} f _ {Z} (z; \theta) d z = \int_ {- b} ^ {q _ {\alpha} (Z; \theta)} \nabla_ {\theta} f _ {Z} (z; \theta) d z + \nabla_ {\theta} q _ {\alpha} (Z; \theta) f _ {Z} \left(q _ {\alpha} (Z; \theta); \theta\right) \tag {13}
$$
Rearranging the term, we get  $\nabla_{\theta}q_{\alpha}(Z;\theta) = -\int_{-b}^{q_{\alpha}(Z;\theta)}\nabla_{\theta}f_{Z}(z;\theta)dz\cdot \left[f_{Z}\big(q_{\alpha}(Z;\theta);\theta \big)\right]^{-1}$ . Plugging back to Equation 12 gives us an intermediate version of  $\nabla_{\theta}\mathbb{D}[Z_{\theta}]$ .
$$
\nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \int_ {0} ^ {1} (2 \alpha - 1) \int_ {- b} ^ {q _ {\alpha} (Z; \theta)} \nabla_ {\theta} f _ {Z} (z; \theta) d z \cdot \left[ f _ {Z} \left(q _ {\alpha} (Z; \theta); \theta\right) \right] ^ {- 1} d \alpha \tag {14}
$$
By switching the integral order of Equation 14 and applying  $\nabla_{\theta}\log (x) = \frac{1}{x}\nabla_{\theta}x$ , we get the final gradient formula Equation 11. The full calculation is in Appendix 8.1.
4.2 Gini Deviation Policy Gradient via Sampling
In a typical application,  $Z$  in Section 4.1 would correspond to the performance of a system, e.g., the total return  $G_{0}$  in RL. Note that in order to compute Equation 11, one needs access to  $\nabla_{\theta}\log f_Z(z;\theta)$ : the sensitivity of the system performance to the parameters  $\theta$ . Usually, the system performance is a complicated function and calculating its probability distribution is intractable. However, in RL, the performance is a function of trajectories. The sensitivity of the trajectory distribution is often easy to compute. This naturally suggests a sampling based algorithm for gradient estimation.
Now consider Equation 11 in the context of RL, i.e.,  $Z = G_{0}$  and  $\theta$  is the policy parameter.
$$
\nabla_ {\theta} \mathbb {D} [ G _ {0} ] = - \mathbb {E} _ {g \sim G _ {0}} \left[ \nabla_ {\theta} \log f _ {G _ {0}} (g; \theta) \int_ {g} ^ {b} \left(2 F _ {G _ {0}} (t) - 1\right) d t \right] \tag {15}
$$
To sample from the total return variable  $G_{0}$ , we need to sample a trajectory  $\tau$  from the environment by executing  $\pi_{\theta}$  and then compute its corresponding return  $R_{\tau} \coloneqq r_1 + \gamma r_2 + \ldots + \gamma^{T-1}r_T$ , where  $r_t$  is the per-step reward at time  $t$ , and  $T$  is the trajectory length. The probability of the sampled return can be calculated as  $f_{G_0}(R_\tau; \theta) = \mu_0(s_0)\prod_{t=0}^{T-1}[\pi_\theta(a_t|s_t)p(r_{t+1}|s_t,a_t)]$ . The gradient of its log-likelihood is the same as that of  $P(\tau|\theta) = \mu_0(s_0)\prod_{t=0}^{T-1}[\pi_\theta(a_t|s_t)p(s_{t+1}|s_t,a_t)]$ , since the difference in transition probability does not alter the policy gradient. It is well known that  $\nabla_\theta\log P(\tau|\theta) = \sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta(a_t|s_t)$ .
For the integral part of Equation 15, it requires the knowledge of the CDF of  $G_{0}$ . In practice, this means we should obtain the full value distribution of  $G_{0}$ , which is usually not easy. One common approach to acquire an empirical CDF or quantile function (inverse CDF) is to get the quantile samples of a distribution and then apply some reparameterization mechanism. For instance, reparameterization is widely used in distributional RL for quantile function estimation. The quantile function has been parameterized as a step function [34, 35], a piece-wise linear function [36], or other higher order spline functions [37]. In this paper, we use the step function parameterization given its simplicity. To do so, suppose we have  $n$  trajectory samples  $\{\tau_i\}_{i=1}^n$  from the environment and their corresponding returns  $\{R_{\tau_i}\}_{i=1}^n$ , the returns are sorted in ascending order such that  $R_{\tau_1} \leq R_{\tau_2} \leq \ldots \leq R_{\tau_n}$ , then each  $R_{\tau_i}$  is regarded as a quantile value of  $G_{0}$  corresponding to the quantile level  $\alpha_i = \frac{1}{2} \left( \frac{i-1}{n} + \frac{i}{n} \right)$ , i.e., we assume  $q_{\alpha_i}(G_0; \theta) = R_{\tau_i}$ . This strategy is also common in distributional RL, e.g., see [34, 38]. The largest return  $R_{\tau_n}$  is regarded as the upper bound  $b$  in Equation 15.
Thus, given ordered trajectory samples  $\{\tau_i\}_{i=1}^n$ , an empirical estimation for GD policy gradient is (a detailed example is given in Appendix 8.2)
$$
- \frac {1}{n - 1} \sum_ {i = 1} ^ {n - 1} \eta_ {i} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {i, t} \mid s _ {i, t}\right), \text {w h e r e} \eta_ {i} = \sum_ {j = i} ^ {n - 1} \frac {2 j}{n} \left(R _ {\tau_ {j + 1}} - R _ {\tau_ {j}}\right) - \left(R _ {\tau_ {n}} - R _ {\tau_ {i}}\right) \tag {16}
$$
The sampled trajectories can be used to estimate the gradient for  $\mathbb{E}[G_0]$  in the meantime, e.g., the well known vanilla policy gradient (VPG), which has the form  $\mathbb{E}_{\tau}[R_{\tau}\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_t|s_t)]$ . It is more often used as  $\mathbb{E}_{\tau}[\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_t|s_t)\gamma^t g_t]$ , where  $g_{t} = \sum_{t^{\prime} = t}^{T - 1}\gamma^{t^{\prime} - t}r(s_{t^{\prime}},a_{t^{\prime}})$ , which is known to have lower variance. Usually  $g_{t}$  is further subtracted by a value function to improve stability, called REINFORCE with baseline. Apart from VPG, another choice to maximize  $\mathbb{E}[G_0]$  is using PPO [39].
4.3 Incorporating Importance Sampling
For on-policy policy gradient, samples are abandoned once the policy is updated, which is expensive for our gradient calculation since we are required to sample  $n$  trajectories each time. To improve
the sample efficiency to a certain degree, we incorporate importance sampling (IS) to reuse samples for multiple updates in each loop. For each  $\tau_{i}$ , the IS ratio is  $\rho_{i} = \prod_{t=0}^{T-1} \pi_{\theta}(a_{i,t}|s_{i,t}) / \pi_{\hat{\theta}}(a_{i,t}|s_{i,t})$ , where  $\hat{\theta}$  is the old policy parameter when  $\{\tau_{i}\}_{i=1}^{n}$  are sampled. Suppose the policy gradient for maximizing  $\mathbb{E}[G_0]$  is REINFORCE baseline. With IS, the empirical mean-GD policy gradient is
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {i} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {i, t} \mid s _ {i, t}\right) \left(g _ {i, t} - V \left(s _ {i, t}\right)\right) + \frac {\lambda}{n - 1} \sum_ {i = 1} ^ {n - 1} \rho_ {i} \eta_ {i} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {i, t} \mid s _ {i, t}\right) \tag {17}
$$
where  $g_{i,t}$  is the sum of rewards-to-go as defined above.  $V(s_{i,t})$  is the value function. The first part can also be replaced by PPO-Clip policy gradient. Then we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \min  \left(\frac {\pi_ {\theta} \left(a _ {i , t} \mid s _ {i , t}\right)}{\pi_ {\hat {\theta}} \left(a _ {i , t} \mid s _ {i , t}\right)} A _ {i, t}, f (\epsilon , A _ {i, t})\right) + \frac {\lambda}{n - 1} \sum_ {i = 1} ^ {n - 1} \rho_ {i} \eta_ {i} \sum_ {t = 0} ^ {T - 1} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {i, t} \mid s _ {i, t}\right) \tag {18}
$$
where  $A_{i,t}$  is the advantage estimate, and  $f()$  is the clip function in PPO with  $\epsilon$  being the clip range, i.e.  $f(\epsilon, A_{i,t}) = \mathrm{clip}(\frac{\pi_{\theta}(a_{i,t}|s_{i,t})}{\pi_{\hat{\theta}}(a_{i,t}|s_{i,t})}, 1 - \epsilon, 1 + \epsilon)A_{i,t}$ .
The extreme IS values  $\rho_{i}$  will introduce high variance to the policy gradient. To stabilize learning, one strategy is that in each training loop, we only select  $\tau_{i}$  whose  $\rho_{i}$  lies in  $[1 - \delta, 1 + \delta]$ , where  $\delta$  controls the range. The updating is terminated if the chosen sample size is lower than some threshold, e.g.,  $\beta \cdot n$ ,  $\beta \in (0,1)$ . Another strategy is to directly clip  $\rho_{i}$  by a constant value  $\zeta$ , i.e.,  $\rho_{i} = \min(\rho_{i}, \zeta)$ , e.g., see [40]. In our experiments, we use the first strategy for Equation 17, and the second for Equation 18. We leave other techniques for variance reduction of IS for future study. The full algorithm that combines GD with REINFORCE and PPO is in Appendix 9.
Our experiments were designed to serve two main purposes. First, we investigate whether the GD policy gradient approach could successfully discover risk-averse policies in scenarios where variance-based methods tend to fail. To accomplish this, we manipulated reward choices to assess the ability of the GD policy gradient to navigate risk-averse behavior. Second, we sought to verify the effectiveness of our algorithm in identifying risk-averse policies that have practical significance in both discrete and continuous domains. We aimed to demonstrate its ability to generate meaningful risk-averse policies that are applicable and valuable in practical settings.
Baselines. We compare our method with the original mean-variance policy gradient (Equation 2, denoted as MVO), Tamar's method [8] (denoted as Tamar), MVP [12], and MVPI [14]. Specifically, MVO requires multiple trajectories to compute  $J(\theta) \nabla_{\theta} J(\theta)$ . We use  $\frac{n}{2}$  trajectories to estimate  $J(\theta)$  and another  $\frac{n}{2}$  to estimate  $\nabla_{\theta} J(\theta)$ , where  $n$  is the sample size. MVPI is a general framework for policy iteration whose inner risk-neutral RL solver is not specified. For the environment with discrete actions, we build MVPI on top of Q-Learning or DQN [41]. For continuous action environments, MVPI is built on top of TD3 [42] as in [14]. We use REINFORCE to represent the REINFORCE with baseline method. We use MG as a shorthand of mean-GD to represent our method. In each domain, we ensure each method's policy or value nets have the same neural network architecture.
For policy updating, MVO and MG collect  $n$  episodes before updating the policy. In contrast, Tamar and MVP update the policy after each episode. Non-tabular MVPI updates the policy at each environment step. In hyperparameter search, we use the parameter search range in MVPI [14] as a reference, making reasonable refinements to find an optimal parameter setting. Please refer to Appendix 10 for any missing implementation details. Code is available at<sup>2</sup>.
5.1 Tabular case: Modified Guarded Maze Problem
This domain is a modified Guarded Maze [7] that was previously described in Section 2.2. The original Guarded Maze is asymmetric with two openings to reach the top path (in contrast to a single opening for the bottom path). In addition, paths via the top tend to be longer than paths via the bottom. We modified the maze to be more symmetric in order to reduce preferences arising from certain exploration strategies that might be biased towards shorter paths or greater openings, which may confound risk aversion. Every movement before reaching the goal receives a reward of  $-1$  except moving to the red state, where the reward is sampled from  $\{-15, -1, 13\}$  with probability  $\{0.4, 0.2, 0.4\}$  (mean is  $-1$ ) respectively. The maximum episode length is 100. MVO and MG collect  $n = 50$  episodes before updating the policy. Agents are tested for 10 episodes per evaluation.
$^{2}$ https://github.com/miyunluo/mean-gini
Figure 2: (a) Policy evaluation return and (b,c) optimal risk-aversion rate v.s. training episodes in Maze. Curves are averaged over 10 seeds with shaded regions indicating standard errors. For optimal risk-aversion rate, higher is better.
The failure of variance-based baselines under simple reward manipulation. We first set the goal reward to 20. Here, we report the optimal risk-aversion rate achieved during training. Specifically, we measure the percentage of episodes that obtained the optimal risk-averse path, represented by the white color path in Figure 1, out of all completed episodes up to the current stage of training.
Notice that MVO performs well in this domain when using double sampling to estimate its gradient. Then we increase the goal reward to 40. This manipulation does not affect the return variance of the optimal risk-averse policy, since the reward is deterministic. However, the performances of MVO, Tamar, MVP all decrease, since they are more sensitive to the numerical scale of the return (due to the  $\mathbb{E}[G_0^2]$  term introduced by variance). MVPI is a policy iteration method in this problem, whose learning curve is not intuitive to show. It finds the optimal risk-averse path when the goal reward is 20, but it fails when the goal reward is 40. An analysis for MVPI is given in Appendix 10.2.2. We compare the sensitivity of different methods with respect to  $\lambda$  in Appendix 10.2.4.
Remark. Scaling rewards by a small factor is not an appropriate approach to make algorithms less sensitive to the numerical scale for both total return variance and per-step reward variance, since it changes the original mean-variance objective in both cases.
5.2 Discrete control: LunarLander
This domain is taken from OpenAI Gym Box2D environments [15]. We refer readers to its official documents for the full description. Originally, the agent is awarded 100 if it comes to rest. We divide the ground into two parts by the middle line of the landing pad, as shown in Figure 10 in Appendix. If the agent lands in the right area, an additional noisy reward sampled from  $\mathcal{N}(0,1)$  times 90 is given. A risk-averse agent should learn to land at the left side as much as possible. We include REINFORCE as a baseline to demonstrate the risk-aversion of our algorithm. REINFORCE, MVO and MG collect  $n = 30$  episodes before updating their policies. Agents are tested for 10 episodes per evaluation.
We report the rate at which different methods land on the left in Figure 3(b) (we omit the failed methods), i.e., the percentage of episodes successfully landing on the left per evaluation. MVO, Tamar, and MVP do not learn reasonable policies in this domain according to their performances in Figure 3(a). MVP learns to land in the middle of the learning phase, but soon after fails to land. Since successfully landing results in a large return (success reward is 100), the return square term  $(\mathbb{E}[G_0^2])$  introduced by variance makes MVP unstable. MVPI also fails to land since  $\mathbb{V}[R]$  is sensitive to the numerical scale of rewards. In this domain, the success reward is much larger than other reward values. Furthermore, reward modification in MVPI turns large success rewards into negative values, which prevents the agent from landing on the ground. MG achieves a comparable return with REINFORCE, but clearly learns a risk-averse policy by landing more on the left.
Figure 3: (a) Policy evaluation return and (b) left-landing rate (i.e., risk-averse landing rate) v.s. training episodes in LunarLander. Curves are averaged over 10 seeds with shaded regions indicating standard errors. For landing left rate, higher is better.
Figure 4: (a,c,e) Policy evaluation return and (b,d,f) location visiting rate v.s. training episodes in Mujoco of episode-based methods. Curves are averaged over 10 seeds with shaded regions indicating standard errors. For location visiting rate, lower is better.
5.3 Continuous control: Mujoco
Mujoco [16] is a collection of robotics environments with continuous states and actions in OpenAI Gym [15]. Here, we selected three domains (InvertedPendulum, HalfCheetah, and Swimmer) that are conveniently modifiable, where we are free to modify the rewards to construct risky regions in the environment (Through empirical testing, risk-neutral learning failed when similar noise was introduced to other Mujoco domains. Consequently, identifying the cause for the failure of risk-averse algorithms on other domains became challenging). Motivated by and following [43, 44], we define a risky region based on the X-position. For instance, if X-position  $>0.01$  in InvertedPendulum, X-position  $< -3$  in HalfCheetah, and X-position  $>0.5$  in Swimmer, an additional noisy reward sampled from  $\mathcal{N}(0,1)$  times 10 is given. Location information is appended to the agent's observation. A risk-averse agent should reduce the time it visits the noisy region in an episode. We also include the risk-neutral algorithms as baselines to highlight the risk-aversion degree of different methods.
All the risk-averse policy gradient algorithms still use VPG to maximize the expected return in InvertedPendulum (thus the risk-neutral baseline is REINFORCE). Using VPG is also how these methods are originally derived. However, VPG is not good at more complex Mujoco domains, e.g., see [45]. In HalfCheetah and Swimmer, we combine those algorithms with PPO-style policy gradient to maximize the expected return. Minimizing the risk term remains the same as their original forms. MVPI is an off-policy time-difference method in Mujoco. We train it with 1e6 steps instead of as many episodes as other methods. MVO and MG sample  $n = 30$  episodes in InvertedPendulum and  $n = 10$  in HalfCheetah and Swimmer before updating policies. Agents are tested for 20 episodes per evaluation. The percentage of time steps visiting the noisy region in an episode is shown in Figure 4(b,d,f). Compared with other return variance methods, MG achieves a higher return while maintaining a lower visiting rate. Comparing MVPI and TD3 against episode-based algorithms like MG is not straightforward within the same figure due to the difference in parameter update frequency. MVPI and TD3 update parameters at each environment time step. We shown their learning curves in Figure 5. MVPI also learns risk-averse policies in all three domains according to its learning curves.
We further design two domains using HalfCheetah and Swimmer. The randomness of the noisy reward linearly decreases when agent's forward distance grows. To maximize the expected return and minimize risk, the agent has to move forward as far as possible. The results are shown in Figures 18,19 in Appendix. In these two cases, only MG shows a clear tendency of moving forward, which suggests our method is less sensitive to reward choices compared with methods using  $\mathbb{V}[R]$ .
The return variance and GD during learning in the above environments are also reported in Appendix 10. In general, when other return variance based methods can find the risk-averse policy, MG maintains a lower or comparable return randomness when measured by both variance and GD. When
Figure 5: (a,c,e) Policy evaluation return and (b,d,f) location visiting rate v.s. training episodes in Mujoco of TD3 and MVPI. Curves are averaged over 10 seeds with shaded regions indicating standard errors. For location visiting rate, lower is better.
other methods fail to learn a reasonably good risk-averse policy, MG consistently finds a notably higher return and lower risk policy compared with risk-neutral methods. MVPI has the advantage to achieve low return randomness in location based risky domains, since minimizing  $\mathbb{V}[R]$  naturally avoids the agent from visiting the noisy region. But it fails in distance-based risky domains.
This paper proposes to use a new risk measure, Gini deviation, as a substitute for variance in mean-variance RL. It is motivated to overcome the limitations of the existing total return variance and per-step reward variance methods, e.g., sensitivity to numerical scale and hindering of policy learning. A gradient formula is presented and a sampling-based policy gradient estimator is proposed to minimize such risk. We empirically show that our method can succeed when the variance-based methods will fail to learn a risk-averse or a reasonable policy. This new risk measure may inspire a new line of research in RARL. First, one may study the practical impact of using GD and variance risk measures. Second, hybrid risk-measure may be adopted in real-world applications to leverage the advantages of various risk measures.
Limitations and future work. Our mean-GD policy gradient requires sampling multiple trajectories for one parameter update, making it less sample efficient compared to algorithms that can perform updates per environment step or per episode. As a result, one potential avenue for future work is to enhance the sample efficiency of our algorithm. This can be achieved by more effectively utilizing off-policy data or by adapting the algorithm to be compatible with online, incremental learning.
We thank Ruodu Wang from University of Waterloo and Han Wang from University of Alberta for valuable discussions and insights. Resources used in this work were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute https://vectorinstitute.ai/partners/ and the Natural Sciences and Engineering Council of Canada. Yudong Luo is also supported by a David R. Cheriton Graduate Scholarship, a President's Graduate Scholarship, and an Ontario Graduate Scholarship. Guiliang Liu's research was in part supported by the Start-up Fund UDF01002911 of the Chinese University of Hong Kong, Shenzhen. Yangchen Pan acknowledges funding from the Turing AI World Leading Fellow.
[1] Mohammad Naghshvar, Ahmed K Sadek, and Auke J Wiggers. Risk-averse behavior planning for autonomous driving under uncertainty. arXiv preprint arXiv:1812.01254, 2018.