SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[2] Tomas Björk, Agatha Murgoci, and Xun Yu Zhou. Mean-variance portfolio optimization with state-dependent risk aversion. Mathematical Finance: An International Journal of Mathematics, Statistics and Financial Economics, 24(1):1-24, 2014.
[3] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[4] Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2):294-311, 2002.
[5] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.
[6] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
[7] Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639–32652, 2022.
[8] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. International Conference on Machine Learning, 2012.
[9] Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. Advances in neural information processing systems, 26, 2013.
[10] Harry M Markowitz and G Peter Todd. Mean-variance analysis in portfolio choice and capital markets, volume 66. John Wiley & Sons, 2000.
[11] Duan Li and Wan-Lung Ng. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation. Mathematical finance, 10(3):387-406, 2000.
[12] Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, and Daesub Yoon. A block coordinate ascent algorithm for mean-variance optimization. Advances in Neural Information Processing Systems, 31, 2018.
[13] Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatility reduction. International Joint Conference on Artificial Intelligence, 2020.
[14] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10905–10913, 2021.
[15] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[16] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[17] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[18] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997.
[19] Shie Mannor and John Tsitsiklis. Mean-variance optimization in markov decision processes. International Conference on Machine Learning, 2011.
[20] Matthew J Sobel. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794-802, 1982.
[40] Léon Bottou, Jonas Peters, Joaquin Quinonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
[41] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
[42] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[43] Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference on Machine Learning, pages 7390-7399. PMLR, 2021.
[44] Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations, 2023.
[45] OpanAI. Performance in each mujoco environment. https://spinningup.openai.com/en/latest/spinningup/bench.html.
[46] Arjun K Gupta and Mohammad AS Aziz. Convex ordering of random variables and its applications in econometrics and actuarial science. European Journal of Pure and Applied Mathematics, 3(5):779-785, 2010.
[47] Michael Rothschild and Joseph E Stiglitz. Increasing risk: I. a definition. In Uncertainty in Economics, pages 99–121. Elsevier, 1978.
[48] Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203-228, 1999.
[49] Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. Advances in Neural Information Processing Systems, 32, 2019.
[50] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In International Conference on Learning Representations, 2016.
[51] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach. Advances in neural information processing systems, 28, 2015.
[52] Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint arXiv:1911.03618, 2019.
[53] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spanan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10639-10646, 2021.
Supplementary Information
7 Convex Order, Gini Deviation, and Variance
Convex order describes dominance in terms of variability and is widely used in actuarial science.
Definition 2 ([46]). Consider two random variables  $X$  and  $Y$ ,  $X$  is called convex order smaller than  $Y$ , succinctly  $X \leq_{\mathrm{cx}} Y$ , if  $\mathbb{E}[\psi(X)] \leq \mathbb{E}[\psi(Y)]$ , for all convex function  $\psi()$ , assuming that both expectations exist.
In convex order  $X \leq_{\mathrm{cx}} Y$ ,  $Y$  is also called a mean-preserving spread of  $X$  [47], which intuitively means that  $Y$  is more spread-out (and hence more random) than  $X$ . Thus, it is often desirable for a measure of variability to be monotone with respect to convex order [25]. Both variance and GD, as a measure of variability, are consistent with convex order, i.e.,
- If  $X \leq_{\mathrm{cx}} Y$ , then  $\mathbb{V}[X] \leq \mathbb{V}[Y]$  for all  $X, Y \in \mathcal{M}$
- If  $X \leq_{\mathrm{cx}} Y$ , then  $\mathbb{D}[X] \leq \mathbb{D}[Y]$  for all  $X, Y \in \mathcal{M}$
Proof. It is immediate that  $X \leq_{\mathrm{cx}} Y$  implies  $\mathbb{E}[X] = \mathbb{E}[Y]$ . If we take convex function  $\psi(x) = x^2$ , we can get the order of variance  $\mathbb{V}[X] \leq \mathbb{V}[Y]$ . For the proof of GD, please refer to the following Lemma. Recall that GD can be expressed in the form of signed Choquet integral with a concave function  $h$ .
Lemma 3 ([28], Theorem 2). Convex order consistency of a signed Choquet integral is equivalent to its distortion function  $h$  being concave, i.e.,  $X \leq_{\mathrm{cx}} Y$  if and only if the signed Choquet integral  $\Phi_h(X) \leq \Phi_h(Y)$  for all concave functions  $h \in \mathcal{H}$ .
8 GD Gradient Formula Calculation
8.1 General GD Gradient Formula
Consider a random variable  $Z$ , whose distribution function is controlled by  $\theta$ . Recall that  $q_{\alpha}(Z; \theta)$  represents the  $\alpha$ -level quantile of  $Z_{\theta}$ . According to Equation 14, the gradient of GD is
$$
\nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = \nabla_ {\theta} \Phi_ {h} (Z _ {\theta}) = - \int_ {0} ^ {1} (2 \alpha - 1) \int_ {- b} ^ {q _ {\alpha} (Z; \theta)} \nabla_ {\theta} f _ {Z} (z; \theta) d z \frac {1}{f _ {Z} (q _ {\alpha} (Z ; \theta) ; \theta)} d \alpha
$$
To make the integral over  $\alpha$  clearer, we rewrite  $q_{\alpha}(Z;\theta)$  as  $F_{Z_{\theta}}^{-1}(\alpha)$ , where  $F_{Z_{\theta}}$  is the CDF.
$$
\nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \int_ {0} ^ {1} (2 \alpha - 1) \int_ {- b} ^ {F _ {Z _ {\theta}} ^ {- 1} (\alpha)} \nabla_ {\theta} f _ {Z} (z; \theta) d z \frac {1}{f _ {Z} (F _ {Z _ {\theta}} ^ {- 1} (\alpha) ; \theta)} d \alpha
$$
Switching the integral order, we get
$$
\begin{array}{l} \nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \int_ {- b} ^ {b} \int_ {F _ {Z _ {\theta}} (z)} ^ {1} (2 \alpha - 1) \nabla_ {\theta} f _ {Z} (z; \theta) \frac {1}{f _ {Z} \left(F _ {Z _ {\theta}} ^ {- 1} (\alpha) ; \theta\right)} d \alpha d z \tag {19} \\ = - \int_ {- b} ^ {b} \nabla_ {\theta} f _ {Z} (z; \theta) \int_ {F _ {Z _ {\theta}} (z)} ^ {1} (2 \alpha - 1) \frac {1}{f _ {Z} (F _ {Z _ {\theta}} ^ {- 1} (\alpha) ; \theta)} d \alpha d z \\ \end{array}
$$
Denote  $t = F_{Z_{\theta}}^{-1}(\alpha)$ , then  $\alpha = F_{Z_{\theta}}(t)$ . Here, we further change the inner integral from  $d\alpha$  to  $dF_{Z_{\theta}}(t)$ , i.e.,  $d\alpha = dF_{Z_{\theta}}(t) = f_Z(t;\theta)dt$ . The integral range for  $t$  is now from  $F_{Z_{\theta}}^{-1}(F_{Z_{\theta}}(z)) = z$  to  $F_{Z_{\theta}}^{-1}(1) = b$ .
$$
\begin{array}{l} \nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \int_ {- b} ^ {b} \nabla_ {\theta} f _ {Z} (z; \theta) \int_ {z} ^ {b} \left(2 F _ {Z _ {\theta}} (t) - 1\right) \frac {1}{f _ {Z} (t ; \theta)} d F _ {Z _ {\theta}} (t) d z \tag {20} \\ = - \int_ {- b} ^ {b} \nabla_ {\theta} f _ {Z} (z; \theta) \int_ {z} ^ {b} \left(2 F _ {Z _ {\theta}} (t) - 1\right) d t d z \\ \end{array}
$$
Figure 6: An example of parameterizing (inverse) CDF given six quantiles. The function is highlighted in the bold line of orange color.
Applying  $\nabla_{\theta}\log (x) = \frac{1}{x}\nabla_{\theta}x$  to  $\nabla_{\theta}f_{Z}(z;\theta)$ , we have
$$
\begin{array}{l} \nabla_ {\theta} \mathbb {D} [ Z _ {\theta} ] = - \int_ {- b} ^ {b} f _ {Z} (z; \theta) \nabla_ {\theta} \log f _ {Z} (z; \theta) \int_ {z} ^ {b} \left(2 F _ {Z _ {\theta}} (t) - 1\right) d t d z \tag {21} \\ = - \mathbb {E} _ {z \sim Z _ {\theta}} \Big [ \nabla_ {\theta} \log f _ {Z} (z; \theta) \int_ {z} ^ {b} \big (2 F _ {Z _ {\theta}} (t) - 1 \big) d t \Big ] \\ \end{array}
$$
8.2 GD Policy Gradient via Sampling
Following Section 4.2, we consider estimating the following equation via sampling.
$$
\nabla_ {\theta} \mathbb {D} [ G _ {0} ] = - \mathbb {E} _ {g \sim G _ {0}} \left[ \nabla_ {\theta} \log f _ {G _ {0}} (g; \theta) \int_ {g} ^ {b} \left(2 F _ {G _ {0}} (t) - 1\right) d t \right]
$$
Here sampling from  $G_{0}$  corresponds to sampling trajectory  $\tau$  with its return  $R_{\tau}$  from the environment. As discussed in the main paper, for a trajectory  $\tau_{i}$ ,  $\nabla_{\theta}\log f_{G_0}(R_{\tau_i},\theta)$  is estimated as  $\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_{i,t}|s_{i,t})$ .
The CDF function  $F_{G_0}$  is parameterized by a step function given its quantiles  $\{R_{\tau_i}\}_{i=1}^n$ , which satisfy  $R_{\tau_1} \leq R_{\tau_2} \leq \ldots \leq R_{\tau_n}$ . An example of the step function is shown in Figure 6. With this parameterization, the integral over CDF can be regarded as the area below the step function.
Thus, for each  $\tau_{i}$ , the integral over CDF is approximated as  $(R_{\tau_n}$  is treated as  $b)$
$$
\int_ {R _ {\tau_ {i}}} ^ {R _ {\tau_ {n}}} 2 F _ {G _ {0}} (t) d t \approx \sum_ {j = i} ^ {n - 1} 2 \times \frac {j}{n} \left(R \left(\tau_ {j + 1}\right) - R \left(\tau_ {j}\right)\right) \tag {22}
$$
Aggregating all the calculations together yields Equation 16.
9 Mean-GD Policy Gradient Algorithm
We consider maximizing  $\mathbb{E}[G_0] - \lambda \mathbb{D}[G_0]$  in this paper. Maximizing the first term, i.e.,  $\mathbb{E}[G_0]$  has been widely studied in risk-neutral RL. For on-policy policy gradient, we can use vanilla policy gradient (VPG), e.g., REINFORCE with baseline, or more advanced techniques like PPO [39].
To make fair comparison with other risk-averse policy gradient methods, we first initialize mean-GD policy gradient with VPG. To improve sample efficiency, we incorporate IS for multiple updates. Taking advantage of having  $n$  samples, we can select those trajectories whose IS ratio is in the range  $[1 - \delta, 1 + \delta]$  for calculation to reduce gradient variance. In more complex continuous domains, e.g., Mujoco, we combine GD policy gradient with PPO since VPG is not good at Mujoco [45]. However, in this case, policy may have a significant difference per update, where the former IS selection strategy can no longer be applied. Then we directly clip the IS ratio by a constant, though it is biased [40]. The full algorithms are summarized in Algorithm 1 and 2. We omit the parameters for PPO in Algorithm 2 for simplicity. We still report the PPO parameter settings in Section 10.
Algorithm 1 Mean-Gini Deviation Policy Gradient (with REINFORCE baseline)
Input: Iterations number  $K$  sample size  $n$  inner update number  $M$  policy learning rate  $\alpha_{\theta}$  value   
learning rate  $\alpha_{\phi}$  importance sampling range  $\delta$  , inner termination parameter  $\beta$  , trade-off parameter   
. Initialize policy  $\pi_{\theta}$  parameter  $\theta$  ,value  $V_{\phi}$  parameter  $\phi$    
for  $k = 1$  to  $K$  do Sample  $n$  trajectories  $\{\tau_i\}_{i = 1}^n$  by  $\pi_{\theta}$  , compute return  $\{R(\tau_i)\}_{i = 1}^n$  Compute rewards-to-go for each state in  $\tau_{i}$  ..  $g_{i,t}$    
for  $m = 1$  to  $M$  do Compute importance sampling ratio for each trajectory  $\{\rho_i\}_{i = 1}^n$  Select  $\mathcal{D} = \{\tau_s\}$  whose  $\rho_{s}\in [1 - \delta ,1 + \delta ]$  Sort trajectories such that  $R(\tau_1)\leq \ldots \leq R(\tau_{|\mathcal{D}|})$  if  $|\mathcal{D}| <   n\cdot \beta$  then break end if mean_grad  $= 0$  gini_grad  $= 0$    
for  $i = 1$  to  $|\mathcal{D}|$  do mean_grad  $+ = \rho_{i}\cdot \sum_{0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_{i,t}|s_{i,t})(g_{i,t} - V\phi (s_{i,t}))$  Update  $V_{\phi}$  by mean-squared error  $\frac{1}{T}\sum_{t = 0}^{T - 1}(V_{\phi}(s_{i,t}) - g_{i,t})^{2}$  with learning rate  $\alpha_{\phi}$  end for   
for  $i = 1$  to  $|\mathcal{D}| - 1$  do gini_grad  $+ = -\rho_{i}\cdot \eta_{i}\sum_{t = 0}^{T - 1}\nabla_{\theta}\log \pi_{\theta}(a_{i,t}|s_{i,t})$  where  $\eta_{i} = \sum_{j = i}^{|\mathcal{D}| - 1}\frac{2j}{|\mathcal{D}|} (R_{j + 1} - R_j) - (R_{|\mathcal{D}|} - R_i)$  end for Update  $\pi_{\theta}$  by  $(\frac{1}{|\mathcal{D}|}$  mean_grad -  $\frac{\lambda}{|\mathcal{D}| - 1}$  gini_grad) with learning rate  $\alpha_{\theta}$  (Equation 17) end for   
end for
10 Experiments Details
10.1 General Descriptions of Different Methods
Among the methods compared in this paper, Tamar [8] and MVP [12] are on-policy policy gradient methods. MVO and MG are policy gradient methods, but sample  $n$  trajectories and use IS to update. Non-tabular MVPI [14] is an off-policy time-difference method.
Policy gradeint methods. MVO, Tamar, MVP, are originally derived based on VPG. Since VPG is known to have a poor performance in Mujoco, we also combined these mean-variance methods with PPO. Thus these mean-variance methods and our mean-GD method have different instantiations to maximize the expected return in different domains:
- With VPG: in Maze, LunarLander, InvertedPendulum.
- With PPO: in HalfCheetah, Swimmer.
When the risk-neutral policy gradient is VPG, for MVO and MG, it is REINFORCE with baseline; for Tamar and MVP, we strictly follow their papers to implement the algorithm, where no value function is used. MVO and MG collect  $n$  trajectories and use the IS strategy in Algorithm 1 to update policies. Tamar and MVP do not need IS, and update policies at the end of each episode.
When the risk-neutral policy gradient is PPO, we augment PPO with the variance or GD policy gradient from the original methods. MVO and MG collect  $n$  trajectories and use the IS strategy in Algorithm 2 to compute the gradient for the risk term. Tamar and MVP still update policies once at the end of each episode.
MVPI. We implemented three versions of MVPI in different domains:
- Tabular: in Maze, MVPI is a policy iteration method (Algorithm 1 in [14]).
- With DQN: in LunarLander, since this environment has discrete action space.
- With TD3: in InvertedPendulum, HalfCheetah, Swimmer, since these environments have continuous action space.
Algorithm 2 Mean-Gini Deviation Policy Gradient (with PPO)
Input: Iterations number  $K$ , sample size  $n$ , inner update number  $M$ , policy learning rate  $\alpha_{\theta}$ , value learning rate  $\alpha_{\phi}$ , importance sampling clip bound  $\zeta$ , trade-off parameter  $\lambda$ . Initialize policy  $\pi_{\theta}$  parameter  $\theta$ , value  $V_{\phi}$  parameter  $\phi$ .  
for  $k = 1$  to  $K$  do  
Sample  $n$  trajectories  $\{\tau_i\}_{i=1}^n$  by  $\pi_{\theta}$ , compute return  $\{R(\tau_i)\}_{i=1}^n$   
Compute rewards-to-go for each state in  $\tau_i$ :  $g_{i,t}$   
Compute advantages for each state-action in  $\tau_i$ :  $A(s_{i,t},a_{i,t})$  based on current  $V_{\phi}$   
for  $m = 1$  to  $M$  do  
Compute importance sampling ratio for each trajectory  $\{\rho_i\}_{i=1}^n$ , and  $\rho_i = \min(\rho_i,b)$   
Sort trajectories such that  $R(\tau_1) \leq \ldots \leq R(\tau_n)$   
mean_grad = 0, gini_grad = 0  
for  $i = 1$  to  $n$  do  
mean_grad += PPO-Clip actor grad  
Update  $V_{\phi}$  by mean-squared error  $\frac{1}{T}\sum_{t=0}^{T-1}(V_{\phi}(s_{i,t}) - g_{i,t})^2$  with learning rate  $\alpha_{\phi}$   
end for  
for  $i = 1$  to  $n-1$  do  
gini_grad += - $\rho_i \cdot \eta_i$ $\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta(a_{i,t}|s_{i,t})$ , where  
 $\eta_i = \sum_{j=i}^{n-1}\frac{2j}{n}(R_{j+1} - R_j) - (R_n - R_i)$   
end for  
Update  $\pi_{\theta}$  by  $(\frac{1}{nT}$  mean_grad -  $\frac{\lambda}{(n-1)T}$  gini_grad) with learning rate  $\alpha_{\theta}$  (Equation 18)  
end for
We summarize the components required in different methods in Table 1.
Table 1: Model components in different methods
Policy func	Value func	Additional training variables
MVO-VPG	✓	✓
MVO-PPO	✓	✓
Tamar-VPG	✓	×	J,V (mean, variance)
Tamar-PPO	✓	✓	J,V (mean, variance)
MVP-VPG	✓	×	y (dual variable)
MVP-PPO	✓	✓	y (dual variable)
MG-VPG	✓	✓
MG-PPO	✓	✓
MVPI-Q-Learning	×	✓
MVPI-DQN	×	✓
MVPI-TD3	✓	✓
10.2 Modified Guarded Maze Problem
The maze consists of a  $6 \times 6$  grid. The agent can visit every free cell without a wall. The agent can take four actions (up, down, left, right). The maximum episode length is 100.
Policy function. For methods requiring a policy function, i.e., MVO, Tamar, MVP, MG, the policy is represented as
$$
\pi_ {\theta} (a | s) = \frac {e ^ {\phi (s , a) \cdot \theta}}{\sum_ {b} e ^ {\phi (s , b) \cdot \theta}} \tag {23}
$$
where  $\phi(s, a)$  is the state-action feature vector. Here we use one-hot encoding to represent  $\phi(s, a)$ . Thus, the dimension of  $\phi(s, a)$  is  $6 \times 6 \times 4$ . The derivative of the logarithm is
$$
\nabla_ {\theta} \log \pi_ {\theta} (a | s) = \phi (s, a) - \mathbb {E} _ {b \sim \pi_ {\theta} (\cdot | s)} \phi (s, b) \tag {24}
$$
Value function. For methods requiring a value function, i.e., REINFORCE baseline used in MVO and MG, and Q-learning in MVPI, the value function is represented as  $V_{\omega}(s) = \phi(s) \cdot \omega$  or  $Q_{\omega}(s,a) = \phi(s,a) \cdot \omega$ . Similarly,  $\phi(s)$  is a one-hot encoding.
Optimizer. The policy and value loss are optimized by stochastic gradient descent (SGD).
10.2.1 Learning Parameters
We set discount factor  $\gamma = 0.999$
MVO: policy learning rate is  $1\mathrm{e} - 5 \in \{5\mathrm{e} - 5, 1\mathrm{e} - 5, 5\mathrm{e} - 6\}$ , value function learning rate is 100 times policy learning rate.  $\lambda = 1.0 \in \{0.6, 0.8, 1.0, 1.2\}$ . Sample size  $n = 50$ . Maximum inner update number  $M = 10$ . IS ratio range  $\delta = 0.5$ . Inner termination ratio  $\beta = 0.6$ .
Tamar: policy learning rate is  $1\mathrm{e} - 5 \in \{5\mathrm{e} - 5, 1\mathrm{e} - 5, 5\mathrm{e} - 6\}$ ,  $J, V$  learning rate is 100 times policy learning rate. Threshold  $b = 50 \in \{10, 50, 100\}$ ,  $\lambda = 0.1 \in \{0.1, 0.2, 0.4\}$ .
MVP: policy learning rate is  $1\mathrm{e} - 5 \in \{5\mathrm{e} - 5, 1\mathrm{e} - 5, 5\mathrm{e} - 6\}$ ,  $y$  learning rate is the same.  $\lambda = 0.1 \in \{0.1, 0.2, 0.4\}$ .
MG: policy learning rate is  $1\mathrm{e} - 4 \in \{5\mathrm{e} - 4, 1\mathrm{e} - 4, 5\mathrm{e} - 5\}$ , value function learning rate is 100 times policy learning rate.  $\lambda = 1.2 \in \{0.8, 1.0, 1.2\}$ . Sample size  $n = 50$ . Maximum inner update number  $M = 10$ . IS ratio range  $\delta = 0.5$ . Inner termination ratio  $\beta = 0.6$ .
MVPI: Q function learning rate 5e-3∈{5e-3, 1e-3, 5e-4},  $\lambda = 0.2\in \{0.2,0.4,0.6\}$ .
10.2.2 Analysis for MVPI in Maze (MVPI-Q-Learning)
MVPI-Q-Learning finds the optimal risk-averse path when goal reward is 20 but fails when goal reward is 40. Since it is not intuitive to report the learning curve for a policy iteration method where its reward is modified in each iteration, we give an analysis here.
The value of dual variable  $y$  in Equation 5 is  $(1 - \gamma)\mathbb{E}[G_0]$  given the current policy. Recall that the maximum episode length is 100. At the beginning, when the Q function is randomly initialized (i.e., it is a random policy),  $\mathbb{E}[G_0] = \sum_{t=0}^{99} 0.999^t (-1) \approx -95.2$ . Thus  $y = (1 - 0.999) \times (-95.2) = -0.0952$ , the goal reward after modification is  $r_{\mathrm{goal}} = 20 - 0.2 \times 20^2 + 2 \times 0.2 \times 20 \times y \approx -60.7$ . For the red state, its original reward is sampled from  $\{-15, -1, 13\}$ . After the reward modification, it becomes sampling from  $\{-59.4, -1.16, -21.2\}$ . Thus the expected reward of the red state is now  $r_{\mathrm{red}} = 0.4 \times (-59.4) + 0.2 \times (-1.16) + 0.4 \times (-21.2) = -32.472$ . Given the maximum episode length is 100, the optimal policy is still the white path in Figure 1. (Because the expected return for the white path is  $\sum_{t=0}^{9} 0.999^t (-1) + 0.999^{10} (-60.7) \approx -70$ . The expected return for a random walk is  $\sum_{t=0}^{99} 0.999^t (-1) \approx -95.2$ . The expected return for the shortest path going through the red state is even lower than the white path since the reward of the red state after modification is pretty negative: -32.472.)
However, when goal reward is 40, after modification, the goal reward becomes  $r_{\mathrm{goal}} = 40 - 0.2 \times 40^2 + 2 \times 0.2 \times 40 \times y \approx -281.5$ . In this case, the optimal policy has to avoid the goal state since it leads to a even lower return.
10.2.3 Return Variance and Gini Deviation in Maze
We report the return's variance and GD during learning for different methods, as shown in Figure 7 and 8. Tamar [8] is unable to reach the goal in both settings. MVO fails to reach the goal when the return magnitude increases. MVP [12]'s optimal risk-aversion rate is much lower than MG. MG can learn a risk averse policy in both settings with lower variance and GD, which suggests it is less sensitive to the return numerical scale.
Figure 7: Expected return, return variance and Gini deviation of different methods in Maze when goal reward is 20. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
Figure 8: Expected return, return variance and Gini deviation of different methods in Maze when goal reward is 40. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
10.2.4 Sensitivity to Trade-off Parameter
We report the learning curves of total return based methods with different  $\lambda$  when goal reward is 20 in Figure 9. The learning parameters are the same as shown in Section 10.2.1.
10.3 LunarLander Discrete