SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The agent's goal is to land the lander on the ground without crashing. The state dimension is 8. The action dimension is 4. The detailed reward information is available at this webpage  ${}^{3}$  . We divide the whole ground into left and right parts by the middle line of the landing pad as shown in Figure 10. If the agent lands in the right part, an additional noisy reward signal sampled from  $\mathcal{N}\left( {0,1}\right)$  tims 90 is given. We set the maximum episode length to 1000 . Note that the original reward for successfully landing is 100 , thus the numerical scale of both return and reward is relatively large in this domain.
Policy function. The policy is a categorical distribution in REINFORCE, MVO, Tamar, MVP and MG, modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Softmax function is applied to the output to generate categorical probabilities.
Value function. The value function in REINFORCE, MVO, MG, and  $Q$  function in MVPI-DQN is a neural network with two hidden layers. The hidden size is 128. Activation is ReLU.
Optimizer. The optimizer for policy and value functions is Adam.
10.3.1 Learning Parameters
Discount factor is  $\gamma = 0.999$
REINFORCE (with baseline): the policy learning rate is  $7\mathrm{e} - 4 \in \{7\mathrm{e} - 4, 3\mathrm{e} - 4, 7\mathrm{e} - 5\}$ , value function learning rate is 10 times policy learning rate.
MVO: policy learning rate is  $7\mathrm{e} - 5 \in \{7\mathrm{e} - 4,3\mathrm{e} - 4,7\mathrm{e} - 5\}$ , value function learning rate is 10 times policy learning rate.  $\lambda = 0.4 \in \{0.4,0.6,0.8\}$ . Sample size  $n = 30$ . Maximum inner update number  $M = 10$ . IS ratio range  $\delta = 0.5$ . Inner termination ratio  $\beta = 0.6$
Tamar: policy learning rate is  $7\mathrm{e} - 5 \in \{7\mathrm{e} - 4, 3\mathrm{e} - 4, 7\mathrm{e} - 5\}$ .  $J, V$  learning rate is 100 times the policy learning rate. Threshold  $b = 50 \in \{10, 50, 100\}$ .  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MVP: policy learning rate is  $7\mathrm{e} - 5 \in \{7\mathrm{e} - 4, 3\mathrm{e} - 4, 7\mathrm{e} - 5\}$ .  $y$  learning rate is the same.  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MG: policy learning rate is  $7\mathrm{e} - 4 \in \{7\mathrm{e} - 4,3\mathrm{e} - 4,7\mathrm{e} - 5\}$ , value function learning rate is 10 times policy learning rate.  $\lambda = 0.6 \in \{0.4,0.6,0.8\}$ . Sample size  $n = 30$ . Maximum inner update number  $M = 10$ . IS ratio range  $\delta = 0.5$ . Inner termination ratio  $\beta = 0.6$
MVPI: Q function learning rate is  $7\mathrm{e} - 4 \in \{7\mathrm{e} - 4, 3\mathrm{e} - 4, 7\mathrm{e} - 5\}$ ,  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ . Batch size is 64.
10.3.2 Return Variance and Gini Deviation in LunarLander
The return's variance and GD of different methods during training is shown in Figure 11. All the risk-averse methods, apart from ours, fail to learn a reasonable policy in this domain. Our method achieves a comparable return, but with lower variance and GD compared with risk-neutral method.
<sup>3</sup>https://www.gymlibrary.dev/environments/box2d/lunar_lander/
Figure 9: Policy evaluation return and optimal risk-aversion rate v.s. training episodes in Maze (goal reward is 20) for MVO, Tamar, MVP, and MG with different  $\lambda$ . The reasonable  $\lambda$  range varies in different methods. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
Figure 10: Divide the ground of LunarLander into left and right parts by the middle (red) line. If landing in the right area, an additional reward sampled from  $\mathcal{N}(0,1)$  times 90 is given.
Figure 11: Return variance and Gini deviation of different methods in LunarLander. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
10.4 InvertedPendulum
(The description of the Mujoco environments can be found at this webpage 4.)
The agent's goal is to balance a pole on a cart. The state dimension is 4 (X-position is already contained). The action dimension is 1. At each step, the environment provides a reward of 1. If the agent reaches the region X-coordinate  $>0.01$ , an additional noisy reward signal sampled from  $\mathcal{N}(0,1)$  times 10 is given. To avoid the initial random speed forcing the agent to the X-coordinate  $>0.01$  region, we decrease the initial randomness for the speed from  $U(-0.01,0.01)$  to  $U(-0.0005,0.0005)$ , where  $U()$  represents the uniform distribution. The game ends if angle between the pole and the cart is greater than 0.2 radian or a maximum episode length 500 is reached.
Policy function. The policy is a normal distribution in REINFORCE, and VPG based methods (MVO, Tamar, MVP, and MG), modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Tanh is applied to the output to scale it to  $(-1, 1)$ . The output times the maximum absolute value of the action serves as the mean of the normal distribution. The logarithm of standard deviation is an independent trainable parameter.
The policy is a deterministic function in TD3 and MVPI, modeled as a neural network with two hidden layers. The hidden size is 128. Activation is ReLU. Tanh is applied to the output to scale it to  $(-1,1)$ . The output times the maximum absolute value of the action is the true action executed in the environment.
Value function. The value function in REINFORCE, VPG based MVO, VPG based MG, TD3, and MVPI is a neural network with two hidden layers. The hidden size is 128. Activation is ReLU.
Optimizer. Optimizer for both policy and value function is Adam.
10.4.1 Learning Parameters
Discount factor  $\gamma = 0.999$
REINFORCE (with baseline): policy learning rate is  $1\mathrm{e} - 4 \in \{1\mathrm{e} - 4,5\mathrm{e} - 5,5\mathrm{e} - 4\}$ , value function learning rate is 10 times policy learning rate.
<sup>4</sup>https://www.gymlibrary.dev/environments/mujoco/
MVO: policy learning rate is  $1\mathrm{e} - 5 \in \{1\mathrm{e} - 4, 5\mathrm{e} - 5, 1\mathrm{e} - 5\}$ , value function learning rate is 10 times policy learning rate.  $\lambda = 0.6 \in \{0.2, 0.4, 0.6\}$ . Sample size  $n = 30$ . Maximum inner update number  $M = 10$ . IS ratio range  $\delta = 0.5$ . Inner termination ratio  $\beta = 0.6$ .
Tamar: policy learning rate is  $1\mathrm{e} - 5 \in \{1\mathrm{e} - 4, 5\mathrm{e} - 5, 1\mathrm{e} - 5\}$ .  $J, V$  learning rate is 100 times policy learning rate. Threshold  $b = 50 \in \{10, 50, 100\}$ .  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MVP: policy learning rate is  $1\mathrm{e} - 5 \in \{1\mathrm{e} - 4, 5\mathrm{e} - 4, 1\mathrm{e} - 5\}$ .  $y$  learning rate is the same.  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MG: policy learning rate is 1e-4 ∈ {1e-4, 5e-5, 1e-4}, value function learning rate is 10 times policy learning rate. λ = 1.0 ∈ {0.6, 1.0, 1.4}. Sample size n = 30. Maximum inner update number M = 10. IS ratio range δ = 0.5. Inner termination ratio β = 0.6.
MVPI: Policy and value function learning rate  $3\mathrm{e} - 4 \in \{3\mathrm{e} - 4, 7\mathrm{e} - 5, 1\mathrm{e} - 5\}$ ,  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ . Batch size is 256.
TD3: Policy and value function learning rate  $3\mathrm{e} - 4 \in \{3\mathrm{e} - 4,7\mathrm{e} - 5,1\mathrm{e} - 5\}$ . Batch size is 256.
10.4.2 Return Variance and Gini Deviation in InvertedPendulum
The return variance and GD of different methods are shown in Figure 12 and 13.
Figure 12: Return variance and Gini deviation of policy gradient methods in InvertedPendulum. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
Figure 13: Return variance and Gini deviation of TD3 and MVPI in InvertedPendulum. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
10.5 HalfCheetah
The agent controls a robot with two legs. The state dimension is 18 (add X-position). The action dimension is 6. The reward is determined by the speed between the current and the previous time step and a penalty over the magnitude of the input action (Originally, only speed toward right is positive, we make the speed positive in both direction so that agent is free to move left or right). If the agent reaches the region X-coordinate  $< -3$ , an additional noisy reward signal sampled from  $\mathcal{N}(0,1)$  times 10 is given. The game ends when a maximum episode length 500 is reached.
Policy function. The policy is a normal distribution in PPO, and PPO based methods (MVO, Tamar, MVP, and MG). The architecture is the same as in InvertedPendulum. Hidden size is 256.
The policy of TD3 and MVPI is the same as in InvertedPendulum. Hidden size is 256.
Value function. The value function in PPO, PPO based methods (MVO, Tamar, MVP, and MG), TD3 and MVPI is a neural network with two hidden layers. The hidden size is 256. Activation is ReLU.
Optimizer. Optimizer for policy and value is Adam.
10.5.1 Learning Parameters
Discount factor is  $\gamma = 0.99$
Common parameters of PPO and PPO based methods. GAE parameter: 0.95, Entropy coef: 0.01, Critic coef: 0.5, Clip  $\epsilon$ : 0.2, Grad norm: 0.5.
PPO. policy learning rate  $3\mathrm{e} - 4 \in \{3\mathrm{e} - 4, 7\mathrm{e} - 5, 1\mathrm{e} - 5\}$ , value function learning rate is the same. Inner update number  $M = 5$ .
MVO. policy learning rate  $7\mathrm{e} - 5 \in \{3\mathrm{e} - 4, 7\mathrm{e} - 5, 1\mathrm{e} - 5\}$ , value function learning rate is the same. Sample size  $n = 10$ . Inner update number 5.
Tamar. policy learning rate  $7\mathrm{e} - 5 \in \{3\mathrm{e} - 4, 7\mathrm{e} - 5, 1\mathrm{e} - 5\}$ , value function learning rate is the same.  $J, V$  learning rate is 100 times policy learning rate. Threshold  $b = 50 \in \{10, 50, 100\}$ .  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MVP. policy learning rate  $7\mathrm{e} - 5 \in \{3\mathrm{e} - 4, 7\mathrm{e} - 5, 1\mathrm{e} - 5\}$ , value function and  $y$  learning rate is the same.  $\lambda = 0.2 \in \{0.2, 0.4, 0.6\}$ .
MG. policy learning rate  $3\mathrm{e - }4\in \{3\mathrm{e - }4,7\mathrm{e - }5,1\mathrm{e - }5\}$  , value function learning rate is the same. Sample size  $n = 10$  . Inner update number  $M = 5$
TD3. policy learning rate  $3\mathrm{e} - 4 \in \{3\mathrm{e} - 4,7\mathrm{e} - 5,1\mathrm{e} - 5\}$ , value function learning rate is the same. Batch size is 256.
MVPI. policy learning rate  $3\mathrm{e} - 4 \in \{3\mathrm{e} - 4,7\mathrm{e} - 5,1\mathrm{e} - 5\}$ , value function learning rate is the same.  $\lambda = 0.2 \in \{0.2,0.4,0.6\}$ . Batch size is 256.
10.5.2 Return Variance and Gini Deviation in HalfCheetah
See Figures 14 and 15.
Figure 14: Return variance and Gini deviation of on-policy methods in HalfCheetah. Curves are averaged over 10 seeds with shaded regions indicating standard errors
Figure 15: Return variance and Gini deviation of off-policy methods in HalfCheetah. Curves are averaged over 10 seeds with shaded regions indicating standard errors
10.6 Swimmer
The agent controls a robot with two rotors (connecting three segments) and learns how to move. The state dimension is 10 (add XY-positions). The action dimension is 2. The reward is determined by the speed between the current and the previous time step and a penalty over the magnitude of the input action (Originally, only speed toward right is positive, we make the speed positive in both direction so that agent is free to move left or right). If agent reaches the region X-coordinate  $>0.5$ , an additional noisy reward signal sampled from  $\mathcal{N}(0,1)$  times 10 is given. The game ends when a maximum episode length 500 is reached.
The neural network architectures and learning parameters are the same as in HalfCheetah.
10.6.1 Return Variance and Gini Deviation in Swimmer
See Figures 16 and 17.
Figure 16: Return variance and Gini deviation of on-policy methods in Swimmer. Curves are averaged over 10 seeds with shaded regions indicating standard errors
Figure 17: Return variance and Gini deviation of off-policy methods in Swimmer. Curves are averaged over 10 seeds with shaded regions indicating standard errors
11 Additional Results
We design two other scenarios in HalfCheetah and Swimmer (marked as HalfCheetah1 and Swimmer1 in the figure's caption). Here the randomness of the noisy reward linearly decreases over the forward distance (right direction) the agent has covered. To encourage the agent to move forward, only the forward speed is positive. The additional noisy reward is sampled from  $\mathcal{N}(0,1)$  times 10 times  $1 - \frac{X}{20}$  if  $X$ -position  $>0$ . To maximize the expected return and minimize the risk, the agent should move forward as far as possible.
The learning parameters are the same as Section 10.5.
The distance agents covered in these two environments are shown in Figures 18 19.
Figure 18: The distance agents covered in HalfCheetah1. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
Figure 19: The distance agents covered in Swimmer1. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
12 Additional Related Work and Discussion
12.1 Coherent measure of variability and risk
This paper focuses on the measure of variability, i.e., the dispersion of a random variable. Both variance and Gini deviation are measures of variability as discussed in Sections 2 and 3. Gini deviation is further known as a coherent measure of variability.
Another widely adopted risk measure is conditional value at risk (CVaR), which is a coherent risk measure [48]. Coherence is usually important in financial domains. Consider a continuous random variable  $Z$  with the cumulative distribution function  $F(z) = P(Z \leq z)$ . The value at risk (VaR) or quantile at confidence level  $\alpha \in (0,1]$  is defined as  $\mathrm{VaR}_{\alpha}(Z) = \min \{z|F(z) \geq \alpha\}$ . Then the CVaR at confidence level  $\alpha$  is defined as
$$
\mathrm {C V a R} _ {\alpha} (Z) = \frac {1}{\alpha} \int_ {0} ^ {\alpha} \mathrm {V a R} _ {\beta} (Z) d \beta \tag {25}
$$
Due to the nature of CVaR of only considering the tailed quantiles, it does not capture the variability of  $Z$  beyond the quantile level  $\alpha$ . Thus it is not considered as a measure of variability. In [25], measures of variability and measures of risk are clearly distinguished and their coherent properties are different. We summarize their properties here [25].
Consider a measure  $\rho :\mathcal{M}\to (-\infty ,\infty ]$
- (A) Law-invariance: if  $X, Y \in \mathcal{M}$  have the same distributions, then  $\rho(X) = \rho(Y)$ .
(A1) Positive homogeneity:  $\rho (\lambda X) = \lambda \rho (X)$  for all  $\lambda >0,X\in \mathcal{M}$
(A2) Sub-additivity:  $\rho (X + Y)\leq \rho (X) + \rho (Y)$  for  $X,Y\in \mathcal{M}$
(B1) Monotonicity:  $\rho(X) \leq \rho(Y)$  when  $X \leq Y, X, Y \in \mathcal{M}$
(B2) Translation invariance:  $\rho(X - m) = \rho(X) - m$  for all  $m \in \mathbb{R}$ ,  $X \in \mathcal{M}$
(C1) Standardization:  $\rho (m) = 0$  for all  $m\in \mathbb{R}$
(C2) Location invariance:  $\rho(X - m) = \rho(X)$  for all  $m \in \mathbb{R}$ ,  $X \in \mathcal{M}$
**Coherent measure of variability:** A measure of variability is coherent if it satisfies properties (A), (A1), (A2), (C1) and (C2). An example is Gini deviation.
**Coherent risk measure:** A risk measure is coherent if it satisfies properties (A1), (A2), (B1) and (B2). An example is CVaR.
12.2 CVaR optimization in RL
CVaR focuses on the extreme values of a random variable and is usually adopted in RL domains to avoid catastrophic outcomes. Policy gradient is an important method to optimize CVaR, e.g., see [33][49][7]. Since CVaR focuses on the worst returns, policy gradient techniques often ignore high-return trajectories [33, 50]. [7] proposed cross entropy soft risk optimization to improve sample efficiency by optimizing with respect to all trajectories while maintaining risk aversion. A time difference method is proposed in [51]. CVaR is also used for action selection in distributional RL literature, e.g., see [35][52][53].