SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {J} _ {\alpha} ^ {\theta} (R) = \mathbb {C V a R} _ {R \sim P ^ {\theta}} ^ {\alpha} [ R ] = \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} x \cdot P ^ {\theta} (x) \cdot d x
$$
To calculate the policy gradient of  $\tilde{J}_{\alpha}^{\theta}(R)$ , we begin with the conservation of probability mass below the quantile  $q_{\alpha}^{\theta}$ :
$$
\int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} P ^ {\theta} (x) d x \equiv \alpha .
$$
Then, using the Leibniz integral rule, we have
$$
0 = \nabla_ {\theta} \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} P ^ {\theta} (x) d x = \left[ \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} \nabla_ {\theta} P ^ {\theta} (x) d x \right] + \left[ P ^ {\theta} \left(q _ {\alpha} ^ {\theta} (R)\right) \cdot \nabla_ {\theta} q _ {\alpha} ^ {\theta} (R) \right]. \tag {10}
$$
Notice that as a particular consequence of the conservation rule Eq. (10), positive gradients of  $P^{\theta}(x)$  cause the quantile  $q_{\alpha}^{\theta}(R)$  to decrease. This phenomenon makes the CVaR policy gradient sensitive to the selection of baseline, as visualized in Fig. 6b. In fact, the quantile  $q_{\alpha}^{\theta}(R)$  itself is the only baseline that permits unbiased gradients:
$$
\begin{array}{l} \nabla_ {\theta} \tilde {J} _ {\alpha} ^ {\theta} (R) = \nabla_ {\theta} \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} x \cdot P ^ {\theta} (x) \cdot d x = \\ \left[ \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} x \cdot \nabla_ {\theta} P ^ {\theta} (x) d x \right] + \left[ q _ {\alpha} ^ {\theta} (R) \cdot P ^ {\theta} \left(q _ {\alpha} ^ {\theta} (R)\right) \cdot \nabla_ {\theta} q _ {\alpha} ^ {\theta} (R) \right] \stackrel {{E q. (1 0)}} {=} \tag {11} \\ \left[ \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} x \cdot \nabla_ {\theta} P ^ {\theta} (x) d x \right] - \left[ q _ {\alpha} ^ {\theta} (R) \cdot \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} \nabla_ {\theta} P ^ {\theta} (x) d x \right] = \\ \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} \left(x - q _ {\alpha} ^ {\theta} (R)\right) \cdot \nabla_ {\theta} P ^ {\theta} (x) d x, \\ \end{array}
$$
which gives us Eq. (4).
A.2 PG for CVaR in Meta RL
We now turn to PG for CVaR optimization in MRL. We rely on the definitions and notations of Section 3, as well as Definition 1 and Assumption 1 in Section 4. Notice that  $J_{\alpha}^{\theta}(R)$  of Eq. (2) can be
(a) Mean PG
#
(b) CVaR PG, Eq. (4)
#
Figure 6: Illustration of the policy gradient estimation in standard RL. (a) In Mean PG, the expected gradient is independent of the baseline: even if most of the distribution  $P^{\pi_{\theta}}$  seems to be "pushed upwards", its normalization to a total probability of 1 forces the probability of low returns to decrease for that of high returns will increase. (b) In CVaR PG, due to the effect of Eq. (10), any baseline except for  $q_{\alpha}^{\theta}(R)$  leads to biased gradients.
written in integral form as
$$
J _ {\alpha} ^ {\theta} (R) = \mathbb {C V a R} _ {\tau \sim D} ^ {\alpha} \left[ \mathbb {E} _ {R \sim P _ {\tau} ^ {\theta}} [ R ] \right] = \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \cdot P _ {z} ^ {\theta} (x) \cdot d x \cdot d z.
$$
In Eq. (11) above, the boundary of the integral over the  $\alpha$ -tail is simply the scalar  $q_{\alpha}^{\theta}$ . In MRL, this is replaced by the boundary of the set  $\Omega_{\alpha}^{\theta}$ , defined in a general topological space. Thus, we begin by characterizing this boundary.
Lemma 1 (The boundary of  $\Omega_{\alpha}^{\theta}$ ). Under Assumption 1,  $\forall z \in \partial \Omega_{\alpha}^{\theta}: \int_{-\infty}^{\infty} x \cdot P_{z}^{\theta}(x) \cdot dx = q_{\alpha}(V_{\tau}^{\theta})$ .
Proof. Since  $v(z) = V_z^\theta$  is a continuous function between topological spaces, by denoting  $B = (-\infty, q_\alpha(V_\tau^\theta)]$  we have
$$
\partial \Omega_ {\alpha} ^ {\theta} = \partial v ^ {- 1} (B) \stackrel {{\text {c o n t i n u o u s}} v} {{\subseteq}} v ^ {- 1} (\partial B) = v ^ {- 1} (\{q _ {\alpha} (V _ {\tau} ^ {\theta}) \}) = \{z \in \Omega | V _ {z} ^ {\theta} = q _ {\alpha} (V _ {\tau} ^ {\theta}) \},
$$
hence  $\forall z\in \partial \Omega_{\alpha}^{\theta}:V_{z}^{\theta} = q_{\alpha}(V_{\tau}^{\theta})$  . Notice that  $V_{z}^{\theta} = \int_{-\infty}^{\infty}xP_{z}^{\theta}(x)\cdot dx.$
Finally, we can prove Theorem 1.
Proof of Theorem 1. First, we consider the conservation of probability:
$$
\int_ {\Omega_ {\alpha} ^ {\theta}} D (z) d z \equiv \alpha
$$
The gradient of this integral can be calculated using a high-dimensional generalization of the Leibniz integral rule, named Reynolds Transport Theorem (RTT, Tromba and Marsden [1996]):
$$
0 \stackrel {\text {d e r i v a t i v e o f}} {=} \nabla_ {\theta} \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) d z \stackrel {\text {R T T}} {=} \int_ {\Omega_ {\alpha} ^ {\theta}} \nabla_ {\theta} D (z) d z + \int_ {\partial \Omega_ {\alpha} ^ {\theta}} D (z) \cdot \left(v _ {b} \cdot n\right) \cdot d A \tag {12}
$$
$$
\widetilde {\nabla_ {\theta} D (z) \equiv 0} \int_ {\partial \Omega_ {\alpha} ^ {\theta}} D (z) \cdot (v _ {b} \cdot n) \cdot d A,
$$
where  $n(z,\theta)$  is the outward-pointing unit vector that is normal to the surface  $\partial \Omega_{\alpha}^{\theta}$ , and  $v_{b}(z,\theta)$  is the velocity of the area element on the surface with area  $dA$ . Notice that  $\nabla_{\theta}D(z)\equiv 0$  in the last
#
Figure 7: Illustration of the meta policy gradient estimation. In Mean meta-PG (left, Eq. (3)), the PG is estimated for all tasks. In CVaR meta-PG (center, Eq. (6)), it is only calculated over the low-return tasks. Since the returns distribution is decoupled from the tasks distribution, shifting the baseline (right) does not insert bias to the PG, in accordance with Theorem 1.
equality expresses a substantial difference from the standard RL settings in Eq. (10): there, we have  $\nabla_{\theta}P^{\theta}(x)$ , which does not necessarily vanish.
Next, we turn to the meta policy gradient itself, again using Reynolds Transport Theorem:
$$
\begin{array}{l} \nabla_ {\theta} J _ {\alpha} ^ {\theta} (R) = \nabla_ {\theta} \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \stackrel {\mathrm {R T T}} {=} \\ \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] + \left[ \int_ {\partial \Omega_ {\alpha} ^ {\theta}} D (z) \left(\int_ {- \infty} ^ {\infty} x P _ {z} ^ {\theta} (x) \cdot d x\right) \cdot (v _ {b} \cdot n) \cdot d A \right] \stackrel {{L e m m a 1}} {{=}} \\ \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] + \left[ q _ {\alpha} \left(V _ {\tau} ^ {\theta}\right) \int_ {\partial \Omega_ {\alpha} ^ {\theta}} D (z) \cdot \left(v _ {b} \cdot n\right) \cdot d A \right] \stackrel {{E q. (1 2)}} {{=}} \\ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z. \tag {13} \\ \end{array}
$$
Finally, we show that any  $\theta$ -independent additive baseline (highlighted in the equation) does not change the gradient calculation:
$$
\begin{array}{l} \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} (x - b) \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z = \\ \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] - \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} b \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] = \\ \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] - \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \cdot b \cdot \nabla_ {\theta} \left(\int_ {- \infty} ^ {\infty} P _ {z} ^ {\theta} (x) \cdot d x\right) \cdot d z \right] \overset {\int_ {- \infty} ^ {\infty} P _ {z} ^ {\theta} (x) d x \equiv 1} {\underset {=} {=}} \\ \left[ \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} x \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z \right] - 0 = \nabla_ {\theta} J _ {\alpha} ^ {\theta} (R), \tag {14} \\ \end{array}
$$
which completes the proof. Notice that we used the identity  $\nabla_{\theta}\int_{-\infty}^{\infty}P_{z}^{\theta}(x)dx = \nabla_{\theta}1 = 0$ ; this does not hold for the analogous term in the standard RL settings in Eq. (10),  $\nabla_{\theta}\int_{-\infty}^{q_{\alpha}^{\theta}(x)}P^{\theta}(x)dx$ , whose gradient depends on  $\nabla_{\theta}q_{\alpha}^{\theta}(x)$  according to the Leibniz integral rule.
Proof. Recall that by Eq. (8),  $G = \frac{1}{\alpha N} \sum_{i=1}^{N} \mathbf{1}_{R_i \leq \hat{q}_\alpha^\theta} \sum_{m=1}^{M} g_{i,m}$ . Denoting  $G_i = \sum_{m=1}^{M} g_{i,m}$  and substituting  $\hat{q}_\alpha^\theta = q_\alpha^\theta$ , we have
$$
G = \frac {1}{N} \sum_ {i = 1} ^ {N} \alpha^ {- 1} \mathbf {1} _ {R _ {i} \leq q _ {\alpha} ^ {\theta}} G _ {i}.
$$
Expectation: Since  $\{G_{i}\}$  are i.i.d, and using the law of total probability, we obtain
$$
\begin{array}{l} \mathbb {E} _ {D} \left[ \alpha^ {- 1} \mathbf {1} _ {R _ {i} \leq q _ {\alpha} ^ {\theta}} G _ {i} \right] = \alpha \cdot \left(\alpha^ {- 1} \cdot 1 \cdot \mathbb {E} _ {D} \left[ G _ {1} \mid R _ {1} \leq q _ {\alpha} ^ {\theta} \right]\right) + (1 - \alpha) \cdot \left(\alpha^ {- 1} \cdot 0 \cdot \mathbb {E} _ {D} \left[ G _ {1} \mid R _ {1} > q _ {\alpha} ^ {\theta} \right]\right) \\ = \mathbb {E} _ {D} \left[ G _ {1} \mid R _ {1} \leq q _ {\alpha} ^ {\theta} \right]. \\ \end{array}
$$
By switching the task sample distribution to  $D_{\alpha}^{\theta}$ , and using the definition of  $D_{\alpha}^{\theta}$ , we simply have
$$
\mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ \alpha^ {- 1} \mathbf {1} _ {R _ {i} \leq q _ {\alpha} ^ {\theta}} G _ {i} \right] = \alpha^ {- 1} \mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ G _ {1} \right] = \alpha^ {- 1} \mathbb {E} _ {D} \left[ G _ {1} \mid R _ {1} \leq q _ {\alpha} ^ {\theta} \right].
$$
Together, we obtain  $\mathbb{E}_{D_{\alpha}^{\theta}}[\alpha G] = \mathbb{E}_D[G]$  as required.
Variance: For the original distribution, since  $\{G_i\}$  are i.i.d, we have
$$
\begin{array}{l} N \cdot \operatorname {V a r} _ {D} (G) = \operatorname {V a r} _ {D} \left(\alpha^ {- 1} \mathbf {1} _ {R _ {1} \leq q _ {\alpha} ^ {\theta}} G _ {1}\right) \\ = \mathbb {E} _ {D} \left[ \alpha^ {- 2} \mathbf {1} _ {R _ {1} \leq q _ {\alpha} ^ {\theta}} G _ {1} ^ {2} \right] - \mathbb {E} _ {D} \left[ \alpha^ {- 1} \mathbf {1} _ {R _ {1} \leq q _ {\alpha} ^ {\theta}} G _ {1} \right] ^ {2} \\ = \mathbb {E} _ {D} [ \alpha \alpha^ {- 2} G _ {1} ^ {2} | R _ {1} \leq q _ {\alpha} ^ {\theta} ] - \mathbb {E} _ {D} [ \alpha \alpha^ {- 1} G _ {1} | R _ {1} \leq q _ {\alpha} ^ {\theta} ] ^ {2} \\ = \alpha^ {- 1} \mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ G _ {1} ^ {2} \right] - \mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ G _ {1} \right] ^ {2} \\ \geq \alpha^ {- 1} \left(\mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ G _ {1} ^ {2} \right] - \mathbb {E} _ {D _ {\alpha} ^ {\theta}} \left[ G _ {1} \right] ^ {2}\right) \\ = \alpha^ {- 1} \operatorname {V a r} _ {D _ {\alpha} ^ {\theta}} (G _ {1}). \\ \end{array}
$$
For the tail distribution  $D_{\alpha}^{\theta}$ , however,
$$
N \cdot \operatorname {V a r} _ {D _ {\alpha} ^ {\theta}} (\alpha G) = \alpha^ {2} \operatorname {V a r} _ {D _ {\alpha} ^ {\theta}} (\alpha^ {- 1} \mathbf {1} _ {R _ {1} \leq q _ {\alpha} ^ {\theta}} G _ {1}) = \operatorname {V a r} _ {D _ {\alpha} ^ {\theta}} (G _ {1}),
$$
C The Cross Entropy Method
C.1 Background
The Cross Entropy Method (CEM, de Boer et al. [2005]) is a general approach to rare-event sampling and optimization. In this work, we use its sampling version to sample high-risk tasks from the tail of  $D$ . As described in Section 5, the CEM repeatedly samples from the parameterized distribution  $D_{\phi}$ , and updates  $\phi$  according to the  $\beta$ -tail of the sampled batch. Since every iteration focuses on the tail of its former, we intuitively expect exponential convergence to the tail of the original distribution. While theoretical convergence analysis does not guarantee the exponential rate [de Mello and Rubinstein, 2003], practically, the CEM often converges within several iterations. For clarity, we provide the pseudo-code for the basic CEM in Algorithm 3. In this version, the CEM repeatedly generates samples from the tail of the given distribution  $D_{\phi_0}$ .
Algorithm 3: The Cross Entropy Method (CEM)
1 Input: distribution  $D_{\phi_0}$ ; score function  $R$ ; target level  $q$ ; batch size  $N$ ; CEM quantile  $\beta$ .
2  $\phi \gets \phi_0$
3 while true do // Sample
4 Sample  $z\sim D_{\phi}^{N}$
5  $w_{i}\gets D_{\phi_{0}}(z_{i}) / D_{\phi}(z_{i})\quad (1\leq i\leq N)$
6 Print  $z$
7  $q^{\prime}\gets \max \left(q,q_{\beta}\left(\{R(z_{i})\}_{i = 1}^{N}\right)\right)$
8  $\phi \gets \operatorname{argmax}_{\phi'} \sum_{i=1}^{N} w_i \mathbf{1}_{R(z_i) \leq q'} \log D_{\phi'}(z_i)$
C.2 Discussion
The CEM is the key to the flexible robustness level of RoML (Algorithm 2): it can learn to sample not only the single worst-case task, but all the  $\alpha$  tasks with the lowest returns.
The CEM searches for a task distribution within a parametric family of distributions. This approach can handle infinite task spaces, and learn the difficulty of tasks in the entire task space from a mere finite sample of tasks. For example, assume that the tasks correspond to environment parameters that take continuous values within some bounded box (as in Section 6.2 and Section 6.3). The CEM can fit a distribution over an entire subset of the box – from a mere finite batch of tasks. This property lets the CEM learn the high-risk tasks quickly and accelerates the meta-training, as demonstrated in Section 6 and Appendix D.3.
On the other hand, this approach relies on the structure in the task space. If the tasks do not have a natural structure like the ones in the bounded box, it is not trivial to define the parametric family of distributions. This is the case in certain supervised meta learning problems. For example, in the common meta-classification problem [Finn et al., 2017], the task space consists of subsets of classes to classify. This is a discrete space without a trivial metric between tasks. Hence, it is difficult for the CEM to learn the  $\alpha$  lowest-return tasks from a finite sample. Thus, while RoML is applicable to supervised meta learning as well as MRL, certain task spaces require further adjustment, such as a meaningful embedding of the task space. This challenge is left for future work.
D Experiments: Detailed Settings and Results
D.1 Khazad Dum
At the end of the hall the floor vanished and fell to an unknown depth. The outer door could only be reached by a slender bridge of stone, without kerb or rail, that spanned the chasm with one curving spring of fifty feet. It was an ancient defence of the Dwarves against any enemy that might capture the First Hall and the outer passages. They could only pass across it in single file. [Tolkien, 1954]
The detailed settings of the Khazad Dum environment presented in Section 6.1 are as follows. Every task is carried for  $K = 4$  episodes of  $T = 32$  times steps. The return corresponds to the undiscounted sum of the rewards  $(\gamma = 1)$ . Every time step has a cost of  $1 / T$  points if the L1-distance of the agent from the target is larger than 5; and for distances between 0 and 5, the cost varies linearly between 0 and  $1 / T$ . By reaching the destination, the agent obtains a reward of  $5 / T$ , and has no more costs for the rest of the episode. By falling to the abyss, the agent can no longer reach the goal and is bound to suffer a cost of  $1 / T$  for every step until the end of the episode. Every step, the agent observes its location (represented using a soft one-hot encoding, similarly to Greenberg et al. [2022]) and chooses whether to move left, right, up or down. If the agent attempts to move into a wall, it remains in place.
The tasks are characterized by the rain intensity, distributed  $\tau \sim Exp(0.1)$ . The rain only affects the episode when the agent crosses the bridge: then, the agent suffers from an additive normally-distributed action noise  $\mathcal{N}(0,\tau^2)$ , in addition to a direct damage translated into a cost of  $3\cdot \tau$ . For RoML, the CEM is implemented over the exponential family of distributions  $Exp(\phi)$  with  $\phi_0 = 0.1$  and  $\beta = 0.05$ . In this toy benchmark we use no regularization ( $\nu = 0$ ).
In addition to the test returns throughout meta-training shown in Fig. 3, Fig. 8 displays the final test returns at the end of the meta-training, over 30 seeds and 3000 test tasks per seed.
Figure 8: Khazad-Dum: Mean and CVaR returns over 30 seeds and 3000 test tasks.
D.2 Continuous Control
In all the MuJoCo benchmarks introduced in Section 6.2, each task's meta-rollout consists of 2 episodes  $\times 200$  time-steps per episode. Below we describe the task distributions, as well as their parameterization for the CEM. In each benchmark, we used CEM quantile  $\beta = 0.2$  and regularization of  $\nu = 0.2$  samples per batch.
- HalfCheetah-Vel: The original task distribution is uniform  $\tau \sim U([0,7])$  in HalfCheetah (the task space  $[0,7]$  was extended in comparison to Zintgraf et al. [2019], to create a more significant tradeoff between tasks). We rewrite it as  $\tau = 7\tilde{\tau}$ ,  $\tilde{\tau} \sim Beta(2\phi, 2 - 2\phi)$  with  $\phi_0 = 0.5$  (leading to  $Beta(1,1)$ , which is indeed the uniform distribution). The CEM learns to modify  $\phi$ . Notice that this parameterization satisfies  $\mathbb{E}_{D_{\phi}}[\tilde{\tau}] = \phi$ .
- Humanoid-Vel: Same as HalfCheetah-Vel, with task space [0, 2.5] instead of [0, 7].
- Ant-Goal: The target location is random within a circle of radius 5. We represent the target location in polar coordinates, and write  $r \sim Beta(2\phi_1, 2 - 2\phi_1)$  and  $\theta \sim Beta(2\phi_2, 2 - 2\phi_2)$  (up to multiplicative factors 5 and  $2\pi$ ). The original distribution parameter is  $\phi_0 = (0.5, 0.5)$ , and the CEM learns to modify it.
- HalfCheetah-Mass, Humanoid-Mass, Ant-Mass: The task  $\tau \in [0.5, 2]$  corresponds to the multiplicative factor of the body mass (e.g.,  $\tau = 2$  is a doubled mass). The original task distribution is uniform over the log factor, i.e.,  $\log_2\tau \sim U([-1, 1])$ . Again, we re-parameterize the uniform distribution as Beta, and learn to modify its parameter.
- HalfCheetah-Body, Humanoid-Body, Ant-Body: The 3 components of the task correspond to multiplicative factors of different physical properties, and each of them is distributed independently and uniformly in log, i.e.,  $\forall 1 \leq j \leq 3 : \log_2\tau_j \sim U([-1,1])$ . We re-parameterize this as 3 independent Beta distributions with parameters  $\phi = (\phi_1, \phi_2, \phi_3)$ .
- HalfCheetah 10D-task: Again, the task components correspond to multiplicative factors of different properties of the model. This time, there are 10 different properties (i.e., the task space is 10-dimensional), but each of them varies in a smaller range:  $\log_2\tau_j\sim$
Table 2: Mean return over 1000 test tasks, for different models and MuJoCo environments. Standard deviation is presented over 30 seeds. CVaR returns are displayed in Table 1.
HalfCheetah	HalfCheetah 10D-task
Vel	Mass	Body	(a)	(b)	(c)
CeSoR	-1316 ± 18	1398 ± 31	1008 ± 34	1222 ± 23	1388 ± 20	1274 ± 32
PAIRED	-545 ± 55	662 ± 30	492 ± 51	551 ± 53	706 ± 36	561 ± 65
CVaR-ML	-574 ± 22	113 ± 8	193 ± 6	263 ± 15	250 ± 11	192 ± 5
PEARL	-534 ± 15	1726 ± 13	1655 ± 6	1843 ± 9	1866 ± 13	1425 ± 6
VariBAD	-82 ± 2	1558 ± 32	1616 ± 28	1893 ± 6	1984 ± 67	1617 ± 12
RoML (VariBAD)	-95 ± 3	1581 ± 32	1582 ± 21	1819 ± 8	1950 ± 20	1616 ± 13
RoML (PEARL)	-519 ± 15	1553 ± 18	1437 ± 8	1783 ± 7	1859 ± 10	1399 ± 8
Humanoid	Ant
Vel	Mass	Body	Goal	Mass	Body
VariBAD	880 ± 4	1645 ± 22	1678 ± 17	-229 ± 3	1473 ± 3	1476 ± 1
RoML (VariBAD)	883 ± 4	1580 ± 17	1618 ± 18	-224 ± 3	1475 ± 2	1472 ± 1
$U([-0.5, 0.5])$ . Each such property is a vector, and is multiplied by  $\tau_{j}$  when executing the task  $\tau$ . The 10 properties are selected randomly, among all the variables of type float array in env.model. We generate 3 such MRL environments - with 3 different random sets of 10 task-variables each. Some examples for properties are inertia, friction and mass.
In the experiments, we rely on the official implementations of VariBAD [Zintgraf et al., 2019] and PEARL [Rakelly et al., 2019], both published under the MIT license. CVaR-ML and RoML are implemented on top of these baseline, and their running times are indistinguishable from the baselines. All experiments were performed on machines with Intel Xeon 2.2 GHZ CPU and NVIDIA's V100 GPU. Each experiment (meta-training and testing) required 12-72 hours, depending on the environment and the baseline algorithm.
Table 2 and Fig. 9 present detailed results for our MuJoCo experiments, in addition to the results presented in Section 6.2.
(a) HalfCheetah-Vel
(c) Humanoid-Vel
(b) HalfCheetah-Body (mass, damping and head-size)
(e) Humanoid-Mass
(d) Humanoid-Body (mass, damping and head-size)
(f) Ant-Mass
(g) Ant-Goal
Figure 9: Average return per range of tasks in the various MuJoCo environments (global average in parentheses). HalfCheetah-Mass is displayed in Fig. 4d.
D.3 The Cross Entropy Method
We implemented RoML using the Dynamic Cross Entropy Method implementation of Greenberg [2022]. Below we concentrate results related to the CEM functionality in all the experiments:
- Learned sample distribution: One set of figures corresponds to the learned sample distribution, as measured via  $\phi$  throughout the meta-training.
- Sample returns: A second set of figures corresponds to the returns over the sampled tasks (corresponding to  $D_{\phi}$ ): ideally, we would like them to align with the  $\alpha$ -tail of the reference returns (corresponding to  $D_{\phi_0}$ ). Thus, the figures present the mean sample return along with the mean and CVaR reference returns (the references are estimated from the sample returns using Importance Sampling weights). In most figures, we see that the sample returns (in green) shift significantly from the mean reference return (in blue) towards the CVaR reference return (orange), at least for part of the training. Note that in certain environments, the distinction between difficulty of tasks can only be made after the agent has already learned a basic meaningful policy, hence the sample returns do not immediately deviate from the mean reference.
(a) Learned sample distribution
Figure 10: The CEM in Khazad-Dum. Note that the effect of the CEM concentrates at the first half of the meta-training; once the meta-policy learns to focus on the long path, the agent becomes invariant to the sampled tasks, and the sampler gradually returns to the original task distribution  $\phi \approx \phi_0$ .
(b) Sample returns
(b) HalfCheetah-Mass
(c) HalfCheetah-Body
(d) Humanoid-Mass
Figure 11: Learned sample distribution in MuJoCo benchmarks. Notice that for HalfCheetah-Body, the CEM has to control 3 different task parameters simultaneously.
(a) HalfCheetah-Vel
Figure 13: Sine Regression: The 3 left figures show the learned sample distribution, corresponding to average amplitudes, phases and frequencies. The CEM immediately learns that the amplitude has the strongest effect on test loss, whereas frequency has a moderate effect and phase has none. The right figure shows in green the mean sample scores (in supervised learning these are the negative losses instead of the returns).
Figure 12: Sample returns in MuJoCo benchmarks.
D.4 Ablation Test: RoML with a Naive Sampler
RoML learns to be robust to high-risk tasks by using the CEM to over-sample them throughout training. The dynamic CEM presented in Section 5 is designed to identify and sample high-risk tasks from a possibly-infinite task-space, where the task values change with the evolving policy throughout training.