SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1 guarantees unbiased gradients when using Algorithm 1; however, it does not bound their variance. In particular, Line 8 applies the learning step to a subset of only  $\alpha NM$  meta-rollouts out of  $NM$ , which increases the estimator variance by a factor of  $\alpha^{-1}$  compared to mean optimization. This could be prevented if we knew the set  $\Omega_{\alpha}^{\theta}$  of tail-tasks (for the current  $\pi_{\theta}$ ), and sampled only these tasks, using the distribution  $D_{\alpha}^{\theta}(z) = \alpha^{-1}\mathbf{1}_{V_z^\theta \leq q_\alpha(V_\tau^\theta)}D(z)$ . Proposition 1 shows that this would indeed recover the sample efficiency.
Proposition 1 (Variance reduction). Denote by  $G$  the estimator of  $\nabla_{\theta}J_{\alpha}^{\theta}(R)$  in Eq. (8), assume there is no quantile error  $(\hat{q}_{\alpha}^{\theta} = q_{\alpha}^{\theta})$ , and denote  $\mathbb{E}_D[\cdot ] = \mathbb{E}_{z_i\sim D,R_{i,m}\sim P_{z_i}^{\theta}}[\cdot ]$ . Then, switching the task sample distribution to  $D_{\alpha}^{\theta}$  leads to a variance reduction of factor  $\alpha$ :
$$
\mathbb {E} _ {D _ {\alpha} ^ {\theta}} [ \alpha G ] = \mathbb {E} _ {D} [ G ], \quad \operatorname {V a r} _ {D _ {\alpha} ^ {\theta}} (\alpha G) \leq \alpha \cdot \operatorname {V a r} _ {D} (G).
$$
Proof sketch (the complete proof is in Appendix B). We calculate the expectation and variance directly.  $G$  is proportional to  $\mathbf{1}_{R_i\leq q_\alpha^\theta}$  (Eq. (8)). The condition  $R_{i}\leq q_{\alpha}^{\theta}$  leads to multiplication by the probability  $\alpha$  when sampled from  $D$  (where it corresponds to the  $\alpha$ -tail); but not when sampled from  $D_{\alpha}^{\theta}$  (where it is satisfied w.p. 1). This factor  $\alpha$  cancels out the ratio between the expectations of  $G$  and  $\alpha G$  (thus the expectations are identical) - but not the ratio  $\alpha^2$  between their variances.
Figure 2: Left: RoML uses the cross entropy method to modify the task distribution  $D_{\phi}$ , which is used to generate the next meta-rollouts. Right: illustration of an arbitrary point of time in training: the task distribution  $D_{\phi}$  (blue) is learned according to the task values of the current meta-policy  $\pi_{\theta}$  (red). Since low-return tasks are over-sampled, the learned meta-policy is more robust to the selection of task.
Following the motivation of Proposition 1, we wish to increase the number of train tasks that come from the tail distribution  $\Omega_{\alpha}^{\theta}$ . To that end, we assume to have certain control over the sampling of training tasks. This assumption is satisfied in most simulated environments, as well as many real-world scenarios. For example, when training a driver, we choose the tasks, roads and times of driving throughout training. In this section, we propose a method to make these choices.
We begin with parameterizing the task distribution  $D$ : we consider a parametric family  $D_{\phi}$  such that  $D = D_{\phi_0}$ . Then, we wish to modify the parameter  $\phi$  so that  $D_{\phi}$  aligns with  $D_{\alpha}^{\theta}$  as closely as possible. To that end, we use the Cross Entropy Method (CEM, de Boer et al. [2005]), which searches for  $\phi^{*}$  that minimizes the KL-divergence (i.e., cross entropy) between the two:
$$
\begin{array}{l} \phi^ {*} \in \operatorname {a r g m i n} _ {\phi^ {\prime}} D _ {K L} \left(D _ {\alpha} ^ {\theta} \mid \mid D _ {\phi^ {\prime}}\right) = \operatorname {a r g m a x} _ {\phi^ {\prime}} \mathbb {E} _ {z \sim D _ {\phi_ {0}}} \left[ \mathbf {1} _ {V _ {z} ^ {\theta} \leq q _ {\alpha} (V _ {\tau} ^ {\theta})} \log D _ {\phi^ {\prime}} (z) \right] \\ \begin{array}{l} \arg \max  _ {\phi^ {\prime}} \mathbb {E} _ {z ^ {\prime}} \sim D _ {\phi_ {0}} \left[ - V _ {z ^ {\prime}} \leq q _ {\alpha} \left(V _ {\tau} ^ {\theta}\right) \log E _ {\phi^ {\prime}} (z) \right] \\ = \arg \max  _ {\phi^ {\prime}} \mathbb {E} _ {z \sim D _ {\phi}} \left[ w (z) \mathbf {1} _ {V _ {z ^ {\prime}} ^ {\theta} \leq q _ {\alpha} \left(V _ {\tau} ^ {\theta}\right)} \log D _ {\phi^ {\prime}} (z) \right], \end{array} \tag {9} \\ \end{array}
$$
where  $w(z) = \frac{D_{\phi_0}(z)}{D_{\phi}(z)}$  is the importance sampling weight corresponding to  $z \sim D_{\phi}$ . Note that Eq. (9) has a closed-form solution for most of the standard families of distributions [de Boer et al., 2005].
Algorithm 2: Robust Meta RL (RoML)
1 Input: Meta-learning algorithm (Definition 2); robustness level  $\alpha \in (0,1]$ ; parametric task distribution  $D_{\phi}$ ; original parameter  $\phi_0$ ;  $N$  tasks per batch,  $\nu \in [0,1)$  of them sampled from the original  $D_{\phi_0}$ ; CEM quantile  $\beta \in (0,1)$
2 Initialize:
3  $\phi \gets \phi_0,\quad N_o\gets \lfloor \nu N\rfloor ,\quad N_s\gets \lceil (1 - \nu)N\rceil$
4 while not finished training do // Sample tasks
7  $\{\Lambda_i\}_{i = 1}^N\gets$  rollout(  $\{z_i\}_{i = 1}^N)$
8  $R_{i}\gets \mathrm{return}(\Lambda_{i}),\quad \forall i\in \{1,\ldots ,N\}$
9  $\mathsf{ML}(\{\Lambda_i\}_{i = 1}^N)$  // Estimate reference quantile
10  $w_{o,i}\gets 1,$ $\forall i\in \{1,\ldots ,N_o\}$
11  $w_{\phi ,i}\gets \frac{D_{\phi_0}(z_{\phi,i})}{D_{\phi}(z_{\phi,i})},\quad \forall i\in \{1,\ldots ,N_s\}$
12  $w\gets \left(w_{o,1},\dots ,w_{o,N_o},w_{\phi ,1},\dots ,w_{\phi ,N_s}\right)$
13  $\hat{q}_{\alpha}\gets$  weightedquantile(  $\{R_i\}$  ,w,  $\alpha$  // Compute sample quantile
14  $q_{\beta}\gets$  quantile  $(\{R_i\} ,\beta)$  // Update sampler
15  $q\gets \max (\hat{q}_\alpha ,q_\beta)$
16  $\phi \gets \operatorname{argmax}_{\phi'} \sum_{i \leq N} w_i \mathbf{1}_{R_i \leq q} \log D_{\phi'}(z_i)$
5 Sample  $\{z_{o,i}\}_{i = 1}^{N_o}\sim D_{\phi_0},\{z_{\phi ,i}\}_{i = 1}^{N_s}\sim D_{\phi}$  6  $z\gets (z_{o,1},\dots,z_{o,N_o},z_{\phi ,1},\dots,z_{\phi ,N_s})$  // Rollouts and meta-learning step
For a batch of  $N$  tasks sampled from  $D = D_{\phi_0}$ , Eq. (9) essentially chooses the  $\alpha N$  tasks with the lowest returns, and updates  $\phi$  to focus on these tasks. This may be noisy unless  $\alpha N \gg 1$ . Instead, the CEM chooses a larger number of tasks  $\beta > \alpha$  for the update, where  $\beta$  is a hyper-parameter.  $\phi$  is updated according to these  $\beta N$  lowest-return tasks, and the next batch is sampled from  $D_{\phi} \neq D_{\phi_0}$ . This repeats iteratively: every batch is sampled from  $D_{\phi}$ , where  $\phi$  is updated according to the  $\beta N$  lowest-return tasks of the former batch. Each task return is also compared to the  $\alpha$ -quantile of the original distribution  $D_{\phi_0}$ . If more than  $\beta N$  tasks yield lower returns, the CEM permits more samples for the update step. The return quantile over  $D_{\phi_0}$  can be estimated from  $D_{\phi}$  at any point using importance sampling weights. See more details about the CEM in Appendix C.1.
In our problem, the target distribution is the tail of  $D_{\phi_0}$ . Since the tail is defined by the agent returns in these tasks, it varies with the agent and is non-stationary throughout training. Thus, we use the dynamic-target CEM of Greenberg [2022]. To smooth the changes in the sampled tasks, the sampler is also regularized to always provide certain
exposure to all the tasks: we force  $\nu$  percent of every batch to be sampled from the original distribution  $D = D_{\phi_0}$ , and only  $1 - \nu$  percent from  $D_{\phi}$ .
Putting this together, we obtain the Robust Meta RL algorithm (RoML), summarized in Algorithm 2 and Fig. 2. RoML does not require multiple meta-rollouts per update (parameter  $M$  in Algorithm 1), since it directly models high-risk tasks. Similarly to CVaR-ML, RoML is a meta-algorithm and can operate on top of any meta-learning baseline (Definition 2). Given the baseline implementation, only the tasks sampling procedure needs to be modified, which makes RoML easy to implement.
Limitations: The CEM's adversarial tasks sampling relies on several assumptions. Future research may reduce some of these assumptions, while keeping the increased data efficiency of RoML.
First, as mentioned above, we need at least partial control over the selection of training tasks. This assumption is common in other RL frameworks [Dennis et al., 2020, Jiang et al., 2021], and often holds in both simulations and the real world (e.g., choosing in which roads and hours to train driving).
Second, the underlying task distribution  $D$  is assumed to be known, and the sample distribution is limited to the chosen parameterized family  $\{D_{\phi}\}$ . For example, if  $\tau \sim U([0,1])$ , the user may choose the family of Beta distributions  $\text{Beta}(\phi_1,\phi_2)$  (where  $\text{Beta}(1,1) \equiv U([0,1])$ ), as demonstrated in Appendix D.2. The selected family expresses implicit assumptions on the task-space. For example, if the probability density function is smooth, close tasks will always have similar sample probabilities; and if the family is unimodal, high-risk tasks can only be over-sampled from around a single peak. This approach is useful for generalization across continuous task-spaces – where the CEM can never observe the infinitely many possible tasks. Yet, it may pose limitations in certain discrete task-spaces, if there is no structured relationship between tasks.
We implement RoML and CVaR-ML on top of two different risk-neutral MRL baselines - VariBAD [Zintgraf et al., 2019] and PEARL [Rakelly et al., 2019]. As a risk-averse reference for comparison, we use CeSoR [Greenberg et al., 2022], an efficient sampling-based method for CVaR optimization in RL, implemented on top of PPO. As another reference, we use the Unsupervised Environment Design algorithm PAIRED [Dennis et al., 2020], which uses regret minimization to learn robust policies on a diverse set of tasks.
Section 6.1 demonstrates the mean/CVaR tradeoff, as our methods learn substantially different policies from their baseline. Section 6.2 demonstrates the difficulty of the naive CVaR-ML in more challenging control benchmarks, and the RoML's efficacy in them. The ablation test in Appendix D.4 demonstrates that RoML deteriorates significantly when the CEM is replaced by a naive adversarial sampler. Section 6.3 presents an implementation of CVaR-ML and RoML on top of MAML [Finn et al., 2017] for supervised meta-learning. In all the experiments, the running times of RoML and CVaR-ML are indistinguishable from their baselines (RoML's CEM computations are negligible).
Hyper-parameters: To test the practical applicability of RoML as a meta-algorithm, in every experiment, we use the same hyper-parameters for RoML, CVaR-ML and their baseline. In particular, we use the baseline's default hyper-parameters whenever applicable (Zintgraf et al. [2019], Rakelly et al. [2019] in Section 6.2, and Finn et al. [2017] in Section 6.3). That is, we use the same hyper-parameters as originally tuned for the baseline, and test whether RoML improves the robustness without any further tuning of them. As for the additional hyper-parameters of the meta-algorithm itself: in Algorithm 1, we use  $M = 1$  meta-rollout per task; and in Algorithm 2, we use  $\beta = 0.2$ ,  $\nu = 0$  unless specified otherwise (similarly to the CEM in Greenberg et al. [2022]). For the references PAIRED and CeSoR, we use the hyper-parameters of Dennis et al. [2020], Greenberg et al. [2022]. Each experiment is repeated for 30 seeds. See more details in Appendix D. The code is available in our repositories: VariBAD, PEARL, CeSoR, PAIRED and MAML.
6.1 Khazad Dum
We demonstrate the tradeoff between mean and CVaR optimization in the Khazad Dum benchmark, visualized in Fig. 3. The agent begins at a random point in the bottom-left part of the map, and has to reach the green target as quickly as possible, without falling into the black abyss. The bridge is not covered and thus is exposed to wind and rain, rendering its floor slippery and creating an additive action noise (Fig. 3b) – to a level that varies with the weather. Each task is characterized by the rain intensity, which is exponentially distributed. The CEM in RoML is allowed to modify the parameter of this exponential distribution. Note that the agent is not aware of the current task (i.e., the weather), but may infer it from observations. We set the target risk level to  $\alpha = 0.01$ , and train each meta-agent for a total of  $5 \cdot 10^{6}$  frames. See complete details in Appendix D.1.
We implement CVaR-ML and RoML on top of VariBAD. As shown in Fig. 3, VariBAD learns to try the short path – at the risk of rare falls into the abyss. By contrast, our CVaR-optimizing methods take the longer path and avoid risk. This policy increases the cumulative cost of time-steps, but leads to higher CVaR returns, as shown in Fig. 3f. In addition to superior CVaR, RoML also provides competitive average returns in this example (Fig. 3e). Finally, in accordance with Proposition 1, RoML learns significantly faster than CVaR-ML.
(a) VariBAD
(b) VariBAD
(c) CVaR-ML
(e) Mean
Figure 3: Khazad-Dum: (a-d) Sample episodes. (e-f) Test return vs. training iteration, with  $95\%$  confidence intervals over 30 seeds.
(d) RoML
(f) CVaR
Figure 4: MuJoCo: (a-c) Sample frames, where tasks vary in mass, head size and damping level. Notice that in Humanoid, RoML handles the low-return tasks of large mass by leaning the center of mass forward, so that gravity pulls the humanoid forward. (d) Average return per range of tasks in HalfCheetah-Mass. RoML learns to act robustly: it is less sensitive to the task, and in particular performs better on high-risk tasks.
(b) HalfCheetah-Body
(a) Hum-Mass (VariBAD)
(c) Hum-Mass (RoML)
(d) HalfCheetah-Mass
We rely on standard continuous control problems from the MuJoCo framework [Todorov et al., 2012]: training a cheetah to run (HalfCheetah), and training a Humanoid and an Ant to walk. For each of the 3 environments, we create 3 meta-learning versions: (1) Goal or Vel [Finn et al., 2017], where each task corresponds to a different location or velocity objective, respectively; (2) Mass, where each task corresponds to a different body mass; and (3) Body, where each task corresponds to different mass, head size and physical damping level (similarly to Wang and Van Hoof [2022]). In addition, to experiment with high-dimensional task spaces, we randomly draw 10 numeric variables from env.model in HalfCheetah, and let them vary between tasks. We define 3 such environments with different random sets of task variables (HalfCheetah 10D-task a,b,c). For each of the 12 environments above, we set a target risk level of  $\alpha = 0.05$  and optimize for  $K = 2$  episodes per task. Additional implementation details are specified in Appendix D.2.
Interestingly, the naive approach of CVaR-ML consistently fails to meta-learn in all the cheetah environments. It remains unsuccessful even after large number of steps, indicating a difficulty beyond sample inefficiency. A possible explanation is the effectively decreased batch size of CVaR-ML. PAIRED and CeSoR also fail to adjust to the MRL environments, and obtain poor CVaR returns.
RoML, on the other hand, consistently improves the CVaR returns (Table 1) compared to its baseline (VariBAD or PEARL), while using the same hyper-parameters as the baseline. The VariBAD baseline presents better returns and running times than PEARL on HalfCheetah, and thus is used for the 6 extended environments (Humanoid and Ant). RoML improves the CVaR return in comparison to the baseline algorithm in all the 18 experiments (6 with PEARL and 12 with VariBAD).
In 5 out of 18 experiments, RoML slightly improves the average return compared to its baseline, and not only the CVaR (Table 2 in the appendix). This indicates that low-return tasks can sometimes be improved at the cost of high-return tasks, but without hurting average performance. In addition, this may indicate that over-sampling difficult tasks forms a helpful learning curriculum.
The robustness of RoML to the selected task is demonstrated in Fig. 4d. In multi-dimensional task spaces, RoML learns to focus the sampling modification on the high-impact variables, as demonstrated
Table 1:  $\mathrm{CVaR}_{0.05}$  return over 1000 test tasks, for different models and MuJoCo environments. Standard deviation is presented over 30 seeds. Mean returns are displayed in Table 2.
HalfCheetah	HalfCheetah 10D-task
Vel	Mass	Body	(a)	(b)	(c)
CeSoR	-2606 ± 25	902 ± 36	478 ± 27	637 ± 26	981 ± 31	664 ± 26
PAIRED	-725 ± 65	438 ± 37	218 ± 51	229 ± 59	354 ± 53	81 ± 65
CVaR-ML	-897 ± 23	38 ± 6	76 ± 5	120 ± 11	141 ± 11	81 ± 4
PEARL	-1156 ± 23	1115 ± 19	800 ± 5	1140 ± 33	1623 ± 23	1016 ± 5
VariBAD	-202 ± 6	1072 ± 16	835 ± 30	1126 ± 6	1536 ± 39	988 ± 13
RoML (VariBAD)	-184 ± 4	1259 ± 19	935 ± 17	1227 ± 13	1697 ± 24	999 ± 20
RoML (PEARL)	-1089 ± 31	1186 ± 34	808 ± 6	1141 ± 27	1657 ± 18	1024 ± 6
Humanoid	Ant
Vel	Mass	Body	Goal	Mass	Body
VariBAD	801 ± 10	1283 ± 18	1290 ± 19	-500 ± 9	1370 ± 6	1365 ± 4
RoML (VariBAD)	833 ± 4	1378 ± 20	1365 ± 21	-454 ± 8	1385 ± 3	1368 ± 4
6.3 Beyond RL: Robust Supervised Meta-Learning
Our work focuses on robustness in MRL. However, the concept of training on harder data to improve robustness, as embodied in RoML, is applicable beyond the scope of RL. As a preliminary proof-of-concept, we apply RoML to a toy supervised meta-learning problem of sine regression, based on Finn et al. [2017]: The input is  $x \in [0, 2\pi)$ , the desired output is  $y = A \sin (\omega x + b)$ , and the task is defined by the parameters  $\tau = (A, b, \omega)$ . Similarly to Finn et al. [2017], the model is fine-tuned for each task via a gradient-descent optimization step over 10 samples  $\{(x_i, y_i)\}_{i=1}^{10}$ , and is tested on another set of 10 samples. The goal is to find model weights that adapt quickly to new task data.
in Fig. 11 in the appendix. Finally, qualitative inspection shows that RoML learns to handle larger masses, for example, by leaning forward and letting gravity pull the agent forward, as displayed in Fig. 4c (see animations on GitHub).
Figure 5: Supervised Sine Regression: CVaR loss over 10000 test tasks, against the number of tuning gradient-steps at test time. The  $95\%$  confidence intervals are calculated over 30 seeds.
We implement CVaR-ML and RoML on top of MAML [Finn et al., 2017]. As shown in Fig. 5, RoML achieves better CVaR losses over tasks than both CVaR-ML and MAML. The complete setting and results are presented in Appendix E.
7 Summary and Future Work
We defined a robust MRL objective and derived the CVaR-ML algorithm to optimize it. In contrast to its analogous algorithm in standard RL, we proved that CVaR-ML does not present biased gradients, yet it does inherit the same data inefficiency. To address the latter, we introduced RoML and demonstrated its advantage in sample efficiency and CVaR return.
Future research may address the CEM-related limitations of RoML discussed at the end of Section 5. Another direction for future work is extension of RoML to other scenarios, especially where a natural task structure can be leveraged to improve task robustness, e.g., in supervised learning and coordinate-based regression [Tancik et al., 2020].
Finally, RoML is easy to implement, operates agnostically as a meta-algorithm on top of existing MRL methods, and can be set to any desired robustness level. We believe that these properties, along with our empirical results, make RoML a promising candidate for MRL in risk-sensitive applications.
Acknowledgements: This work has received funding from the European Union's Horizon Europe Programme, under grant number 101070568.
Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforcement learning with online self-supervision. In International Conference on Machine Learning, pages 17811-17829. PMLR, 2022.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. ICLR, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
Frederik Schubert, Theresa Eimer, Bodo Rosenhahn, and Marius Lindauer. Automatic risk adaptation in distributional reinforcement learning. arXiv preprint arXiv:2106.06317, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. AAAI'15, page 2993-2999, 2015.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations, 2020. URL https://arxiv.org/abs/2012.02189.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
J. R. R. Tolkien. The Lord of the Rings: The Fellowship of the Ring. George Allen and Unwin, 1954.
Anthony Tromba and Jerrold E Marsden. Vector calculus. Freeman, 1996.
Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pages 23055-23077. PMLR, 2022.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256, 1992.
Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. Prototransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107.14035, 2021.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representations, 2019.
1 Introduction 1
2 Related Work 3
3 Preliminaries 3
4 CVaR Optimization in Meta-Learning 4
5 Efficient CVaR-ML 6
6 Experiments 8
7 Summary and Future Work 10
A Policy Gradient for CVaR Optimization in Meta RL 15
B Proof of Proposition 1 18
C The Cross Entropy Method 19
D Experiments: Detailed Settings and Results 19
E Supervised Meta-Learning 24
A Policy Gradient for CVaR Optimization in Meta RL
In this section we provide the complete proof for Theorem 1. For completeness, Appendix A.1 recaps of the analogous proof in standard (non-meta) RL, before moving on to the proof in Appendix A.2. This allows us to highlight the differences between the two.
The substantial difference is that in RL, the CVaR is defined directly over the low-return trajectories, and the policy parameter  $\theta$  affects the probability of each trajectory in the tail (Eq. (10)). In MRL (Eq. (2)), on the other hand, the CVaR is defined over the low-return tasks, whose probability is not affected directly by  $\theta$  (Eq. (12)). This allows a decoupling between  $\theta$  and  $\tau$ , which results in Theorem 1.
Another high-level intuition is as follows. In RL, the CVaR-PG gradient is invariant to successful strategies, hence must be explicitly negative for the unsuccessful ones (in order not to encourage them, see Fig. 6). In MRL, within the tasks of interest, the gradient always encourages successful strategies on account of the unsuccessful ones (Fig. 7).
A.1 Recap: PG for CVaR in (non-meta) RL
We briefly recap the calculation of Propositions 1 and 2 in Tamar et al. [2015] for CVaR policy gradient under the standard RL settings.
Definition 3 (CVaR return in (non-meta) RL). Consider an MDP  $(S, A, P, \gamma, P_0)$  with the cumulative reward (i.e., return)  $R \sim P^{\theta}$ , whose  $\alpha$ -quantile is  $q_{\alpha}^{\theta}(R)$ . Recall the CVaR objective defined in Section 3: