SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Technion, Nvidia Research gido@campus.technion.ac.il
Bar Ilan University, Nvidia Research gchechik@nvidia.com
Technion, Nvidia Research shie@ee.technion.ac.il
Eli Meirom
Nvidia Research emeirom@nvidia.com
A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both biased gradients and data inefficiency. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and oversampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.
Reinforcement learning (RL) has achieved impressive results in a variety of applications in recent years, including cooling systems control [Luo et al., 2022] and conversational chatbots [Cohen et al., 2022]. A significant challenge in extending this success to mass production is the variation between instances of the problem, e.g., different cooling systems or different chatbot end-users. Meta-RL (MRL) addresses this challenge by learning a "meta-policy" that quickly adapts to new tasks [Thrun and Pratt, 1998, Finn et al., 2017]. In the examples above, MRL would maximize the average return of the adapted policy for a new cooling system or a new end-user.
Figure 1: (a) An illustration of driving tasks, characterized by various weather conditions and traffic density. (b) The returns of two meta-policies  $\pi_1$ ,  $\pi_2$  on these tasks.  $\pi_1$  has a higher average return, but  $\pi_2$  is more robust to high-risk tasks. The task space is discretized only for illustration purposes.
However, optimizing the return of the average client might not suffice, as certain clients may still experience low or even negative returns. If  $10\%$  of the clients report poor performance, it may deter potential clients from adopting the new technology - even if its average return is high. This highlights
the need for MRL systems that provide robust returns across tasks. Robustness is further motivated by risk sensitivity in many natural RL applications, such as medical treatment [Yu et al., 2021] and driving [Shalev-Shwartz et al., 2016]. For example, as illustrated in Fig. 1, an agent should drive safely at any road profile – even if at some roads the driving would be more cautious than necessary.
A common approach for risk-averse optimization is the max-min objective [Collins et al., 2020]; in MRL, that would mean searching for a meta-policy with the highest expected-return in the worst possible task. This expresses the most extreme risk aversion, which only attends to the one worst case out of all the possible outcomes. Furthermore, in certain problems, worst cases are inevitable (e.g., in certain medical treatments, a fatal outcome cannot be avoided), thus optimizing the minimum return might not provide any meaningful result. A more general objective is the average return over the worst  $\alpha$  quantiles ( $0 \leq \alpha \leq 1$ ), also known as the Conditional Value-at-Risk (CVaR). Notice that the CVaR is a generalization of both the mean (for  $\alpha = 1$ ) and the minimum (for  $\alpha = 0$ ).
CVaR is a coherent risk measure used for risk management in various fields [Filippi et al., 2020], including banking regulation [Acerbi and Szekely, 2014] and RL [Tamar et al., 2015, Hiraoka et al., 2019]. In this work, we extend CVaR optimization to MRL, by replacing the standard MRL objective
$$
\underset {\pi} {\operatorname {a r g m a x}} J ^ {\theta} (R), \quad J ^ {\theta} (R) = \mathbb {E} _ {\tau , R} [ R ], \tag {1}
$$
with a CVaR objective, which measures the robustness of a policy to high-risk tasks:
$$
\underset {\pi} {\operatorname {a r g m a x}} J _ {\alpha} ^ {\theta} (R), \quad J _ {\alpha} ^ {\theta} (R) = \mathrm {C V a R} _ {\tau} ^ {\alpha} [ \mathbb {E} _ {R} [ R ] ]. \tag {2}
$$
In both equations,  $\tau$  is a random task and  $R$  is the random return of policy  $\pi_{\theta}$  in  $\tau$ . Intuitively, the CVaR return expresses robustness to the selected task, in analogy to robustness to the realized model in standard RL. To further motivate Eq. (2), note that CVaR optimization in RL is equivalent to robust optimization under uncertain perturbations [Chow et al., 2015].
In Section 4, we follow the standard approach of policy gradient (PG) for  $\mathrm{CVaR}_{\alpha}$  optimization in RL, and apply it to MRL. That is, for every batch of  $N$  trajectories, we apply the learning step to the  $\alpha N$  trajectories with the lowest returns. This standard approach, CVaR-PG, is known to suffer from a major limitation: in an actor-critic framework, the critic leads to a biased gradient estimator – to the extent that it may point to the opposite direction [Tamar et al., 2015]. This limitation is quite severe: many CVaR-PG implementations [Tamar et al., 2015, Greenberg et al., 2022] rely on vanilla PG without a critic (REINFORCE, Williams [1992]); others pay the price of gradient bias – in favor of advanced actor-critic methods that reduce the gradient variance [Rajeswaran et al., 2017].
This limitation is particularly concerning in meta RL, where high complexity and noise require more sophisticated algorithms than REINFORCE. Fortunately, Section 4 eliminates this concern: in MRL, in contrast to RL, the CVaR policy gradient is proven to remain unbiased regardless of the choice of critic. Hence, our proposed method - CVaR Meta Learning (CVaR-ML) - can be safely applied on top of any MRL algorithm. This makes CVaR-ML a meta-algorithm: given an arbitrary MRL algorithm, CVaR-ML generates a robust version of it.
Nevertheless, in CVaR optimization methods, another source of gradients variance and sample inefficiency is the large proportion of data not being utilized. Every iteration, we rollout trajectories for  $N$  tasks, but only use  $\alpha N$  of them for training. To mitigate this effect, we introduce in Section 5 the Robust Meta RL algorithm (RoML). RoML assumes that tasks can be selected during training. It learns to identify tasks with lower returns and over-samples them. By training on high-risk tasks, the meta-agent learns policies that are robust to them without discarding data. Hence, RoML increases the sample efficiency by a factor of up to  $\alpha^{-1}$ . Unlike common adversarial methods, which search for the worst-case sample (task) that minimizes the return [Collins et al., 2020], RoML lets the user specify the desired level of robustness  $\alpha$ , and addresses the entire  $\alpha$ -tail of the return distribution.
We test our algorithms on several domains. Section 6.1 considers a navigation problem, where both CVaR-ML and RoML obtain better CVaR returns than their risk-neutral baseline. Furthermore, they learn substantially different navigation policies. Section 6.2 considers several continuous control environments with varying tasks. These environments are challenging for CVaR-ML, which entirely fails to learn. Yet, RoML preserves its effectiveness and consistently improves the robustness of the returns. In addition, Section 6.3 demonstrates that under certain conditions, RoML can be applied to supervised settings as well - providing robust supervised meta-learning.
As a meta-algorithm, in each experiment RoML improves the robustness of its baseline algorithm - using the same hyper-parameters as the baseline. The average return is also improved in certain experiments, indicating that even the risk-neutral objective of Eq. (1) may benefit from robustness.
Contribution: (a) We propose a principled CVaR optimization framework for robust meta-RL. While the analogous problem in standard RL suffers from biased gradients and data inefficiency, we (b) prove theoretically that MRL is immune to the former, and (c) address the latter via the novel Robust Meta RL algorithm (RoML). Finally, (d) we demonstrate the robustness of RoML experimentally.
Meta-RL for the average task is widely researched, including methods based on gradients [Finn et al., 2017, Gupta et al., 2018], latent memory [Zintgraf et al., 2019, Rakelly et al., 2019] and offline meta learning [Dorfman et al., 2020, Pong et al., 2022]. It is used for applications ranging from robotics [Nagabandi et al., 2018] to education [Wu et al., 2021]. Adversarial meta learning was studied for minimax optimization of the lowest-return task, in supervised meta learning [Collins et al., 2020, Goldblum et al., 2020] and MRL [Lin et al., 2020]. Other works studied the robustness of MRL to distributional shifts [Mendonca et al., 2020, Ajay et al., 2022]. However, the CVaR task objective has not been addressed yet in the framework of MRL.
Risk-averse RL. In standard RL, risk awareness is widely studied for both safety [García and Fernández, 2015, Greenberg and Mannor, 2021] and robustness [Derman et al., 2020]. CVaR specifically was studied using PG [Tamar et al., 2015, Rajeswaran et al., 2017, Hiraoka et al., 2019, Huang et al., 2021a], value iteration [Chow et al., 2015] and distributional RL [Dabney et al., 2018, Schubert et al., 2021, Lim and Malik, 2022]. CVaR optimization was also shown equivalent to mean optimization under robustness [Chow et al., 2015], motivating robust-RL methods [Pinto et al., 2017, Godbout et al., 2021]. In this work, we propose a meta-learning framework and algorithms for CVaR optimization, and point to both similarities and differences from the standard RL setting.
Sampling. In Section 5, we use the cross-entropy method [de Boer et al., 2005] to sample high-risk tasks for training. The cross-entropy method has been studied in standard RL for both optimization [Mannor et al., 2003, Huang et al., 2021b] and sampling [Greenberg et al., 2022]. Sampling in RL was also studied for regret minimization in the framework of Unsupervised Environment Design [Dennis et al., 2020, Jiang et al., 2021]; and for accelerated curriculum learning in the framework of Contextual RL [Klink et al., 2020, Eimer et al., 2021]. By contrast, we address MRL (where the current task is unknown to the agent, unlike Contextual RL), and optimize the CVaR risk measure instead of the mean.
MRL. Consider a set of Markov Decision Processes (MDPs)  $\{(S, A, \tau, \mathcal{P}_{\tau}, \mathcal{P}_{0,\tau}, \gamma)\}_{\tau \in \Omega}$ , where the distribution of transitions and rewards  $\mathcal{P}_{\tau}$  and the initial state distribution  $\mathcal{P}_{0,\tau}$  both depend on task  $\tau \in \Omega$ . The task itself is drawn from a distribution  $\tau \sim D$  over a general space  $\Omega$ , and is not known to the agent. The agent can form a belief regarding the current  $\tau$  based on the task history  $h$ , which consists of repeating triplets of states, actions and rewards [Zintgraf et al., 2019]. Thus, the meta-policy  $\pi_{\theta}(a; s, h)$  ( $\theta \in \Theta$ ) maps the current state  $s \in S$  and the history  $h \in \prod(S \times A \times \mathbb{R})$  (consisting of state-action-reward triplets) to a probability distribution over actions.
A meta-rollout is defined as a sequence of  $K \geq 1$  episodes of length  $T \in \mathbb{N}$  over a single task  $\tau$ :  $\Lambda = \{\{(s_{k,t},a_{k,t},r_{k,t})\}_{t=1}^{T}\}_{k=1}^{K}$ . For example, in a driving problem,  $\tau$  might be a geographic area or type of roads, and  $\Lambda$  a sequence of drives on these roads. The return of the agent over a meta-rollout is defined as  $R(\Lambda) = \frac{1}{K}\sum_{k=1}^{K}\sum_{t=0}^{T}\gamma^{t}r_{k,t}$ , where  $r_{k,t}$  is the (random variable) reward at step  $t$  in episode  $k$ . Given a task  $\tau$  and a meta-policy  $\pi_{\theta}$ , we denote by  $P_{\tau}^{\theta}(x)$  the conditional PDF of the return  $R$ . With a slight abuse of notation, we shall use  $P_{\tau}^{\theta}(\Lambda)$  to also denote the PDF of the meta-rollout itself. The standard MRL objective is to maximize the expected return  $J^{\theta}(R) = \mathbb{E}_{\tau,R}[R]$ .
While the meta policy  $\pi_{\theta}(s,a;h)$  is history-dependent, it can still be learned using standard policy gradient (PG) approaches, by considering  $h$  as part of an extended state space  $\tilde{s} = (s,h)\in \tilde{S}$ . Then,
the policy gradient can be derived directly:
$$
\nabla_ {\theta} J ^ {\theta} (R) = \int_ {\Omega} D (z) \int_ {- \infty} ^ {\infty} (x - b) \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z, \tag {3}
$$
where  $D$  is a probability measure over the task space  $\Omega$ ,  $P_{z}^{\theta}(x)$  is the PDF of  $R$  (conditioned on  $\pi_{\theta}$  and  $\tau = z$ ), and  $b$  is any arbitrary baseline that is independent of  $\theta$  [Agrawal, 2019]. While a direct gradient estimation via Monte Carlo sampling is often noisy, its variance can be reduced by an educated choice of baseline  $b$ . In the common actor-critic framework [Mnih et al., 2016], a learned value function  $b = V(s; h)$  is used. This approach is used in many SOTA algorithms in deep RL, e.g., PPO [Schulman et al., 2017]; and by proxy, in MRL algorithms that rely on them, e.g., VariBAD [Zintgraf et al., 2019].
A major challenge in MRL is the extended state space  $\tilde{S}$ , which now includes the whole task history. Common algorithms handle the task history via a low-dimensional embedding that captures transitions and reward function [Zintgraf et al., 2019]; or using additional optimization steps w.r.t. task history [Finn et al., 2017]. Our work does not compete with such methods, but rather builds upon them: our methods operate as meta-algorithms that run on top of existing MRL baselines.
CVaR-PG. Before moving on to CVaR optimization in MRL, we first recap the common PG approach for standard (non-meta) RL. For a random variable  $X$  and  $\alpha$ -quantile  $q_{\alpha}(X)$ , the CVaR is defined as  $\mathrm{CVaR}_{\alpha}(X) = \mathbb{E}[X \mid X \leq q_{\alpha}(X)]$ . For an MDP  $(S, A, P, P_0, \gamma)$ , the CVaR-return objective is  $\tilde{J}_{\alpha}^{\theta}(R) = \mathrm{CVaR}_{R \sim P^{\theta}}^{\alpha}[R] = \int_{-\infty}^{q_{\alpha}^{\theta}(R)} x \cdot P^{\theta}(x) \cdot dx$ , whose corresponding policy gradient is [Tamar et al., 2015]:
$$
\nabla_ {\theta} \tilde {J} _ {\alpha} ^ {\theta} (R) = \int_ {- \infty} ^ {q _ {\alpha} ^ {\theta} (R)} \left(x - q _ {\alpha} ^ {\theta} (R)\right) \cdot \nabla_ {\theta} P ^ {\theta} (x) \cdot d x. \tag {4}
$$
Given a sample of  $N$  trajectories  $\{\{(s_{i,t},a_{i,t})\}_{t = 1}^{T}\}_{i = 1}^{N}$  with returns  $\{R_i\}_{i = 1}^N$ , the policy gradient can be estimated by [Tamar et al., 2015, Rajeswaran et al., 2017]:
$$
\nabla_ {\theta} \tilde {J} _ {\alpha} ^ {\theta} (R) \approx \frac {1}{\alpha N} \sum_ {i = 1} ^ {N} \mathbf {1} _ {R _ {i} \leq \hat {q} _ {\alpha} ^ {\theta}} \cdot \left(R _ {i} - \hat {q} _ {\alpha} ^ {\theta}\right) \cdot \sum_ {t = 1} ^ {T} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {i, t}; s _ {i, t}\right), \tag {5}
$$
where  $\hat{q}_{\alpha}^{\theta}$  is an estimator of the current return quantile.
Notice that in contrast to mean-PG, in CVaR optimization the baseline cannot follow an arbitrary critic, but should approximate the total return quantile  $q_{\alpha}^{\theta}(R)$ . Tamar et al. [2015] showed that any baseline  $b \neq q_{\alpha}^{\theta}(R)$  inserts bias to the CVaR gradient estimator, potentially to the level of pointing to the opposite direction (as discussed in Appendix A.1 and Fig. 6). As a result, CVaR-PG methods in RL either are limited to basic REINFORCE with a constant baseline [Greenberg et al., 2022], or use a critic for variance reduction at the cost of biased gradients [Rajeswaran et al., 2017].
Another major source of gradient-variance in CVaR-PG is its reduced sample efficiency: notice that Eq. (5) only exploits  $\approx \alpha N$  trajectories out of each batch of  $N$  trajectories (due to the term  $\mathbf{1}_{R_i\leq \hat{q}_\alpha^\theta}$ ), hence results in estimation variance larger by a factor of  $\alpha^{-1}$ .
4 CVaR Optimization in Meta-Learning
In this section, we show that unlike standard RL, CVaR-PG in MRL permits a flexible baseline without presenting biased gradients. Hence, policy gradients for CVaR objective in MRL is substantially different from both mean-PG in MRL (Eq. (3)) and CVaR-PG in RL (Eq. (4)).
To derive the policy gradient, we first define the policy value per task and the tail of tasks.
Definition 1. The value of policy  $\pi_{\theta}$  in task  $\tau$  is denoted by  $V_{\tau}^{\theta} = \mathbb{E}_{R\sim P_{\tau}^{\theta}}[R]$ . Notice that  $V_{\tau}^{\theta}$  depends on the random variable  $\tau$ . We define the  $\alpha$ -tail of tasks w.r.t.  $\pi_{\theta}$  as the tasks with the lowest values:  $\Omega_{\alpha}^{\theta} = \{z\in \Omega |V_{z}^{\theta}\leq q_{\alpha}(V_{\tau}^{\theta})\}$ .
Assumption 1. To simplify integral calculations, we assume that for any  $z \in \Omega$  and  $\theta \in \Theta$ ,  $R$  is a continuous random variable (i.e., its conditional PDF  $P_z^\theta(x)$  has no atoms). We also assume that  $v(z) = V_z^\theta$  is a continuous function for any  $\theta \in \Theta$ .
Theorem 1 (Meta Policy Gradient for CVaR). Under Assumption 1, the policy gradient of the CVaR objective in Eq. (2) is
$$
\nabla_ {\theta} J _ {\alpha} ^ {\theta} (R) = \int_ {\Omega_ {\alpha} ^ {\theta}} D (z) \int_ {- \infty} ^ {\infty} (x - b) \nabla_ {\theta} P _ {z} ^ {\theta} (x) \cdot d x \cdot d z, \tag {6}
$$
where  $b$  is any arbitrary baseline independent of  $\theta$ .
Proof intuition (the formal proof is in Appendix A). In RL, the CVaR objective measures the  $\alpha$  lowest-return trajectories. When the policy is updated, the cumulative probability of these trajectories changes and no longer equals  $\alpha$ . Thus, the new CVaR calculation must add or remove trajectories (as visualized in Fig. 6 in the appendix). This adds a term in the gradient calculation, which causes the bias in CVaR-PG. By contrast, in MRL, the CVaR measures the  $\alpha$  lowest-return tasks  $\Omega_{\alpha}^{\theta}$ . Since the task distribution does not depend on the policy, the probability of these tasks is not changed - but only the way they are handled by the agent (Fig. 7). Thus, no bias term appears in the calculation. Note that  $\Omega_{\alpha}^{\theta}$  does change throughout the meta-learning - due to changes in task values (rather than task probabilities); this is a different effect and is not associated with gradient bias.
According to Theorem 1, the CVaR PG in MRL permits any baseline  $b$ . As discussed in Section 3, this flexibility is necessary, for example, in any actor-critic framework.
To estimate the gradient from meta-rollouts of the tail tasks, we transform the integration of Eq. (6) into an expectation:
Corollary 1. Eq. (6) can be written as
$$
\nabla_ {\theta} J _ {\alpha} ^ {\theta} (R) = \mathbb {E} _ {\tau \sim D} \left[ \mathbb {E} _ {\Lambda \sim P _ {\tau} ^ {\theta}} [ g (\Lambda) ] \mid V _ {\tau} ^ {\theta} \leq q _ {\alpha} \left(V _ {\tau} ^ {\theta}\right) \right], \tag {7}
$$
where  $g(\Lambda) = (R(\Lambda) - b)\sum_{\substack{1 \leq k \leq K, \\ 1 \leq t \leq T}} \nabla_{\theta} \log \pi_{\theta}(a_{k,t}; \tilde{s}_{k,t})$ ; and  $\tilde{s}_{k,t} = (s_{k,t}, h_{k,t})$  is the extended state (that includes all the task history  $h_{k,t}$  until trajectory  $k$ , step  $t$ ).
Proof. We apply the standard log trick  $\nabla_{\theta}P_{z}^{\theta} = P_{z}^{\theta}\cdot \nabla_{\theta}\log P_{z}^{\theta}$  to Eq. (6), after substituting the meta- rollout PDF:  $P_z^\theta (\Lambda) = \prod_{k = 1}^K\left[P_{0,z}(s_{k,0})\cdot \prod_{t = 1}^T P_z(s_{k,t + 1},r_{k,t}|s_{k,t},a_{k,t})\pi_\theta (a_{k,t};\tilde{s}_{k,t})\right]$ .
For a practical Monte-Carlo estimation of Eq. (7), given a task  $z_{i}$ , we need to estimate whether  $V_{z_i}^\theta \leq q_\alpha(V_\tau^\theta)$ . To estimate  $V_{z_i}^\theta$ , we can generate  $M$  i.i.d meta-rollouts with returns  $\{R_{i,m}\}_{m=1}^M$  and calculate their average return  $\hat{V}_{z_i}^\theta = R_i = \sum_{m=1}^M R_{i,m}/M$ . Then, the quantile  $q_\alpha(V_\tau^\theta)$  can be estimated over a batch of tasks  $\hat{q}_\alpha = q_\alpha(\{R_i\}_{i=1}^N)$ . If  $\hat{V}_{z_i}^\theta \leq \hat{q}_\alpha$ , we use all the meta-rollouts of  $z_i$  for the gradient calculation (including meta-rollouts that by themselves have a higher return  $R_{i,m} > \hat{q}_\alpha$ ). Notice that we use  $M$  i.i.d meta-rollouts, each consisting of  $K$  episodes (the episodes within a meta-rollout are not independent, due to agent memory).
Putting it together, we obtain the sample-based gradient estimator of Eq. (7):
$$
\nabla_ {\theta} J _ {\alpha} ^ {\theta} (R) \approx \frac {1}{\alpha N} \sum_ {i = 1} ^ {N} \mathbf {1} _ {R _ {i} \leq \hat {q} _ {\alpha} ^ {\theta}} \sum_ {m = 1} ^ {M} g _ {i, m}, \tag {8}
$$
$$
g _ {i, m} := (R _ {i, m} - b) \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {T} \nabla_ {\theta} \log \pi_ {\theta} (a _ {i, m, k, t}; \tilde {s} _ {i, m, k, t}),
$$
where  $a_{i,m,k,t}$ ,  $\tilde{s}_{i,m,k,t}$  are the action and the state-and-history at task  $i$ , meta-rollout  $m$ , trajectory  $k$  and step  $t$ .
The procedure described above follows the principles of CVaR-PG in (non-meta) RL, as the learning rule is only applied to the tail of the sampled batch. However, in MRL we consider a batch of tasks rather than a batch of trajectories. As discussed in Theorem 1 and its proof, this distinction has a substantial impact on the gradient and the resulting algorithm. Specifically, Eq. (8) allows for greater flexibility than Eq. (4), as it permits any baseline  $b$  that does not depend on  $\theta$ . This allows gradient calculation using any PG algorithm, including SOTA methods such as PPO [Schulman et al., 2017] (which are already used in MRL methods such as VariBAD Zintgraf et al. [2019]). Therefore, in contrast to standard RL, CVaR optimization is not restricted to basic REINFORCE.
Algorithm 1: CVaR Meta Learning (CVaR-ML)
1 Input: Meta-learning algorithm (Definition 2); robustness level  $\alpha \in (0,1]$ ; task distribution  $D$ ;  $N$  tasks per batch;  $M$  meta-rollouts per task
2 while not finished training do
// Sample tasks  
3 Sample  $\{z_i\}_{i=1}^N \sim D$  // Run meta-rollouts  
4  $\{\{\Lambda_{i,m}\}_{m=1}^M\}_{i=1}^N \leftarrow$  rollouts(\{z_i\}_{i=1}^N, M)  
5  $R_{i,m} \gets$  return( $\Lambda_{i,m}$ ),  $\forall i, m$  // Compute sample quantile  
6  $R_i \gets$  mean(\{R_{i,m}\}_{m=1}^M),  $\forall i$   
7  $\hat{q}_\alpha \gets$  quantile(\{R_i\}_{i=1}^N,  $\alpha$ ) // Meta-learning algorithm train step  
8 ML( $\{\Lambda_{i,m} \mid R_i \leq \hat{q}_\alpha, 1 \leq m \leq M\}$ )
Our CVaR Meta Learning method (CVaR-ML, Algorithm 1) leverages this property to operate as a meta-algorithm, providing a robust version for any given baseline algorithm, such as Finn et al. [2017], Zintgraf et al. [2019]:
Definition 2. A baseline MRL algorithm learns a meta-policy  $\pi_{\theta}$  using a training step ML. Given a batch of meta-rollouts  $\{\Lambda_i\}$ , ML( $\{\Lambda_i\}$ ) updates  $\pi_{\theta}$ .
Notice that CVaR-ML only handles task filtering, and uses the baseline training step ML as a black box (Line 8). Hence, it can be used with any MRL baseline – not just PG methods. In fact, by using a supervised meta-learning baseline, CVaR-ML can be applied to the su
pervised setting as well with minimal modifications, namely, replacing meta-rollouts with examples and returns with losses.
5 Efficient CVaR-ML