Title: Train Hard, Fight Easy: Robust Meta Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta reinforcement learning

robust reinforcement learning

safe reinforcement learning

risk sensitive reinforcement learning
Abstract: A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both **biased gradients** and **data inefficiency**. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and over-sampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.

Train Hard, Fight Easy:

Abstract

A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both biased gradients and data inefficiency . We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm ( RoML ). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and oversampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.

1 Introduction

Reinforcement learning (RL) has achieved impressive results in a variety of applications in recent years, including cooling systems control [Luo et al., 2022] and conversational chatbots [Cohen et al., 2022]. A significant challenge in extending this success to mass production is the variation between instances of the problem, e.g., different cooling systems or different chatbot end-users. Meta-RL (MRL) addresses this challenge by learning a 'meta-policy' that quickly adapts to new tasks [Thrun and Pratt, 1998, Finn et al., 2017]. In the examples above, MRL would maximize the average return of the adapted policy for a new cooling system or a new end-user.

州

T1

1.0

0.8

Driving(,)

Driving(，）

本

0.6

E[return]

业

T2

0.4

0.2

唱

Driving(,)

Driving(，)

0.0

(a)

(b)

However, optimizing the return of the average client might not suffice, as certain clients may still experience low or even negative returns. If 10% of the clients report poor performance, it may deter potential clients from adopting the new technology - even if its average return is high. This highlights

Shie Mannor

Technion, Nvidia Research shie@ee.technion.ac.il

Eli Meirom

Nvidia Research emeirom@nvidia.com

the need for MRL systems that provide robust returns across tasks. Robustness is further motivated by risk sensitivity in many natural RL applications, such as medical treatment [Yu et al., 2021] and driving [Shalev-Shwartz et al., 2016]. For example, as illustrated in Fig. 1, an agent should drive safely at any road profile - even if at some roads the driving would be more cautious than necessary.

A common approach for risk-averse optimization is the max-min objective [Collins et al., 2020]; in MRL, that would mean searching for a meta-policy with the highest expected-return in the worst possible task. This expresses the most extreme risk aversion, which only attends to the one worst case out of all the possible outcomes. Furthermore, in certain problems, worst cases are inevitable (e.g., in certain medical treatments, a fatal outcome cannot be avoided), thus optimizing the minimum return might not provide any meaningful result. A more general objective is the average return over the worst α quantiles ( 0 ≤ α ≤ 1 ), also known as the Conditional Value-at-Risk (CVaR). Notice that the CVaR is a generalization of both the mean (for α = 1 ) and the minimum (for α = 0 ).

CVaR is a coherent risk measure used for risk management in various fields [Filippi et al., 2020], including banking regulation [Acerbi and Szekely, 2014] and RL [Tamar et al., 2015, Hiraoka et al., 2019]. In this work, we extend CVaR optimization to MRL, by replacing the standard MRL objective

with a CVaR objective, which measures the robustness of a policy to high-risk tasks:

In both equations, τ is a random task and R is the random return of policy π θ in τ . Intuitively, the CVaR return expresses robustness to the selected task, in analogy to robustness to the realized model in standard RL. To further motivate Eq. (2), note that CVaR optimization in RL is equivalent to robust optimization under uncertain perturbations [Chow et al., 2015].

In Section 4, we follow the standard approach of policy gradient (PG) for CVaR α optimization in RL, and apply it to MRL. That is, for every batch of N trajectories, we apply the learning step to the αN trajectories with the lowest returns. This standard approach, CVaR-PG, is known to suffer from a major limitation: in an actor-critic framework, the critic leads to a biased gradient estimator to the extent that it may point to the opposite direction [Tamar et al., 2015]. This limitation is quite severe: many CVaR-PG implementations [Tamar et al., 2015, Greenberg et al., 2022] rely on vanilla PG without a critic (REINFORCE, Williams [1992]); others pay the price of gradient bias - in favor of advanced actor-critic methods that reduce the gradient variance [Rajeswaran et al., 2017].

This limitation is particularly concerning in meta RL, where high complexity and noise require more sophisticated algorithms than REINFORCE. Fortunately, Section 4 eliminates this concern: in MRL, in contrast to RL, the CVaR policy gradient is proven to remain unbiased regardless of the choice of critic. Hence, our proposed method - CVaR Meta Learning ( CVaR-ML ) - can be safely applied on top of any MRL algorithm. This makes CVaR-ML a meta-algorithm : given an arbitrary MRL algorithm, CVaR-ML generates a robust version of it.

Nevertheless, in CVaR optimization methods, another source of gradients variance and sample inefficiency is the large proportion of data not being utilized. Every iteration, we rollout trajectories for N tasks, but only use αN of them for training. To mitigate this effect, we introduce in Section 5 the Robust Meta RL algorithm ( RoML ). RoML assumes that tasks can be selected during training. It learns to identify tasks with lower returns and over-samples them. By training on high-risk tasks, the meta-agent learns policies that are robust to them without discarding data. Hence, RoML increases the sample efficiency by a factor of up to α -1 α -1 α -1 . Unlike common adversarial methods, which search for the worst-case sample (task) that minimizes the return [Collins et al., 2020], RoML lets the user specify the desired level of robustness α , and addresses the entire α -tail of the return distribution.

We test our algorithms on several domains. Section 6.1 considers a navigation problem, where both CVaR-ML and RoML obtain better CVaR returns than their risk-neutral baseline. Furthermore, they learn substantially different navigation policies. Section 6.2 considers several continuous control environments with varying tasks. These environments are challenging for CVaR-ML, which entirely fails to learn. Yet, RoML preserves its effectiveness and consistently improves the robustness of the returns. In addition, Section 6.3 demonstrates that under certain conditions, RoML can be applied to supervised settings as well - providing robust supervised meta-learning.

As a meta-algorithm, in each experiment RoML improves the robustness of its baseline algorithm - using the same hyper-parameters as the baseline. The average return is also improved in certain experiments, indicating that even the risk-neutral objective of Eq. (1) may benefit from robustness.

Contribution: (a) We propose a principled CVaR optimization framework for robust meta-RL. While the analogous problem in standard RL suffers from biased gradients and data inefficiency, we (b) prove theoretically that MRL is immune to the former, and (c) address the latter via the novel Robust Meta RL algorithm (RoML). Finally, (d) we demonstrate the robustness of RoML experimentally.

2 Related Work

Meta-RL for the average task is widely researched, including methods based on gradients [Finn et al., 2017, Gupta et al., 2018], latent memory [Zintgraf et al., 2019, Rakelly et al., 2019] and offline meta learning [Dorfman et al., 2020, Pong et al., 2022]. It is used for applications ranging from robotics [Nagabandi et al., 2018] to education [Wu et al., 2021]. Adversarial meta learning was studied for minimax optimization of the lowest-return task , in supervised meta learning [Collins et al., 2020, Goldblum et al., 2020] and MRL [Lin et al., 2020]. Other works studied the robustness of MRL to distributional shifts [Mendonca et al., 2020, Ajay et al., 2022]. However, the CVaR task objective has not been addressed yet in the framework of MRL.

Risk-averse RL. In standard RL, risk awareness is widely studied for both safety [García and Fernández, 2015, Greenberg and Mannor, 2021] and robustness [Derman et al., 2020]. CVaR specifically was studied using PG [Tamar et al., 2015, Rajeswaran et al., 2017, Hiraoka et al., 2019, Huang et al., 2021a], value iteration [Chow et al., 2015] and distributional RL [Dabney et al., 2018, Schubert et al., 2021, Lim and Malik, 2022]. CVaR optimization was also shown equivalent to mean optimization under robustness [Chow et al., 2015], motivating robust-RL methods [Pinto et al., 2017, Godbout et al., 2021]. In this work, we propose a meta-learning framework and algorithms for CVaR optimization, and point to both similarities and differences from the standard RL setting.

Sampling. In Section 5, we use the cross-entropy method [de Boer et al., 2005] to sample high-risk tasks for training. The cross-entropy method has been studied in standard RL for both optimization [Mannor et al., 2003, Huang et al., 2021b] and sampling [Greenberg et al., 2022]. Sampling in RL was also studied for regret minimization in the framework of Unsupervised Environment Design [Dennis et al., 2020, Jiang et al., 2021]; and for accelerated curriculum learning in the framework of Contextual RL [Klink et al., 2020, Eimer et al., 2021]. By contrast, we address MRL (where the current task is unknown to the agent, unlike Contextual RL), and optimize the CVaR risk measure instead of the mean.

3 Preliminaries

MRL. Consider a set of Markov Decision Processes (MDPs) { ( S, A, τ, P τ , P 0 ,τ , γ ) } τ ∈ Ω , where the distribution of transitions and rewards P τ and the initial state distribution P 0 ,τ both depend on task τ ∈ Ω . The task itself is drawn from a distribution τ ∼ D over a general space Ω , and is not known to the agent. The agent can form a belief regarding the current τ based on the task history h , which consists of repeating triplets of states, actions and rewards [Zintgraf et al., 2019]. Thus, the meta-policy π θ ( a ; s, h ) ( θ ∈ Θ ) maps the current state s ∈ S and the history h ∈ ∏ ( S × A × R ) (consisting of state-action-reward triplets) to a probability distribution over actions.

A meta-rollout is defined as a sequence of K ≥ 1 episodes of length T ∈ N over a single task τ : Λ = {{ ( s k,t , a k,t , r k,t ) } T t =1 } K k =1 . For example, in a driving problem, τ might be a geographic area or type of roads, and Λ a sequence of drives on these roads. The return of the agent over a meta-rollout is defined as R (Λ) = 1 K ∑ K k =1 ∑ T t =0 γ t r k,t , where r k,t is the (random variable) reward at step t in episode k . Given a task τ and a meta-policy π θ , we denote by P θ τ ( x ) the conditional PDF of the return R . With a slight abuse of notation, we shall use P θ τ (Λ) to also denote the PDF of the meta-rollout itself. The standard MRL objective is to maximize the expected return J θ ( R ) = E τ,R [ R ] .

While the meta policy π θ ( s, a ; h ) is history-dependent, it can still be learned using standard policy gradient (PG) approaches, by considering h as part of an extended state space ˜ s = ( s, h ) ∈ ˜ S . Then,

the policy gradient can be derived directly:

where D is a probability measure over the task space Ω , P θ z ( x ) is the PDF of R (conditioned on π θ and τ = z ), and b is any arbitrary baseline that is independent of θ [Agrawal, 2019]. While a direct gradient estimation via Monte Carlo sampling is often noisy, its variance can be reduced by an educated choice of baseline b . In the common actor-critic framework [Mnih et al., 2016], a learned value function b = V ( s ; h ) is used. This approach is used in many SOTA algorithms in deep RL, e.g., PPO [Schulman et al., 2017]; and by proxy, in MRL algorithms that rely on them, e.g., VariBAD [Zintgraf et al., 2019].

A major challenge in MRL is the extended state space ˜ S , which now includes the whole task history. Common algorithms handle the task history via a low-dimensional embedding that captures transitions and reward function [Zintgraf et al., 2019]; or using additional optimization steps w.r.t. task history [Finn et al., 2017]. Our work does not compete with such methods, but rather builds upon them: our methods operate as meta-algorithms that run on top of existing MRL baselines.

CVaR-PG. Before moving on to CVaR optimization in MRL, we first recap the common PG approach for standard (non-meta) RL. For a random variable X and α -quantile q α ( X ) , the CVaR is defined as CVaR α ( X ) = E [ X | X ≤ q α ( X )] . For an MDP ( S, A, P, P 0 , γ ) , the CVaR-return objective is ˜ J θ α ( R ) = CVaR α R ∼ P θ [ R ] = ∫ q θ α ( R ) -∞ x · P θ ( x ) · dx , whose corresponding policy gradient is [Tamar et al., 2015]:

Given a sample of N trajectories {{ ( s i,t , a i,t ) } T t =1 } N i =1 with returns { R i } N i =1 , the policy gradient can be estimated by [Tamar et al., 2015, Rajeswaran et al., 2017]:

where ˆ q θ α is an estimator of the current return quantile.

̸

Notice that in contrast to mean-PG, in CVaR optimization the baseline cannot follow an arbitrary critic, but should approximate the total return quantile q θ α ( R ) . Tamar et al. [2015] showed that any baseline b = q θ α ( R ) inserts bias to the CVaR gradient estimator, potentially to the level of pointing to the opposite direction (as discussed in Appendix A.1 and Fig. 6). As a result, CVaR-PG methods in RL either are limited to basic REINFORCE with a constant baseline [Greenberg et al., 2022], or use a critic for variance reduction at the cost of biased gradients [Rajeswaran et al., 2017].

Another major source of gradient-variance in CVaR-PG is its reduced sample efficiency: notice that Eq. (5) only exploits ≈ αN trajectories out of each batch of N trajectories (due to the term 1 R i ≤ ˆ q θ α ), hence results in estimation variance larger by a factor of α -1 .

4 CVaR Optimization in Meta-Learning

In this section, we show that unlike standard RL, CVaR-PG in MRL permits a flexible baseline without presenting biased gradients . Hence, policy gradients for CVaR objective in MRL is substantially different from both mean -PG in MRL (Eq. (3)) and CVaR-PG in RL (Eq. (4)).

To derive the policy gradient, we first define the policy value per task and the tail of tasks.

Definition 1. The value of policy π θ in task τ is denoted by V θ τ = E R ∼ P θ τ [ R ] . Notice that V θ τ depends on the random variable τ . We define the α -tail of tasks w.r.t. π θ as the tasks with the lowest values: Ω θ α = { z ∈ Ω | V θ z ≤ q α ( V θ τ ) } .

Assumption 1. To simplify integral calculations, we assume that for any z ∈ Ω and θ ∈ Θ , R is a continuous random variable (i.e., its conditional PDF P θ z ( x ) has no atoms). We also assume that v ( z ) = V θ z is a continuous function for any θ ∈ Θ .

Theorem 1 (Meta Policy Gradient for CVaR) . Under Assumption 1, the policy gradient of the CVaR objective in Eq. (2) is

where b is any arbitrary baseline independent of θ .

Proof intuition (the formal proof is in Appendix A). In RL, the CVaR objective measures the α lowest-return trajectories. When the policy is updated, the cumulative probability of these trajectories changes and no longer equals α . Thus, the new CVaR calculation must add or remove trajectories (as visualized in Fig. 6 in the appendix). This adds a term in the gradient calculation, which causes the bias in CVaR-PG. By contrast, in MRL, the CVaR measures the α lowest-return tasks Ω θ α . Since the task distribution does not depend on the policy, the probability of these tasks is not changed - but only the way they are handled by the agent (Fig. 7). Thus, no bias term appears in the calculation. Note that Ω θ α does change throughout the meta-learning - due to changes in task values (rather than task probabilities); this is a different effect and is not associated with gradient bias.

According to Theorem 1, the CVaR PG in MRL permits any baseline b . As discussed in Section 3, this flexibility is necessary, for example, in any actor-critic framework.

To estimate the gradient from meta-rollouts of the tail tasks, we transform the integration of Eq. (6) into an expectation:

Corollary 1. Eq. (6) can be written as

where g (Λ) = ( R (Λ) -b ) ∑ 1 ≤ k ≤ K, 1 ≤ t ≤ T ∇ θ log π θ ( a k,t ; ˜ s k,t ) ; and ˜ s k,t = ( s k,t , h k,t ) is the extended state (that includes all the task history h k,t until trajectory k , step t ).

Proof. We apply the standard log trick ∇ θ P θ z = P θ z · ∇ θ log P θ z to Eq. (6), after substituting the metarollout PDF: P θ z (Λ) = ∏ K k =1 [ P 0 ,z ( s k, 0 ) · ∏ T t =1 P z ( s k,t +1 , r k,t | s k,t , a k,t ) π θ ( a k,t ; ˜ s k,t ) ] .

For a practical Monte-Carlo estimation of Eq. (7), given a task z i , we need to estimate whether V θ z i ≤ q α ( V θ τ ) . To estimate V θ z i , we can generate M i.i.d meta-rollouts with returns { R i,m } M m =1 , and calculate their average return ˆ V θ z i = R i = ∑ M m =1 R i,m /M . Then, the quantile q α ( V θ τ ) can be estimated over a batch of tasks ˆ q α = q α ( { R i } N i =1 ) . If ˆ V θ z i ≤ ˆ q α , we use all the meta-rollouts of z i for the gradient calculation (including meta-rollouts that by themselves have a higher return R i,m > ˆ q α ). Notice that we use M i.i.d meta-rollouts, each consisting of K episodes (the episodes within a meta-rollout are not independent, due to agent memory).

Putting it together, we obtain the sample-based gradient estimator of Eq. (7):

where a i,m,k,t , ˜ s i,m,k,t are the action and the state-and-history at task i , meta-rollout m , trajectory k and step t .

The procedure described above follows the principles of CVaR-PG in (non-meta) RL, as the learning rule is only applied to the tail of the sampled batch. However, in MRL we consider a batch of tasks rather than a batch of trajectories. As discussed in Theorem 1 and its proof, this distinction has a substantial impact on the gradient and the resulting algorithm. Specifically, Eq. (8) allows for greater flexibility than Eq. (4), as it permits any baseline b that does not depend on θ . This allows gradient calculation using any PG algorithm, including SOTA methods such as PPO [Schulman et al., 2017] (which are already used in MRL methods such as VariBAD Zintgraf et al. [2019]). Therefore, in contrast to standard RL, CVaR optimization is not restricted to basic REINFORCE.

Algorithm 1: CVaR Meta Learning (CVaR-ML)

1 Input : Meta-learning algorithm (Definition 2); robustness level α ∈ (0 , 1] ; task distribution D ; N tasks per batch; M meta-rollouts per task

2 while not finished training do // Sample tasks

3 Sample { z i } N i =1 ∼ D // Run meta-rollouts

4 {{ Λ i,m } M m =1 } N i =1 ← rollouts ( { z i } N i =1 , M )

5 R i,m ← return (Λ i,m ) , ∀ i, m // Compute sample quantile

6 R i ← mean ( { R i,m } M m =1 ) , ∀ i

7 ˆ q α ← quantile ( { R i } N i =1 , α ) // Meta-learning algorithm train step ( )

8 ML { Λ i,m | R i ≤ ˆ q α , 1 ≤ m ≤ M }

Our CVaR Meta Learning method ( CVaRML , Algorithm 1) leverages this property to operate as a meta-algorithm , providing a robust version for any given baseline algorithm, such as Finn et al. [2017], Zintgraf et al. [2019]:

Definition 2. A baseline MRL algorithm learns a meta-policy π θ using a training step ML . Given a batch of meta-rollouts { Λ i } , ML ( { Λ i } ) updates π θ .

Notice that CVaR-ML only handles task filtering, and uses the baseline training step ML as a black box (Line 8). Hence, it can be used with any MRLbaseline - not just PG methods. In fact, by using a supervised meta-learning baseline, CVaR-ML can be applied to the su- pervised setting as well with minimal modifications, namely, replacing meta-rollouts with examples and returns with losses.

5 Efficient CVaR-ML

Theorem 1 guarantees unbiased gradients when using Algorithm 1; however, it does not bound their variance. In particular, Line 8 applies the learning step to a subset of only αNM meta-rollouts out of NM , which increases the estimator variance by a factor of α -1 compared to mean optimization. This could be prevented if we knew the set Ω θ α of tail-tasks (for the current π θ ), and sampled only these tasks, using the distribution D θ α ( z ) = α -1 1 V θ z ≤ q α ( V θ τ ) D ( z ) . Proposition 1 shows that this would indeed recover the sample efficiency.

Proposition 1 (Variance reduction) . Denote by G the estimator of ∇ θ J θ α ( R ) in Eq. (8), assume there is no quantile error ( ˆ q θ α = q θ α ), and denote E D [ · ] = E z i ∼ D,R i,m ∼ P θ z i [ · ] . Then, switching the task sample distribution to D θ α leads to a variance reduction of factor α :

Proof sketch (the complete proof is in Appendix B). We calculate the expectation and variance directly. G is proportional to 1 R i ≤ q θ α (Eq. (8)). The condition R i ≤ q θ α leads to multiplication by the probability α when sampled from D (where it corresponds to the α -tail); but not when sampled from D θ α (where it is satisfied w.p. 1). This factor α cancels out the ratio between the expectations of G and αG (thus the expectations are identical) - but not the ratio α 2 between their variances.

RobustMeta-RL(RoML)

task value:E[R|t,Ie]

truevalue

task probability: D(t)

X

sampledreturn

Rollouts{A}

qa(value)

original DΦ

Legend

learnedD

Crossentropy

Metalearner:

Data

method:update

Environment

updateθ

Parameters

update

Taskdistribution

Tasks

Meta-policy

Learned

model

DΦ

{t}~DΦ

To

task:tEΩ

Following the motivation of Proposition 1, we wish to increase the number of train tasks that come from the tail distribution Ω θ α . To that end, we assume to have certain control over the sampling of training tasks. This assumption is satisfied in most simulated environments, as well as many real-world scenarios. For example, when training a driver, we choose the tasks, roads and times of driving throughout training. In this section, we propose a method to make these choices.

We begin with parameterizing the task distribution D : we consider a parametric family D ϕ such that D = D ϕ 0 . Then, we wish to modify the parameter ϕ so that D ϕ aligns with D θ α as closely as possible. To that end, we use the Cross Entropy Method (CEM, de Boer et al. [2005]), which searches for ϕ ∗ that minimizes the KL-divergence (i.e., cross entropy) between the two:

where w ( z ) = D ϕ 0 ( z ) D ϕ ( z ) is the importance sampling weight corresponding to z ∼ D ϕ . Note that Eq. (9) has a closed-form solution for most of the standard families of distributions [de Boer et al., 2005].

Algorithm 2: Robust Meta RL (RoML)

1 Input : Meta-learning algorithm (Definition 2); robustness level α ∈ (0 , 1] ; parametric task distribution D ϕ ; original parameter ϕ 0 ; N tasks per batch, ν ∈ [0 , 1) of them sampled from the original D ; CEM quantile β ∈ (0 , 1)

ϕ 0 2 Initialize: 3 ϕ ← ϕ 0 , N o ←⌊ νN ⌋ , N s ←⌈ (1 -ν ) N ⌉ 4 while not finished training do // Sample tasks 5 Sample { z o,i } N o i =1 ∼ D ϕ 0 , { z ϕ,i } N s i =1 ∼ D ϕ 6 z ← ( z o, 1 , . . . , z o,N o , z ϕ, 1 , . . . , z ϕ,N s ) // Rollouts and meta-learning step 7 { Λ i } N i =1 ← rollout ( { z i } N i =1 ) 8 R i ← return (Λ i ) , ∀ i ∈ { 1 , . . . , N } 9 ML ( { Λ i } N i =1 ) // Estimate reference quantile 10 w o,i ← 1 , ∀ i ∈ { 1 , . . . , N o } 11 w ϕ,i ← D ϕ 0 ( z ϕ,i ) D ϕ ( z ϕ,i ) , ∀ i ∈ { 1 , . . . , N s } 12 w ← ( w o, 1 , . . . , w o,N o , w ϕ, 1 , . . . , w ϕ,N s ) 13 ˆ q α ← weighted_quantile ( { R i } , w, α ) // Compute sample quantile 14 q β ← quantile ( { R i } , β ) // Update sampler 15 q ← max(ˆ q α , q β ) 16 ϕ ← argmax ϕ ′ ∑ i ≤ N w i 1 R i ≤ q log D ϕ ′ ( z i )

For a batch of N tasks sampled from D = D ϕ 0 , Eq. (9) essentially chooses the αN tasks with the lowest returns, and updates ϕ to focus on these tasks. This may be noisy unless αN ≫ 1 . Instead, the CEM chooses a larger number of tasks β > α for the update, where β is a hyper-parameter. ϕ is updated according to these βN lowestreturn tasks, and the next batch is sampled from D ϕ = D ϕ 0 . This repeats iteratively: every batch is sampled from D ϕ , where ϕ is updated according to the βN lowestreturn tasks of the former batch. Each task return is also compared to the α -quantile of the original distribution D ϕ 0 . If more than βN tasks yield lower returns, the CEM permits more samples for the update step. The return quantile over D ϕ 0 can be estimated from D ϕ at any point using importance sampling weights. See more details about the CEM in Appendix C.1.

̸

In our problem, the target distribution is the tail of D ϕ 0 . Since the tail is defined by the agent returns in these tasks, it varies with the agent and is non-stationary throughout training. Thus, we use the dynamic-target CEM of Greenberg [2022]. To smooth the changes in the sampled tasks, the sampler is also regularized to always provide certain exposure to all the tasks: we force ν percent of every batch to be sampled from the original distribution D = D ϕ 0 , and only 1 -ν percent from D ϕ .

Putting this together, we obtain the Robust Meta RL algorithm ( RoML ), summarized in Algorithm 2 and Fig. 2. RoML does not require multiple meta-rollouts per update (parameter M in Algorithm 1), since it directly models high-risk tasks. Similarly to CVaR-ML, RoML is a meta-algorithm and can operate on top of any meta-learning baseline (Definition 2). Given the baseline implementation, only the tasks sampling procedure needs to be modified, which makes RoML easy to implement.

Limitations: The CEM's adversarial tasks sampling relies on several assumptions. Future research may reduce some of these assumptions, while keeping the increased data efficiency of RoML.

First, as mentioned above, we need at least partial control over the selection of training tasks. This assumption is common in other RL frameworks [Dennis et al., 2020, Jiang et al., 2021], and often holds in both simulations and the real world (e.g., choosing in which roads and hours to train driving).

Second, the underlying task distribution D is assumed to be known, and the sample distribution is limited to the chosen parameterized family { D ϕ } . For example, if τ ∼ U ([0 , 1]) , the user may choose the family of Beta distributions Beta ( ϕ 1 , ϕ 2 ) (where Beta (1 , 1) ≡ U ([0 , 1]) ), as demonstrated in Appendix D.2. The selected family expresses implicit assumptions on the task-space. For example, if the probability density function is smooth, close tasks will always have similar sample probabilities; and if the family is unimodal, high-risk tasks can only be over-sampled from around a single peak. This approach is useful for generalization across continuous task-spaces - where the CEM can never observe the infinitely many possible tasks. Yet, it may pose limitations in certain discrete task-spaces, if there is no structured relationship between tasks.

6 Experiments

We implement RoML and CVaR-ML on top of two different risk-neutral MRL baselines VariBAD [Zintgraf et al., 2019] and PEARL [Rakelly et al., 2019]. As a risk-averse reference for comparison, we use CeSoR [Greenberg et al., 2022], an efficient sampling-based method for CVaR optimization in RL, implemented on top of PPO. As another reference, we use the Unsupervised Environment Design algorithm PAIRED [Dennis et al., 2020], which uses regret minimization to learn robust policies on a diverse set of tasks.

Section 6.1 demonstrates the mean/CVaR tradeoff, as our methods learn substantially different policies from their baseline. Section 6.2 demonstrates the difficulty of the naive CVaR-ML in more challenging control benchmarks, and the RoML's efficacy in them. The ablation test in Appendix D.4 demonstrates that RoML deteriorates significantly when the CEM is replaced by a naive adversarial sampler. Section 6.3 presents an implementation of CVaR-ML and RoML on top of MAML [Finn et al., 2017] for supervised meta-learning. In all the experiments, the running times of RoML and CVaR-ML are indistinguishable from their baselines (RoML's CEM computations are negligible).

Hyper-parameters: To test the practical applicability of RoML as a meta-algorithm, in every experiment, we use the same hyper-parameters for RoML, CVaR-ML and their baseline . In particular, we use the baseline's default hyper-parameters whenever applicable (Zintgraf et al. [2019], Rakelly et al. [2019] in Section 6.2, and Finn et al. [2017] in Section 6.3). That is, we use the same hyper-parameters as originally tuned for the baseline, and test whether RoML improves the robustness without any further tuning of them. As for the additional hyper-parameters of the metaalgorithm itself: in Algorithm 1, we use M = 1 meta-rollout per task; and in Algorithm 2, we use β = 0 . 2 , ν = 0 unless specified otherwise (similarly to the CEM in Greenberg et al. [2022]). For the references PAIRED and CeSoR, we use the hyper-parameters of Dennis et al. [2020], Greenberg et al. [2022]. Each experiment is repeated for 30 seeds. See more details in Appendix D. The code is available in our repositories: VariBAD, PEARL, CeSoR, PAIRED and MAML.

6.1 Khazad Dum

We demonstrate the tradeoff between mean and CVaR optimization in the Khazad Dum benchmark, visualized in Fig. 3. The agent begins at a random point in the bottom-left part of the map, and has to reach the green target as quickly as possible, without falling into the black abyss. The bridge is not covered and thus is exposed to wind and rain, rendering its floor slippery and creating an additive action noise (Fig. 3b) - to a level that varies with the weather. Each task is characterized by the rain intensity, which is exponentially distributed. The CEM in RoML is allowed to modify the parameter of this exponential distribution. Note that the agent is not aware of the current task (i.e, the weather), but may infer it from observations. We set the target risk level to α = 0 . 01 , and train each meta-agent for a total of 5 · 10 6 frames. See complete details in Appendix D.1.

We implement CVaR-ML and RoML on top of VariBAD. As shown in Fig. 3, VariBAD learns to try the short path - at the risk of rare falls into the abyss. By contrast, our CVaR-optimizing methods take the longer path and avoid risk. This policy increases the cumulative cost of time-steps, but leads to higher CVaR returns, as shown in Fig. 3f. In addition to superior CVaR, RoML also provides competitive average returns in this example (Fig. 3e). Finally, in accordance with Proposition 1, RoML learns significantly faster than CVaR-ML.

validation mean

(a) VariBAD

(b) VariBAD

(c) CVaR-ML

(d) RoML

(e) Mean

(f) CVaR

6

0

10

10

6

6

0

8

10

6

10

0.2

CVaRo.01

0.4

0.5

0.4

0.6

0.6 -

validation

0.7

0.8

VariBAD

0.8

VariBAD

RoML

60-

RoML

1.0

CVaR-ML

1.0

CVaR-ML

0

500

1000

1500

2000

0

500

1000

1500

2000

iteration

iteration

6.2 Continuous Control

We rely on standard continuous control problems from the MuJoCo framework [Todorov et al., 2012]: training a cheetah to run ( HalfCheetah ), and training a Humanoid and an Ant to walk. For each of the 3 environments, we create 3 meta-learning versions: (1) Goal or Vel [Finn et al., 2017], where each task corresponds to a different location or velocity objective, respectively; (2) Mass , where each task corresponds to a different body mass; and (3) Body , where each task corresponds to different mass, head size and physical damping level (similarly to Wang and Van Hoof [2022]). In addition, to experiment with high-dimensional task spaces, we randomly draw 10 numeric variables from env.model in HalfCheetah, and let them vary between tasks. We define 3 such environments with different random sets of task variables ( HalfCheetah 10D-task a,b,c ). For each of the 12 environments above, we set a target risk level of α = 0 . 05 and optimize for K = 2 episodes per task. Additional implementation details are specified in Appendix D.2.

Interestingly, the naive approach of CVaR-ML consistently fails to meta-learn in all the cheetah environments. It remains unsuccessful even after large number of steps, indicating a difficulty beyond sample inefficiency. A possible explanation is the effectively decreased batch size of CVaR-ML. PAIRED and CeSoR also fail to adjust to the MRL environments, and obtain poor CVaR returns.

RoML , on the other hand, consistently improves the CVaR returns (Table 1) compared to its baseline (VariBAD or PEARL), while using the same hyper-parameters as the baseline. The VariBAD baseline presents better returns and running times than PEARL on HalfCheetah, and thus is used for the 6 extended environments (Humanoid and Ant). RoML improves the CVaR return in comparison to the baseline algorithm in all the 18 experiments (6 with PEARL and 12 with VariBAD).

In 5 out of 18 experiments, RoML slightly improves the average return compared to its baseline, and not only the CVaR (Table 2 in the appendix). This indicates that low-return tasks can sometimes be improved at the cost of high-return tasks, but without hurting average performance. In addition, this may indicate that over-sampling difficult tasks forms a helpful learning curriculum.

The robustness of RoML to the selected task is demonstrated in Fig. 4d. In multi-dimensional task spaces, RoML learns to focus the sampling modification on the high-impact variables, as demonstrated

(c) Hum-Mass

(RoML)

(d) HalfCheetah-Mass

1800

return

1600

1400

VariBAD（1558）

RoML(1581)

1200

mass

in Fig. 11 in the appendix. Finally, qualitative inspection shows that RoML learns to handle larger masses, for example, by leaning forward and letting gravity pull the agent forward, as displayed in Fig. 4c (see animations on GitHub).

6.3 Beyond RL: Robust Supervised Meta-Learning

Our work focuses on robustness in MRL. However, the concept of training on harder data to improve robustness, as embodied in RoML, is applicable beyond the scope of RL. As a preliminary proof-of-concept, we apply RoML to a toy supervised meta-learning problem of sine regression, based on Finn et al. [2017]: The input is x ∈ [0 , 2 π ) , the desired output is y = A sin( ωx + b ) , and the task is defined by the parameters τ = ( A,b, ω ) . Similarly to Finn et al. [2017], the model is fine-tuned for each task via a gradient-descent optimization step over 10 samples { ( x i , y i ) } 10 i =1 , and is tested on another set of 10 samples. The goal is to find model weights that adapt quickly to new task data.

We implement CVaR-ML and RoML on top of MAML [Finn et al., 2017]. As shown in Fig. 5, RoML achieves better CVaR losses over tasks than both CVaR-ML and MAML. The complete setting and results are presented in Appendix E.

7 Summary and Future Work

We defined a robust MRL objective and derived the CVaR-ML algorithm to optimize it. In contrast to its analogous algorithm in standard RL, we proved that CVaR-ML does not present biased gradients, yet it does inherit the same data inefficiency. To address the latter, we introduced RoML and demonstrated its advantage in sample efficiency and CVaR return.

Future research may address the CEM-related limitations of RoML discussed at the end of Section 5. Another direction for future work is extension of RoML to other scenarios, especially where a natural task structure can be leveraged to improve task robustness, e.g., in supervised learning and coordinate-based regression [Tancik et al., 2020].

Finally, RoML is easy to implement, operates agnostically as a meta-algorithm on top of existing MRL methods, and can be set to any desired robustness level. We believe that these properties, along with our empirical results, make RoML a promising candidate for MRL in risk-sensitive applications.

Acknowledgements: This work has received funding from the European Union's Horizon Europe Programme, under grant number 101070568.

References

Carlo Acerbi and Balazs Szekely. Back-testing expected shortfall. Risk, 27(11):76-81, 2014.

Shipra Agrawal. Reinforcement learning: Policy gradient methods, 2019. URL https://ieor8100. github.io/rl/docs/Lecture%205%20-%20policy%20gradients.pdf .

Anurag Ajay, Abhishek Gupta, Dibya Ghosh, Sergey Levine, and Pulkit Agrawal. Distributionally adaptive meta reinforcement learning. arXiv preprint arXiv:2210.03104, 2022.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: a cvar optimization approach. Advances in neural information processing systems, 28, 2015.

Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, and Gal Elidan. Dynamic planning in open-ended dialogue using reinforcement learning, 2022. URL https://arxiv.org/abs/ 2208.02294 .

Liam Collins, Aryan Mokhtari, and Sanjay Shakkottai. Task-robust model-agnostic meta-learning. Advances in Neural Information Processing Systems, 33:18860-18871, 2020.

Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pages 1096-1105, 2018.

T. de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134:19-67, 2005.

Tito Homem de Mello and Reuven Y. Rubinstein. Rare event estimation for static models via cross-entropy and importance sampling, 2003.

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049-13061, 2020.

Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A bayesian approach to robust reinforcement learning. In Uncertainty in Artificial Intelligence, pages 648-658. PMLR, 2020.

Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02598, 2020.

Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-paced context evaluation for contextual reinforcement learning. In International Conference on Machine Learning, pages 2948-2958. PMLR, 2021.

Carlo Filippi, Gianfranco Guastaroba, and Maria Grazia Speranza. Conditional value-at-risk beyond finance: a survey. International Transactions in Operational Research, 27(3):1277-1319, 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.

Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015.

Mathieu Godbout, Maxime Heuillet, Sharath Chandra, Rupali Bhati, and Audrey Durand. CARL: Conditional-value-at-risk adversarial reinforcement learning. arXiv preprint arXiv:2109.09470, 2021.

Micah Goldblum, Liam Fowl, and Tom Goldstein. Adversarially robust few-shot learning: A meta-learning approach. Advances in Neural Information Processing Systems, 33:17886-17895, 2020.

Ido Greenberg. Cross entropy method with non-stationary score function. https://pypi.org/ project/cross-entropy-method/ , 2022.

Ido Greenberg and Shie Mannor. Detecting rewards deterioration in episodic reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 3842-3853. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ greenberg21a.html .

Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 2022. URL https: //arxiv.org/abs/2205.05138 .

Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Metareinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.

Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learning robust options by conditional value at risk optimization. NeurIPS, 05 2019.

Audrey Huang, Liu Leqi, Zachary C. Lipton, and Kamyar Azizzadenesheli. On the convergence and optimality of policy gradient for markov coherent risk, 2021a. URL https://arxiv.org/abs/ 2103.02827 .

Kevin Huang, Sahin Lale, Ugo Rosolia, Yuanyuan Shi, and Anima Anandkumar. CEM-GD: Crossentropy method with gradient descent planner for model-based reinforcement learning, 2021b.

Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1897, 2021.

Pascal Klink, Carlo D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33:9216-9227, 2020.

Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 2022.

Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial metareinforcement learning. Advances in Neural Information Processing Systems, 33:10161-10173, 2020.

Jerry Luo, Cosmin Paduraru, Octavian Voicu, Yuri Chervonyi, Scott Munns, Jerry Li, Crystal Qian, Praneet Dutta, Jared Quincy Davis, Ningjia Wu, Xingwei Yang, Chu-Ming Chang, Ted Li, Rob Rose, Mingyan Fan, Hootan Nakhost, Tinglin Liu, Brian Kirkman, Frank Altamura, Lee Cline, Patrick Tonker, Joel Gouker, Dave Uden, Warren Buddy Bryan, Jason Law, Deeni Fatiha, Neil Satra, Juliet Rothenberg, Mandeep Waraich, Molly Carlin, Satish Tallapaka, Sims Witherspoon, David Parish, Peter Dolan, Chenyu Zhao, and Daniel J. Mankowitz. Controlling commercial cooling systems using reinforcement learning, 2022. URL https://arxiv.org/abs/2211.07357 .

Shie Mannor, Reuven Rubinstein, and Yohai Gat. The cross entropy method for fast policy search. Proceedings, Twentieth International Conference on Machine Learning, 2, 07 2003.

Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint arXiv:2006.07178, 2020.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv:1803.11347, 2018.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 2817-2826. JMLR.org, 2017.

Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforcement learning with online self-supervision. In International Conference on Machine Learning, pages 17811-17829. PMLR, 2022.

Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. ICLR, 2017.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.

Frederik Schubert, Theresa Eimer, Bodo Rosenhahn, and Marius Lindauer. Automatic risk adaptation in distributional reinforcement learning. arXiv preprint arXiv:2106.06317, 2021.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.

Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. AAAI'15, page 2993-2999, 2015.

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations, 2020. URL https://arxiv.org/abs/2012.02189 .

Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.

R. R. Tolkien. The Lord of the Rings: The Fellowship of the Ring. George Allen and Unwin, 1954.

Anthony Tromba and Jerrold E Marsden. Vector calculus. Freeman, 1996.

Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pages 23055-23077. PMLR, 2022.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256, 1992.

Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. Prototransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107.14035, 2021.

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representations, 2019.

Contents

1 Introduction 1

2 Related Work 3

3 Preliminaries 3

4 CVaR Optimization in Meta-Learning 4

5 Efficient CVaR-ML 6

6 Experiments 8

7 Summary and Future Work 10

A Policy Gradient for CVaR Optimization in Meta RL 15

B Proof of Proposition 1 18

C The Cross Entropy Method 19

D Experiments: Detailed Settings and Results 19

E Supervised Meta-Learning 24

A Policy Gradient for CVaR Optimization in Meta RL

In this section we provide the complete proof for Theorem 1. For completeness, Appendix A.1 recaps of the analogous proof in standard (non-meta) RL, before moving on to the proof in Appendix A.2. This allows us to highlight the differences between the two.

The substantial difference is that in RL, the CVaR is defined directly over the low-return trajectories, and the policy parameter θ affects the probability of each trajectory in the tail (Eq. (10)). In MRL (Eq. (2)), on the other hand, the CVaR is defined over the low-return tasks, whose probability is not affected directly by θ (Eq. (12)). This allows a decoupling between θ and τ , which results in Theorem 1.

Another high-level intuition is as follows. In RL, the CVaR-PG gradient is invariant to successful strategies, hence must be explicitly negative for the unsuccessful ones (in order not to encourage them, see Fig. 6). In MRL, within the tasks of interest, the gradient always encourages successful strategies on account of the unsuccessful ones (Fig. 7).

A.1 Recap: PG for CVaR in (non-meta) RL

We briefly recap the calculation of Propositions 1 and 2 in Tamar et al. [2015] for CVaR policy gradient under the standard RL settings.

Definition 3 (CVaR return in (non-meta) RL) . Consider an MDP ( S, A, P, γ, P 0 ) with the cumulative reward (i.e., return) R ∼ P θ , whose α -quantile is q θ α ( R ) . Recall the CVaR objective defined in Section 3:

To calculate the policy gradient of ˜ J θ α ( R ) , we begin with the conservation of probability mass below the quantile q θ α :

Then, using the Leibniz integral rule, we have

Notice that as a particular consequence of the conservation rule Eq. (10), positive gradients of P θ ( x ) cause the quantile q θ α ( R ) to decrease. This phenomenon makes the CVaR policy gradient sensitive to the selection of baseline, as visualized in Fig. 6b. In fact, the quantile q θ α ( R ) itself is the only baseline that permits unbiased gradients:

which gives us Eq. (4).

A.2 PG for CVaR in Meta RL

We now turn to PG for CVaR optimization in MRL . We rely on the definitions and notations of Section 3, as well as Definition 1 and Assumption 1 in Section 4. Notice that J θ α ( R ) of Eq. (2) can be

(a) Mean PG

(b) CVaR PG, Eq. (4)

Pπ(R)

Pπ(R)

R

R

baseline

baseline

Pπ(R)

q

Pπ(R)

q

R

R

+baseline

baseline

written in integral form as

In Eq. (11) above, the boundary of the integral over the α -tail is simply the scalar q θ α . In MRL, this is replaced by the boundary of the set Ω θ α , defined in a general topological space. Thus, we begin by characterizing this boundary.

Lemma 1 (The boundary of Ω θ α ) . Under Assumption 1, ∀ z ∈ ∂ Ω θ α : ∫ ∞ -∞ x · P θ z ( x ) · dx = q α ( V θ τ ) .

Proof. Since v ( z ) = V θ z is a continuous function between topological spaces, by denoting B = ( -∞ , q α ( V θ τ )] we have

Finally, we can prove Theorem 1.

Proof of Theorem 1. First, we consider the conservation of probability:

The gradient of this integral can be calculated using a high-dimensional generalization of the Leibniz integral rule, named Reynolds Transport Theorem (RTT, Tromba and Marsden [1996]):

where n ( z, θ ) is the outward-pointing unit vector that is normal to the surface ∂ Ω θ α , and v b ( z, θ ) is the velocity of the area element on the surface with area dA . Notice that ∇ θ D ( z ) ≡ 0 in the last

Meta PG (CVaR,

Meta PG (Mean)

Meta PG (CVaR)

shifted baseline)

PT (R)

PT (R)

PT (R)

task T1

task T1

taskT1

R

R

R

baseline

(?)

()

()

task T2

t2

task T2

task T2

R

R

R

baseline

P (R)

(R)

PT(R)

T3

taskT3

taskT3

taskT3

R

R

R

baseline

baseline

baseline

equality expresses a substantial difference from the standard RL settings in Eq. (10): there, we have ∇ θ P θ ( x ) , which does not necessarily vanish.

Next, we turn to the meta policy gradient itself, again using Reynolds Transport Theorem:

Finally, we show that any θ -independent additive baseline (highlighted in the equation) does not change the gradient calculation:

which completes the proof. Notice that we used the identity ∇ θ ∫ ∞ -∞ P θ z ( x ) dx = ∇ θ 1 = 0 ; this does not hold for the analogous term in the standard RL settings in Eq. (10), ∇ θ ∫ q θ α ( x ) -∞ P θ ( x ) dx , whose gradient depends on ∇ θ q θ α ( x ) according to the Leibniz integral rule.

B Proof of Proposition 1

Proof. Recall that by Eq. (8), G = 1 αN ∑ N i =1 1 R i ≤ ˆ q θ α ∑ M m =1 g i,m . Denoting G i = ∑ M m =1 g i,m and substituting ˆ q θ α = q θ α , we have

Expectation: Since { G i } are i.i.d, and using the law of total probability, we obtain

By switching the task sample distribution to D θ α , and using the definition of D θ α , we simply have

Together, we obtain E D θ α [ αG ] = E D [ G ] as required.

Variance: For the original distribution, since { G i } are i.i.d, we have

For the tail distribution D θ α , however,

which completes the proof.

C The Cross Entropy Method

C.1 Background

The Cross Entropy Method (CEM, de Boer et al. [2005]) is a general approach to rare-event sampling and optimization. In this work, we use its sampling version to sample high-risk tasks from the tail of D . As described in Section 5, the CEM repeatedly samples from the parameterized distribution D ϕ , and updates ϕ according to the β -tail of the sampled batch. Since every iteration focuses on the tail of its former, we intuitively expect exponential convergence to the tail of the original distribution. While theoretical convergence analysis does not guarantee the exponential rate [de Mello and Rubinstein, 2003], practically, the CEM often converges within several iterations. For clarity, we provide the pseudo-code for the basic CEM in Algorithm 3. In this version, the CEM repeatedly generates samples from the tail of the given distribution D ϕ 0 .

Algorithm 3: The Cross Entropy Method (CEM)

C.2 Discussion

The CEM is the key to the flexible robustness level of RoML (Algorithm 2): it can learn to sample not only the single worst-case task, but all the α tasks with the lowest returns.

The CEM searches for a task distribution within a parametric family of distributions. This approach can handle infinite task spaces, and learn the difficulty of tasks in the entire task space from a mere finite sample of tasks. For example, assume that the tasks correspond to environment parameters that take continuous values within some bounded box (as in Section 6.2 and Section 6.3). The CEM can fit a distribution over an entire subset of the box - from a mere finite batch of tasks. This property lets the CEM learn the high-risk tasks quickly and accelerates the meta-training, as demonstrated in Section 6 and Appendix D.3.

On the other hand, this approach relies on the structure in the task space. If the tasks do not have a natural structure like the ones in the bounded box, it is not trivial to define the parametric family of distributions. This is the case in certain supervised meta learning problems. For example, in the common meta-classification problem [Finn et al., 2017], the task space consists of subsets of classes to classify. This is a discrete space without a trivial metric between tasks. Hence, it is difficult for the CEM to learn the α lowest-return tasks from a finite sample. Thus, while RoML is applicable to supervised meta learning as well as MRL, certain task spaces require further adjustment, such as a meaningful embedding of the task space. This challenge is left for future work.

D Experiments: Detailed Settings and Results

D.1 Khazad Dum

At the end of the hall the floor vanished and fell to an unknown depth. The outer door could only be reached by a slender bridge of stone, without kerb or rail, that spanned the chasm with one curving spring of fifty feet. It was an ancient defence of the Dwarves against any enemy that might capture the First Hall and the outer passages. They could only pass across it in single file. [Tolkien, 1954]

The detailed settings of the Khazad Dum environment presented in Section 6.1 are as follows. Every task is carried for K = 4 episodes of T = 32 times steps. The return corresponds to the undiscounted sum of the rewards ( γ = 1 ). Every time step has a cost of 1 /T points if the L1-distance of the agent from the target is larger than 5; and for distances between 0 and 5, the cost varies linearly between 0 and 1 /T . By reaching the destination, the agent obtains a reward of 5 /T , and has no more costs for the rest of the episode. By falling to the abyss, the agent can no longer reach the goal and is bound to suffer a cost of 1 /T for every step until the end of the episode. Every step, the agent observes its location (represented using a soft one-hot encoding, similarly to Greenberg et al. [2022]) and chooses whether to move left, right, up or down. If the agent attempts to move into a wall, it remains in place.

The tasks are characterized by the rain intensity, distributed τ ∼ Exp (0 . 1) . The rain only affects the episode when the agent crosses the bridge: then, the agent suffers from an additive normallydistributed action noise N (0 , τ 2 ) , in addition to a direct damage translated into a cost of 3 · τ . For RoML, the CEM is implemented over the exponential family of distributions Exp ( ϕ ) with ϕ 0 = 0 . 1 and β = 0 . 05 . In this toy benchmark we use no regularization ( ν = 0 ).

In addition to the test returns throughout meta-training shown in Fig. 3, Fig. 8 displays the final test returns at the end of the meta-training, over 30 seeds and 3000 test tasks per seed.

KhazadDum

KhazadDum

0.0

0.0

return

return

-0.1

0.2

-0.2

CVaRo.01

mean

0.4

-0.3

-0.6

-0.4

VariBAD

RoML

CVaR-ML

VariBAD

RoML

CVaR-ML

method

method

D.2 Continuous Control

In all the MuJoCo benchmarks introduced in Section 6.2, each task's meta-rollout consists of 2 episodes × 200 time-steps per episode. Below we describe the task distributions, as well as their parameterization for the CEM. In each benchmark, we used CEM quantile β = 0 . 2 and regularization of ν = 0 . 2 samples per batch.

HalfCheetah-Vel : The original task distribution is uniform τ ∼ U ([0 , 7]) in HalfCheetah (the task space [0 , 7] was extended in comparison to Zintgraf et al. [2019], to create a more significant tradeoff between tasks). We rewrite it as τ = 7˜ τ, ˜ τ ∼ Beta (2 ϕ, 2 -2 ϕ ) with ϕ 0 = 0 . 5 (leading to Beta (1 , 1) , which is indeed the uniform distribution). The CEM learns to modify ϕ . Notice that this parameterization satisfies E D ϕ [˜ τ ] = ϕ .

Humanoid-Vel : Same as HalfCheetah-Vel, with task space [0 , 2 . 5] instead of [0 , 7] .

Ant-Goal : The target location is random within a circle of radius 5. We represent the target location in polar coordinates, and write r ∼ Beta (2 ϕ 1 , 2 -2 ϕ 1 ) and θ ∼ Beta (2 ϕ 2 , 2 -2 ϕ 2 ) (up to multiplicative factors 5 and 2 π ). The original distribution parameter is ϕ 0 = (0 . 5 , 0 . 5) , and the CEM learns to modify it.

HalfCheetah-Mass, Humanoid-Mass, Ant-Mass : The task τ ∈ [0 . 5 , 2] corresponds to the multiplicative factor of the body mass (e.g., τ = 2 is a doubled mass). The original task distribution is uniform over the log factor, i.e., log 2 τ ∼ U ([ -1 , 1]) . Again, we re-parameterize the uniform distribution as Beta , and learn to modify its parameter.

HalfCheetah-Body, Humanoid-Body, Ant-Body : The 3 components of the task correspond to multiplicative factors of different physical properties, and each of them is distributed independently and uniformly in log, i.e., ∀ 1 ≤ j ≤ 3 : log 2 τ j ∼ U ([ -1 , 1]) . We re-parameterize this as 3 independent Beta distributions with parameters ϕ = ( ϕ 1 , ϕ 2 , ϕ 3 ) .

HalfCheetah 10D-task : Again, the task components correspond to multiplicative factors of different properties of the model. This time, there are 10 different properties (i.e., the task space is 10-dimensional), but each of them varies in a smaller range: log 2 τ j ∼

U ([ -0 . 5 , 0 . 5]) . Each such property is a vector, and is multiplied by τ j when executing the task τ . The 10 properties are selected randomly, among all the variables of type float ndarray in env.model . We generate 3 such MRL environments - with 3 different random sets of 10 task-variables each. Some examples for properties are inertia, friction and mass.

In the experiments, we rely on the official implementations of VariBAD [Zintgraf et al., 2019] and PEARL [Rakelly et al., 2019], both published under the MIT license. CVaR-ML and RoML are implemented on top of these baseline, and their running times are indistinguishable from the baselines. All experiments were performed on machines with Intel Xeon 2.2 GHZ CPU and NVIDIA's V100 GPU. Each experiment (meta-training and testing) required 12-72 hours, depending on the environment and the baseline algorithm.

Table 2 and Fig. 9 present detailed results for our MuJoCo experiments, in addition to the results presented in Section 6.2.

(a) HalfCheetah-Vel

(b) HalfCheetah-Body (mass, damping and head-size)

(d) Humanoid-Body (mass, damping and head-size)

(e) Humanoid-Mass

HalfCheetahVel

HalfCheetahBody

HalfCheetahBody

HalfCheetahBody

2000

return

-50-

return

1750 -

return

return

1625

RoML (1582)

VariBAD(1616)

-100

1500

VariBAD(1616)

1500

VariBAD(1616)

1600

150

VariBAD(-82)

1250

RoML (1582)

RoML (1582)

1575

RoML (-95)

mass

damping

head_size

HumanoidVel

HumanoidBody

HumanoidBody

HumanoidBody

900

1800

1700

1675

return

850

VariBAD(880)

return

1600

VariBAD(1678)

return

1600

VariBAD(1678)

return

1650

RoML (1618)

VariBAD (1678)

RoML(883)

1400

RoML (1618)

RoML (1618)

1625

v_goal

mass

damping

head_size

AntMass

AntGoal

AntGoal

1800

1480

-100

return

1600

return

1460

VariBAD(1473)

return

-200 -

RoML (-224)

VariBAD (-228)

return

-220

VariBAD(-228)

1400

RoML(1580)

RoML (1475)

00-

RoML (-224)

-240

12

theta

mass

mass

D.3 The Cross Entropy Method

We implemented RoML using the Dynamic Cross Entropy Method implementation of Greenberg [2022]. Below we concentrate results related to the CEM functionality in all the experiments:

Learned sample distribution : One set of figures corresponds to the learned sample distribution, as measured via ϕ throughout the meta-training.

Sample returns : A second set of figures corresponds to the returns over the sampled tasks (corresponding to D ϕ ): ideally, we would like them to align with the α -tail of the reference returns (corresponding to D ϕ 0 ). Thus, the figures present the mean sample return along with the mean and CVaR reference returns (the references are estimated from the sample returns using Importance Sampling weights). In most figures, we see that the sample returns (in green) shift significantly from the mean reference return (in blue) towards the CVaR reference return (orange), at least for part of the training. Note that in certain environments, the distinction between difficulty of tasks can only be made after the agent has already learned a basic meaningful policy, hence the sample returns do not immediately deviate from the mean reference.

(a) Learned sample distribution

(b) Sample returns

KhazadDum

KhazadDum

0.2

E[task noise]

0.5

0.4

0.4

etur

0.6

Referencemean

0.3

0.8

Reference CVaR1%

0.2

1.0

Samplemean

0.1

0

10

20

30

40

50

60

0

10

20

30

40

50

60

CEMiteration

iteration

E[v_goal] (smoothed)

HalfCheetahVel

(smoothed)

2.00

1.75

HalfCheetahMass

E[task] (smoothed)

2.00

1.75

HalfCheetahBody

(smoothed)

2.00

1.75

HumanoidMass

1.50

1.50

1.50

MMwhMmm

1.25

1.25

1.25

[s9]

0.75

1.00

0.50

0.50

1.00

0.75

mass

damping

E[mass]

1.00

0.75

0.50

1

0.25

0.25

head_size

0.25

0

0

100

200

CE iteration

300

400

500

009

00°0

0

100

200

CE iteration

300

400

500

600

0.00

0

200

400

CE iteration

600

800

1000

1200

0.00

0

100

200

CE iteration

300

400

500

009

(a) HalfCheetah-Vel

(b) HalfCheetah-Mass

(c) HalfCheetah-Body

(d) Humanoid-Mass

(a) HalfCheetah-Vel

(b) HalfCheetah-Mass

(c) HalfCheetah-Body

(d) Humanoid-Mass

HalfCheetahVel

HalfCheetahMass

HalfCheetahBody

HumanoidMass

1500

1600

200

1500

1250

1400

400

1250

1000

1200

-1000

1400

1200

=800

600

0

100

200

300

Reference CVaRs%

Sample mean

Reference mean

400

500

600

eturn

250

1000

500

250

750

0

0

100

200

300

Reference CVaRs%

Sample mean

Reference mean

400

500

600

250

250

750

500

0

0

200

400

600

ReferenceCVaRs%

Sample mean

Reference mean

800

1000

1200

?

1000

800

600

400

200

0

0

100

200

300

Reference CVaRs%

Sample mean

Reference mean

400

500

600

iteration

iteration

iteration

iteration

mean amplitude

5

frequency

3.0

0

4

mean phase

5

2.5

SSO|-=）

S

m

4

2.0

10

2

3

1

mean

1.0

0.5

1.5

S

15

20

Reference mean(IS)

Sample mean

ReferenceCVaRs%(IS)

25

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

CEM iteration

CEMiteration

CEM iteration

CEM iteration

D.4 Ablation Test: RoML with a Naive Sampler

RoML learns to be robust to high-risk tasks by using the CEM to over-sample them throughout training. The dynamic CEM presented in Section 5 is designed to identify and sample high-risk tasks from a possibly-infinite task-space, where the task values change with the evolving policy throughout training.

In this section, we conduct an ablation test to demonstrate the importance of the CEM to this task. To that end, we implement a naive adversarial task sampler. The first M = 100 tasks are sampled from the original distribution D , and the naive sampler memorizes the α M lowest-return tasks, and samples randomly from them for the rest of the training.

As displayed in Table 3, switching to the naive sampler decreases the CVaR returns significantly.

E Supervised Meta-Learning

Below we provide the complete details for the toy Sine Regression experiment of Section 6.3. The input in the problem is x ∈ [0 , 2 π ) , the desired output is y = A sin( ωx + b ) , and the task is defined by the parameters τ = ( A,b, ω ) , distributed uniformly over Ω = [0 . 1 , 5] × [0 , 2 π ] × [0 . 3 , 3] . Similarly to Finn et al. [2017], the model is fine-tuned for each task via a gradient-descent optimization step over 10 samples { ( x i , y i ) } 10 i =1 , and is tested on another set of 10 samples. The goal is to find model weights that adapt quickly to new task data.

CVaR-ML and RoML are implemented with robustness level of α = 0 . 05 , on top of MAML. For the CEM of RoML, we re-parameterize the uniform task distribution using Beta distributions (see CEM details below).

As shown in Fig. 13, RoML learns to focus on tasks (sine functions) with high amplitudes and slightly increased frequencies, without changing the phase distribution. Fig. 14 displays the test losses over 30 seeds, after meta-training for 10000 tasks. Similarly to the MRL experiments, again RoML achieves better CVaR losses than both CVaR-ML and the baseline.

(a)

Mean

(b)

CVaR

0

.

05

model

model

S

4.0

MAML

RoML

CVaR_ML

/aRo.05

S

S

13

MAML

RoML

CVaR_ML

mean

3.5

12

3.0

CVa

11

2.5

0

1

2

3

4

0

1

2

3

4

tuning iteration

tuning iteration

CEMimplementation details: In comparison to Finn et al. [2017], we added the sine frequency as a third parameter in the task space, since the original problem was to simplistic to pose a mean/CVaR tradeoff. The tasks are distributed uniformly, and we reparameterize them for the CEM using the Beta distribution, similarly to Appendix D.2: τ = { τ j } 3 j =1 , τ j ∼ Beta (2 ϕ j , 2 -2 ϕ j ) . On top of this, we add a linear transformation from the Beta distribution domain [0 , 1] to the actual task range ( [0 . 1 , 5] for amplitude, [0 , 2 π ] for phase and [0 . 3 , 3] for frequency). Note that the original uniform distribution is recovered by ϕ 0 = (0 . 5 , 0 . 5 , 0 . 5) . The parameter ϕ j ∈ [0 , 1] , which is controlled by the CEM, equals the expected task E τ ∼ D ϕ [ τ j ] . The other hyper-parameters of RoML are set to β = 0 . 2 (CEM quantile) and ν = 0 (no regularization).

Peer reviews:
comments: SUMMARY:
This is an interesting paper that harnesses a risk-averse optimization from RL and applies the trick to Meta RL (called RoML). It has a theory section that proves that biased gradients in ordinary robust RL objectives are no longer biased when applied to meta RL, an interesting result. there is also a data inefficiency problem that arises in robust RL objectives because out of a batch of N trajectories, they can only make use of a small proportion (alpha) of them. The authors importance sample the set of tail tasks, recovering sample efficiency. Altogether, the authors discuss several key problems that are present in vanilla RL which disappear in meta RL. 

The authors demonstrate experimentally on a variety of tasks the benefits of RoML over baselines.

STRENGTHS:
This paper discovers the cool trick that whereas CVaR has problems associated with its biased estimate of gradients in ordinary RL, these problems are resolved in Meta RL, which is a cool finding backed by theory. The intuition is easy to understand. The experiments are solid.

WEAKNESSES:
I have no concerns. The authors demonstrate strong performance on a variety of tasks, demonstrating its versatility.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the meta-reinforcement learning setting, where the agent learns a meta-policy that adapts to new tasks with few observations. However, traditional meta-RL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty. This may make the agent fail to generalize to risk tasks. In this paper, the authors introduce CVaR into the meta RL framework to solve this problem. They theoretically prove that an unbiased gradient estimator is available in the meta RL setting, which allows a general critic module. They then propose the Robust Meta RL algorithm (RoML) algorithm to generate a robust version of any given MRL algorithm, by identifying and oversampling harder tasks throughout training. Experiments on the gym locomotion tasks demonstrate the advantages of the proposed method.\n\nSTRENGTHS:\n1.\tThis paper is well-written and easy to follow. The figures are clear and illustrative for understanding the idea.\n2.\tThe theoretical analysis shows evidence to support the advantages of the proposed method.\n3.\tThe experiments also support the claim that the proposed method achieves better adaptation to unseen tasks.\n\nWEAKNESSES:\n1.\tI am a little confused about the setting of robust meta-reinforcement learning. What does robustness mean in meta RL? Is there any disturbance to the dynamic system or observation? It seems that the setting mentioned in this paper is that some tasks are harder than others.\n2.\tThe main contribution of this paper is extending the CVaR-PG framework to meta RL setting and eliminating the limitation of the biased gradient estimator. However, the intuition behind the difference between using CVaR in RL and meta-RL settings is not clear. If the task space is not discretized (as mentioned in the caption of Figure 1), the boundary of RL and meta RL will be unclear. So we can still have both easy and hard situations in the RL setting and assume there is an unknown distribution of the difficulty. I think the authors should give some hints in the introduction section about why using CVaR in meta RL is better than in RL. Does the advantage come from the discretized task indicator?\n3.\tThe experimental evaluation is not thorough enough. First, the experiments do not use standard meta-RL benchmarks, such as Meta World. Second, the experimental setting is not consistent with the motivation “Standard MRL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty.” There are no particular high-risk or difficult tasks in the environments （a high-risk task is defined as the high mass in the mujoco environment）. The example in Figure 1 looks good but is not mentioned in the experiment part. Third, potential simple baselines are missing. For example, oversampling the hard task according to the return. This could be a general trick for all meta-learning methods, so I think it should be a fair baseline.\n\nQUESTIONS:\nPoints 1 and 2 of the weaknesses are my questions.\n\nLIMITATIONS:\nThe limitation of the proposed method is discussed in the conclusion section. The algorithm requires control over the selection of training tasks. This assumption is acceptable since it is possible to define and control different tasks in both simulation and real-world cases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to robust Meta-Reinforcement Learning (MRL), aiming to improve the consistency of returns across tasks, crucial for many RL applications. The authors introduce Conditional Value-at-Risk (CVaR) optimization to MRL, and propose a new method called Robust Meta RL algorithm (RoML). This approach overcomes issues of biased gradients, common in CVaR optimization in RL, and increases sample efficiency by identifying and over-sampling lower return tasks. Experimental results support the effectiveness of RoML, showing improved robustness over baseline algorithms in various domains.\n\nSTRENGTHS:\n**Strengths**:\n\n1. The use of CVaR optimization in Meta RL is a novel idea that addresses the issue of robustness, a common challenge in reinforcement learning applications.\n\n2. The theoretical proof that MRL is immune to biased gradients, a common issue in CVaR optimization in RL, is a significant strength.\n\n3. Experimental results highlight the robustness and efficiency of the RoML method across various domains.\n\nWEAKNESSES:\n**Weaknesses**:\n\n1. The RoML algorithm assumes that tasks can be selected during training. This might not always be feasible in real-world applications, potentially limiting its general applicability.\n\n2. While the paper mentions that average return is improved in certain experiments, it does not provide a clear discussion on this, leaving questions on how RoML impacts average return across different applications.\n\nQUESTIONS:\n1. Could you elaborate on the conditions under which RoML improves the average return? Does this improvement come at the cost of performance in other aspects?\n\n2. Could you provide more clarity on how task selection during training works with the RoML algorithm?\n\nLIMITATIONS:\nThis paper includes a few sentences of the main limitations. I suggest authors discuss more on this. \nI don't find any obvious societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aim to address the limitations in the standard meta reinforcement learning (MRL) methods, where they optimize the average return over tasks and often struggle with high-risk or difficult tasks, hence limiting system reliability. To this end, the authors define a robust MRL objective with a controlled robustness level to overcome these limitations. Meanwhile, this work proposed RoML with provable gradient bias reduction and improvement on the data efficiency. Overall, this paper is well-organized and presented. The proposed method is promising with experimental evidence. They show that RoML achieves robust returns, indicating improved performance and reliability in challenging scenarios. By combining the controlled robustness of the MRL objective and the oversampling approach of RoML, this paper contributes to advancing the field of RL in real-world applications with variable and unknown test tasks.\n\nSTRENGTHS:\n1. To address the impact of the biased gradient issue in standard RL, this work proposed to use CVaR in the objective and provides the theoretical guarantee (Theorem 1)\n2. The proposed CVaR-ML has improved data efficiency by a factor up to $1/\\alpha$\n3. The conducted experiments on both risk-averse, risk-neutral baselines shows promising results of RoML.\n\nWEAKNESSES:\n1. The proposed algorithm RoML requires the proper set up of parameter $\\alpha$ and $\\beta$, it will be beneficial to discuss more on the choice of those parameters for different tasks? e.g., do you use grid-search for choosing those parameters?\n\nQUESTIONS:\n1. Does the rollout length matter to the performance? Do you choose different rollout length for different tasks?\n2. Does over-sampling on the low-return tasks increase the computation complexity?\n3. If the task distribution changes overtime, does the over-sampling strategy still be able to guarantee the robustness?\n\nLIMITATIONS:\nThe author has discussed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}