Train Hard, Fight Easy: Robust Meta Reinforcement Learning
Authors: Ido Greenberg
Shie Mannor
Gal Chechik
Eli Meirom
Venue: NeurIPS 2023 poster (2023)
Keywords: meta reinforcement learning
robust reinforcement learning
safe reinforcement learning
risk sensitive reinforcement learning
TL;DR: Making meta reinforcement learning more robust by learning to over-sample high-risk tasks throughout meta training.

Abstract: A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both **biased gradients** and **data inefficiency**. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and over-sampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.

Train Hard, Fight Easy: Robust Meta Reinforcement Learning
Ido Greenberg
Shie Mannor
Technion, Nvidia Research gido@campus. technion ac
Technion, Nvidia Research shieβee. techniin. ac.
Gal Chechik
Eli Meirom
Bar Ilan University, Nvidia Research gchechikenvidia.. com
Nvidia Research emeirom@nvidia. com
Abstract
A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk difficulty. This limits system reliability since test tasks are known in advance. In this work, we define a robust MRL objective with controlled robustness level. Optimization of analogous robust objectives RL known lead to both biased gradients and data inefficiency. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency addressed via the novel Robust Meta RL algorithm (RoML). RoML is meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and over- sampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.
1 Introduction
Reinforcement learning (RL) has achieved impressive results in variety of applica- tions recent years, including cooling sys-
tems control [Luo et al. 2022 and con- versational chatbots [Cohen et al.) [2022] A significant challenge in extending this success to mass production is the varia- tion between instances of the problem, e.g., different cooling systems or different chat- bot end-users. Meta-RL (MRL) addresses this challenge by learning "meta-policy" that quickly adapts to tasks (Thrn and Pratt 1998, |iin er al.] (217). In the ex- amples above, MRL would maximize the average return of the adapted policy for a new cooling system or a new end-user.
Driving!
Driving
S[ewur)
Driving( t
Driving i (
(b)
Figure 1: (a) An illustration of driving tasks, characterized by various weather conditions and traffic density. (b) The returns of two veta-policies mi, n2 these tasks. has higher average return, but 72 more robust to high-risk tasks. The task space discretized only for illustration purposes.
However, optimizing the return of the average client might not suffice, as certain clients may still experience low even negative returns. If 10% of the clients report poor performance, it may deter potential clients from adopting the new technology - even if its average return is high. This highlights
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the need for MRL systems that provide robust returns across tasks. Robustness is further motivated by risk sensitivity in many natural RL applications, such as medical treatment (Yu al. 202T] and driving (Shalev-Shwartz et al. 2016]. For example, as illustrated in Fig. [ an agent should drive safely at any road profile - even if at some roads the driving would be more cautious than necessary.
A common approach for risk-averse optimization is the max-min objective [Collins et al. 2020]; in MRL, that would mean searching for a meta-policy with the highest expected-retumi in the worst possible task. This expresses the most extreme risk aversion, which only attends to the one worst case out of all the possible outcomes. Furthermore, in certain problems, worst cases are inevitable (e.g., in certain medical treatments, a fatal outcome cannot be avoided), thus optimizing the minimum return might not provide any meaningful result. A more general objective is the average return over the worst a quantiles (0 S a S 1), also known as the Conditional Value-at-Risk (CVaR). Notice that the CVaR is generalization of both the mean (for a = 1) and the minimum (for a = 0).
CVaR is a coherent risk measure used for risk management in various fields (Filippi et al. 2020] + including banking regulation [Acerbi and Szekely [014] and RL (Tamar et al.. 2015 Hiraoka et al. 2019]. In this work, we extend CVaR optimization to MRL, by replacing the standard MRL objective
argmax (R), (R) = [.R]
with a CVaR objective, which measures the robustness of a policy to high-risk tasks:
argmax (I),, (²)) = CVaRR [ER[
In both equations, T is a random task and R is the random return of policy vo in 7. Intuitively, the CVaR return expresses robustness the selected task, in analogy robustness to the realized model in standard RL. To further motivate Eq. (2). note that CVaR optimization in RL is equivalent to robust optimization under uncertain perturbations [Chow et al. 2015).
In Section 4. we follow the standard approach of policy gradient (PG) for CVaR, optimization in RL, and apply it to MRL. That is, for every batch of N trajectories, we apply the learning step to the aN trajectories with the lowest returns. This standard approach, CVaR-PG, is known to suffer from major limitation: in an actor-critic framework, the critic leads to a biased gradient estimator to the extent that it may point to the opposite direction [Tamar et al. 2015). This limitation is quite severe: many CVaR-PG implementations [Tamar et al., 2015 @015|Grrenberrg et al., 2022 rely vanilla PG without a critic (REINFORCE, Williams| [1992]; others pay the price of gradient bias- - in favor of advanced actor-critic methods that reduce the gradient variance (Rajeswaran et al.) 2017).
This limitation is particularly concerning in meta RL, where high complexity and noise require more sophisticated algorithms than REINFORCE. Fortunately, Section 4 eliminates this concern: in MRL, in contrast to RI, the CVaR policy gradient proven to remain unbiased regardless of the choice of critic. Hence, our proposed method CVaR Meta Learning (CVaR-ML) can be safely applied on top of any MRL algorithm. This makes CVaR-ML a meta-algorithm: given an arbitrary MRL algorithm, CVaR-ML generates robust version of it.
Nevertheless, in CVaR optimization methods, another source of gradients variance and sample inefficiency is the large proportion of data not being utilized. Every iteration, we rollout trajectories for N tasks, but only use a. of them for training. To mitigate this effect, we introduce in Section 5 the Robust Meta RL algorithm (RoML). RoML assumes that tasks can be selected during training. It learns to identify tasks with lower returns and over-samples them. By training on high-risk tasks, the meta-agent learns policies that are robust to them without discarding data. Hence, RoML. increases the sample efficiency by a factor of up to a -1, Unlike common adversarial methods, which search for the worst-case sample (task) that minimizes the return [Collins et al.) 2020), RoML lets the user specify the desired level of robustness a, and addresses the entire a-tail of the return distribution.
We test our algorithms on several domains. Section 6.1 considers a navigation problem, where both CV and RoML obtain better CVaR returns than their nestral baseline. Furthermore, they learn substantially different navigation policies. Section 6.2 considers several continuous control environments with varying tasks. These environments are challenging for CVaR-ML, which entirely fails to learn. Yet, RoML. preserves its effectiveness and consistently improves the robustness of the returns. addition, Section 663 demonstraass that under certain conditions, RoMI. can be applied to supervised settings as well providing robust supervised meta-learning.

---

As meta-algorithm, in each experiment RoML improves the robustness of its baseline algorithm using the same ayper-parameters as the baseline. The average return is also improved in certain experiments, indicating that even the risk-neutral objective of Eq. ① may benefit from robustness.
Contribution: (a) We propose a principled CVaR optimization framework for robust meta-RI.. While the analogous problem in standard RL suffers from biased gradients and data inefficiency, we (b) prove theoretically that MRL is immune the former, and (c) address the latter via the novel Robust Meta RL algorithm (RoML). Finally, (d) we demonstrate the robustness of RoML experimentally.
2 Related Work
Meta-RL for the average task is widely researched, including methods based on gradients (Finn
et al. 2017 Gupta et al. [018). latent memory (Zintgraf et al.) 2019 Rakelly et al.] (019) and offline
meta learning Dorfman et al. 2020 Pong et al. [002]] It is used for applications ranging from
robotics (Nagabandi al.) 2018 education [Wu al. 2021]. Adversarial meta learning was
studied for minimax optimization of the lowest-return task, in supervised meta learning (Collins)
al. 2020 Goldblum al.] 2020] and MRL [Lin et al. 2020]. Other works studied the robustness
of MRL to distributional shifts [Mendonca et al. 2020 Ajay ct al. 2022). However, the CVaR task
objective has not been addressed yet in the framework of MRL.
Risk-averse RL. In standard RL, risk awareness is widely studied for both safety [García and
Fernández 2015 Greenberg and Mannor [021] and robustness [Derman et al. 2020].
CaR
specifically was studied using PG [Tamar et al. 2015 Rajeswaran et al. [2017] Hiraoka et al.
2019
Huang et al. [021a] value iteration [Chow al. [013] and distributional RL [Dabney et al.
2018
Schubert et al. [021 Lim and Malik 2022]. CVaR optimization was also shown equivalent to
mean
optimization under robustness [Chow et al. 2015), motivating robust-RL methods (Pinto et al.)
2017
Godbout et al. 2021]. In this work, we propose a meta-learning framework and algorithms for CVaR
optimization, and point both similarities and differences from the standard RL setting.
Sampling. In Section 5 we use the cross-cntrops method dd Boer et al. 2005 to sample high-risk tasks for training. The cross-entropy method has been studied in standard RL for both optimization [Mannor et al. 2003 Huang et al. 2021b) and sampling [Greenberg et al.) 2022]. Sampling in RL was also studied for regret minimization in the framework of Unsupervised Environment Design (Dennis) jet al. [2200] Jiang et al. 2021]: : and for accelerated curriculum learning in the framework of Contextual RL Klink et al. 2020 Eimer al. 2021]. By contrast, we address MRL (where the current task is unknown to the agent, unlike Contextual RL), and optimize the CVaR risk measure instead of the mean.
3 Preliminaries
MRL. Consider set of Markov Decision Processes (MDPs) (SS,A, Po.,, where the distribution of transitions and rewards P, and the initial state distribution Po. both depend on D over general space Z, and is not
task Ω. The task itself is drawn from distribution known to the agent. The agent can form a belief regarding the current r based on the task history h, which consists of repeating triplets of states, actions and rewards [Zintgraf et al.) 2019 Thus, the meta-policy (a; s, h) (Ø E θ) maps the current state s E S and the history h E II x X R) (consisting of aitte-actioo-rwwrdd triplets) a probability distribution over actions.
meta-rollout is defined as a sequence of K Z episodes of length E N over a single task r: A = {[( ak,t, Tk.:) )r }KK k= For example, in a driving problem, might be geographic area or type of roads, and A sequence of drives on these roads. The return of the agent over a mota-rollout is defined as R(A)= = * Σ En 7'Tk.2, where t is the (random variable) reward at step in episode k. Given task 7 and a meta- meea--olicy Ty we denote by PM (xx the conditional PDF of the return R. With a slight abuse of notation, we shall use P (A) to also denote the PDF of the meta-rollout itself. The standard MRL objective is to maximize the expected return Jo (R) = [r,.[
While the meta policy (T a; h) is eestory-dependent it can still be learned using standard policy gradient (PG) approaches, by considering h part of an extended state space 3 hs, S. Then,
3

---

the policy gradient can be derived directly:
RR) J. D(2)
(x /
(x- dx dz,
(3)
where D is a probability measure over the task space Q, (x) is the PDF of (conditioned on Ty and = 2), and b is any arbitrary baseline that is independent of 0 [Agrawal 2019]. While a direct gradient estimation via Monte Carlo sampling is often noisy, its variance can be reduced by an educated choice of baseline b. In the common actor-critic framework (Mnih et al. 2016]. learned value function = (ss; h) is used. This approach used in many SOTA algorithms in deep RL, e.g., PPO [Schulman et al., 2017]; and by proxy, in MRL algorithms that rely on them, e.g..
VariBAD Zintgraf et al. 2019].
major challenge in MRL is the extended state space S, which now includes the whole task history. Common algorithms handle the task history via now-dimensional embedding that captures transitions and reward function (Zintgraf et al. [ or using additional optimization steps w.r.t. task history FFin et al. 2017]. Our work does not compete with such methods, but rather builds upon them: methods operate as meta-algorithms that run on top of existing MRL. baselines.
CVaR-PG. Before moving on to CVaR optimization in MRL, we first recap the common PG approach for standard (non-meta) RL. For a random variable and o--uuatile qo (X), , the CVaR is defined
as CVaR,, E [ X S yo (X)]. For an MDP (S, A, P, Po. 7)), the CVaR-return objective is
jØ (R) CVaR Ro [R]
5 (R) x (xx) dx, whose corresponding policy gradient is Tamar
et al. 2015
Vo = | (2)(1)
(r (R)) Vo (x) dr.
Given a sample of N trajectories (( a a. IN with returns {R the policy gradient can be estimated by TTamar et al. 2015 Rajeswaran al. 2017|:
T ve log (
V.Jo
Σ R.s (R, 2 a.
(5)
where o% is estimator of the current return quantile.
Notice that in contrast to mean-PG, in CVaR optimization the baseline cannot follow an arbitrary critic, but should approximate the total return quantile 9% (R). Tamar et al. (2015) showed that any baseline & da (R) inserts bias to the CVaR gradient estimator, potentially to the level of pointing to the opposite direction (as discussed in Appendix [A.] and Fig. 6). As a result, CVaR-PG methods in RL either are limited to basic REINFORCE with a constant baseline [Greenberg et al. 2022), or use a critic for variance reduction the cost of biased gradients (Rajeswaran et 2017
Another major source of gradient-variance in CVaR-PG is its reduced sample efficiency: notice that Eq. (5) only exploits a:N trajectories out of each batch of N trajectories (due to the term 1 sg,
hence results in estimation variance larger by a factor of -1
4 CVaR Optimization in Meta-Learning
In this section, we show that unlike standard RL., CVaR-PG in MRL permits a flexible baseline without presenting biased gradients. Hence, policy gradients for CVaR objective in MRL is substantially different from both mean-PG in MRL (Eq. and CVaR-PG in RL (Eq. (4).
To derive the policy gradient, we first define the policy value per task and the tail of tasks.
Definition 1. The value of policy we in task T is denoted by V = Eg [R]. Notice that V depends on the random variable We define the a-tail of tasks w.r.t. na as the tasks with the lowest values: Ω2 = {= E Ω V < (V.)))
Assumption 1. To simplify integral calculations, we assume that for any 2 E Q and f E θ, R is a continuous random variable (i.e., its conditional PDF Pf (z) has atoms). We also assume that (z)) v continuous function for any 0 e.

---

Theorem 1 (Meta Policy Gradient for CVaR). Under Assumption D objective in Eq. is
the policy gradient of the CVaR
(R) [ D(2) / (a b)VoP (
dr dz,
(6)
where b is any arbitrary baseline independent of O.
Proof intuition (the formal proof is in Appendix In RL, the CVaR objective measures the lowest-return trajectories. When the policy is updated. the cumulative probability of these jectories changes and no longer equals
Thus, the new CVaR calculation must add or remove trajectories (as visualized in Fig- 6 in the appendix). This adds a term in the gradient calculation, which causes the bias in CVaR-PG. By contrast, in MRL, the CVaR measures the a lowest-return tasks D Since the task distribution does not depend on the policy, the probability of these tasks is not changed - but only the way they are handled by the agent (Fig. 7 Thus, no bias term appears the calculation. Note that Ω does change throughout the meta-learning due to changes in task valuey (rather than task probabilities); this is different effect and is not associated with gradient bias.
According to Theorem [ the CVaR PG in MRL permits any baseline b. As discussed in Section this flexibility is necessary, for example, in any actor-critic framework.
To estimate the gradient from meta-rollouts of the tail tasks, we transform the integration of Eq. (6) into expectation:
Corollary 1. Eq. (6) can be written as
( = ENnD EANpy (a(A)] V S 2a (V?)
where g(A) (R(A) - b) Σ iSKSK, Vo log mo (ax,; 8k,e); and 5n, (sk,es hxre) is the extended
ISTST
state (that includes all the task history hR. until trajectory k. step t).
Proof. We apply the standard log trick VeP = P. Vo log P to Eq. 6 after substituting the meta- rollout PDF: ( (A) = (s. IT (2 TR, Sk.es ag ) (ap,3 8g,e)
For practical Moote-Carld estimation of Eq. 77). given task 2, need estimate whether V S 9a (V To estimate V2, we can generate M iid meta-rollouts with returns {m 1 mm=
m..m and calculate their average return V R. Em= Ri.m /M. Then, the quantile q.. (V,7) can
be estimated over batch of tasks @. = qa qa (((a))))) If V < 4.. we use all the meta-rollouts of 2; for the gradient calculation (including meta-rollouts that by themselves have higher return Rim > (.). Notice that we use M i.i.d meta-rollouts, each consisting of K episodes (the episodes within a meta-rollout
not independent, due to agent memory).
Putting together, we obtain the sample-based gradient estimator of Eq. 7
M
VoJ% a.
Σ
R. S00
Ji,m.
(8)
T
Di.m = RRa.
b)
Vo log no (ai,m, ks,)),)),
where 0i,m, ks.m.k.t are the action and the state-and and-history at task i, meta-rollout m, trajectory k and step
The procedure described above follows the principles of CVaR-PG in (non-meta) RL, as the learning rule is only applied the tail of the sampled batch. However, in MRL we consider batch of tasks rather than a batch of trajectories. As discussed Theorem and its proof, this distinction has a substantial impact the gradient and the resulting algorithm. Specifically, Eq. (8) allows for greater flexibility than Eq. (4). as it permits any baseline b that does not depend on 0. This allows gradient calculation using any PG algorithm, including SOTA methods such PPO (Schulman
2017 (which are already used in MRL methods such as VariBAD Zintgraf et al (201)). Therefore, in contrast standard RL, CVRR optimization is not restricted basic REINFORCE.
5

---

Our CVaR Meta Learning method (CVaR- ML. Algorithm D leverages this property to operate as a mela-algorithm, providing robust version for any given baseline algo- rithm, such as Finn et al. [017]. Zintgraf et al. (011)]
Algorithm 1: CVaR Meta Learning (CVaR-ML.
Input: Meta-learning algorithm (Definition 2
robustness level a E (0. 1|; task distribution D;
N tasks per batch; M meta-rollouts per task
2 while not finished training do
// Sample tasks
Definition 2. A baseline MRL algorithm
3 Sample {- ~ D
learns meta-policy my using training step ML. Given batch of meta-rollouts {Ai). ML((A,)) updates me.
// Run meta-rollouts
[fAi.m/m=11 10 )1111111111 - rollouts(f2))@a1 M)
5 Ri, <- return(Ai.m), Ví, m
Notice that CVaR-ML only handles task filter- ing, and uses the bascline training step ML as a black box (Line (). Hence, it can be used with any MRL baseline not just PG methods. fact, by using supervised leaaning baseline, CVaR-ML can be applied to the su- pervised setting as well with minimal modifications, namely, replacing meta-rollouts with examples and returns with losses.
// Compute sample quantile
R3 <- mean( JM m=1)), Vi 7 da f quantile( (R Q)
// Meta-learning algorithm train step
8 ML ({Ai.m R, < la. 1 m m
5 Efficient CVaR-ML
Theorem guarantees unbiased gradients when using Algorithmi however, it does not bound their variance. In particular, Line Sapplies the learning step to a subset only o:N M meta-rollouts out of N.M, which increases the estimator variance by a factor of a compared to mean optimization. This could be prevented if we knew the set 2% of tail-tasks (for the current TT), and sampled only these tasks, using the distribution D (=) = a 1yr (2)). Proposition shows that this would indeed recover the sample efficiency.
Proposition (Variance reduction). Denote by G the estimator of VoJ [ no quantile error (i d.), and denote En|] Ez,D Ra []. Then, switching the task sample distribution to D" leads to a variance reduction of factor a::
in Eq. 8 assume there
Dg [aG] = ED[G], Var (aG) a Varp(G).
Proof sketch (the complete proof is in Appendix
Bj. We calculate the expectation and variance di-
rectly. G is proportional to 1 R. Saf (Eq. (8) The condition Rt < ca leads to multiplication by the
probability a when sampled from D (where it corresponds to the a-tail); but not when sampled from
De (where it is satisfied w.p. 1). This factor cancels out the ratio between the expectations of G and aG (thus the expectations are identical) but not the ratio a between their variances.
Robust Meta-RL (RoML)
task value: E[R] r. ng]
true value sampled retumn au(value) original D2, learned De
task piobabil lity Py(t)
Rolioute (A)
Legend
Cross entropy method: update
Meta leamne:: update
Data
Environment
Task distribution De
Tasks [t)-De
Meta-policy
lask:
Figure 2: Left: RoML uses cross entropy method modify the task distribution De., which is used to generate the next meta-rollouts. Right: illustration of an arbitrary point of time in training: the task distribution Da (blue) is learned according to the task values of the current meta-policy ne (red). Since low-return tasks are over-sampled, the learned meta-policy more robust to the selection of task.

---

Following the motivation of Proposition I we wish to increase the number of train tasks that come from the tail distribution 22_ To that end, we assume to have certain control over the sampling of training tasks. This assumption is satisfied in most simulated environments, as well as many real-world scenarios. For example, when training a driver, we choose the tasks, roads and times of driving throughout training. In this section, we propose method to make these choices.
We begin with parameterizing the task distribution D: we consider a parametric family D& such that D Da-- Then, wish modify the parameter á so that Da aligns with Då closely as possible. To that end, we use the Cross Entropy Method (CEM. de Boer et al. (2005) D. which searches for 6* that minimizes the KL-divergence (i.e., cross entropy) between the two:
6² E argming DEL (D& Dw) argmaxy De [Vv SVa())) log D (z) argmax, En [w(2) Iv log Dy (z)]
where w2)) (2)(2) is the importance sampling weight corresponding
Dg. Note that Eq. 9
Dz(s)
has a closed-form solution for most of the standard families of distributions dde Boer et al. 2005
For a batch of N tasks sampled from D Don Eq. (9) essentially chooses the a.V tasks with the lowest returns, and updates & to focus on these tasks. This may be noisy unless a.V > 1. Instead, the CEM chooses larger number of tasks 3 >> a for the update, where 3 is a hyper-parameee é is updated according to these 3N lowest- return tasks, and the next batch is sampled from D& + Deo- This repeats iteratively: every batch is sampled from Dø. where & is updated according to the 3N lowest- return tasks of the former batch. Each task return is also compared to the a-quaanile of the original distribution D If more than BN tasks yield lower returns, the CEM permits more samples for the update step. The return quantile over D&o can be esti- mated from D& at any point using impor- tance sampling weights. See more details about the CEM in Appendix C.I
Algorithm 2: Robust Meta RL (RoML)
Input: Meta-lcarning algorithm (Definition 2 robustness level a E (0, 1]: parametric task distribution D3: original parameter Φo: N tasks per batch, V E [0, 1) of them sampled from the original Doy: CEM quantile 3 10
Initialize:
3
6 + Φv. N + [vN], N₂ + [(1 (1-v)N)
4 while not finished training do
// Sample tasks
Sample [ No Deo:
{ Do
6 2 < -- No 20.1...., 2,
// Rollouts and beta-learning step
7
[Ar) rollout ((x)
8
R; + return(As). (I, N)
9
ML((A)))))
// Estimate reference quantile
10
Wo,i + 1.
Vi (l....,N, No)
11
Wo + (an(s
Vi (l,....., Dolto,))
In our problem, the target distribution is the tail of D Since the tail is defined by the agent returns in these tasks, it varies with the agent and is non-stationary throughout training. Thus, we use the dynamic- target CEM of Greenberg 2022] To smooth the changes in the sampled tasks, the sampler is also regularized always provide certain exposure to all the tasks: we force percent of every batch to be sampled from the original distribution
w. f (Wo.1...., Wo,N. Wo.1,..., ..., Wo,N,)
13
G. + weighted_ qutrriil_ggnieeee(ee [Rih. a)
1/ Compute sample quantile
14
93 + quantile( (. 9))
1/ Update sampler
+- max(da; qa)
16
Φ < argmax EiSN 1 Sq (og
D Dao and only -v percent from Dø-
Putting this together, we obtain the Robust Meta RI. algorithm (RoML), summarized in Algorithm 2 and Fig. RoML does not require multiple meta-rollouts per update (parameter M in Algorithm []. since it directly models high-risk tasks. Similarly to CVaR-ML, RoML is a meta-algorithm and can operate on top of any meta-learning baseline (Definition 2 Given the bascline implementation, only
the tasks sampling procedure needs to be modified, which makes RoML easy to implement.
Limitations: The CEM's adversarial tasks sampling relies several assumptions. Future research may reduce some of these assumptions, while keeping the increased data efficiency of RoML.
First, mentioned above, we need at least partial control over the selection of training tasks. This assumption common in other RI. frameworks [Dennis et al.) 2020 Tiang et al.) [02T], and often holds in both simulations and the real world (c.g., choosing in which roads and hours train driving).
7

---

Second, the underlying task distribution D is assumed to be known, and the sample distribution is limited to the chosen parameterized family {D For example, if U([0, 1)). the user may choose the family of Beta distributions (eta( Φ2) (where (ela(1. 1) = ([[ 1)), as demonstrated in Appendix D.2 The selected family expresses implicit assumptions on the task-space. For example, the probability density function is smooth, close tasks will always have similar sample probabilities; and if the family is unimodal, high-risk tasks can only be over-sampled from around a single peak. This approach is useful for generalization across continuous task-spaces where the CEM can never observe the infinitely many possible tasks. Yet, it may pose limitations in certain discrete task-spaces, if there is no structured relationship between tasks.
6 Experiments
We implement Ro and CVaR-ML. on top of two different risk-neutral MRL baselines VariBAD (Zintgraf et al. 2019] and PEARL (Rakelly et al.] 2019]. As a risk-averse reference for comparison, we use CeSoR Greenberg et al., 2022]. an efficient sampling-base based method for CVaR optimization in RL, implemented on top of PPO. As another reference, we use the Unsuper- vised Environment Design algorithm PAIRED [Dennis et al. 2020]. which uses regret minimization to learn robust policies on diverse set of tasks.
Section 6.1 demonstrates the VaR tradeoff, as our methods learn substantially different policies from their baseline. Section 6.2 demonstrates the difficulty of the naive CVaR-ML in more challenging control benchmarks, and the RoML's efficacy in them. The ablation test in Appendix D.4] demonstrates that RoML deteriorates significantly when the CEM is replaced by naive adversarial sampler. Section (.33 presents an implementation of CVL and RoML on top of MAML (Finn al., 2017) for supervised meta-learning. In all the experiments, the running times of RoML and CVaR-ML are indistinguishable from their baselines (RoML's CEM computations are negligible).
Hyper-parameters: To test the practical applicability of RoML as algorihh,, in every experiment, we use the same syper-parameters for RoML, CVaR-ML and their baseline. In particular, we use the bascline's 's default syper-parametert whenever applicable (Zintgraf et al. [019]]
Rakelly et al. [019) in Section 6.2, and Finn et al. (2017) in Section 6.3) That 18, we use the
same hyper-parameters as originally tuned for the baseline, and test whether RoML improves the robustness without any further tuning of them. As for the additional hyper-parameters of the meta-
algorithm itself: in Algorithm [ we M = 1 meta-rollout per task; and in Algorithm 2 we use
3 = 0.2, V = 0 unless specified otherwise (similarly to the CEM in GGrenbeerg et al. (2022)) ), For
the references PAIRED and CeSoR, we use the hyper-parameters Dennis et al. [2020] Greenberg et al. 2022]. Each experiment is repeated for 30 seeds. See more details in Appendix
The code is
available in our repositories: VariBAD PEARL CeSoR PAIRED and MAML
6.1 Khazad Dum
We demonstrate the tradeoff between mean and CVaR optimization in the Khazad Dum benchmark, visualized in Fig. 3 The agent begins at random point in the bottom-left part of the map. and has to reach the green target quickly possible, without falling into the black abyss. The bridge is not covered and thus exposed to wind and rain, rendering its floor slippery and creating an additive action noise (Fig. Bb) to a level that varies with the weather. Each task is characterized by the rain intensity, which is exponentially distributed. The CEM in RoML is allowed to modify the parameter of this exponential distribution. Note that the agent is not aware of the current task (i.e, the weather), but may infer it from observations. We set the target risk level to a = 0.01, and train each meta-agent for total of 5- 10 frames. See complete details in Appendix D.I
We implement CVaR-ML and RoML on top of VariBAD. As shown in Fig. 33, VariBAD learns to try the short path- at the risk of rare falls into the abyss. By contrast, our CVaR CVaR-optimizing methods take the longer path and avoid risk. This policy increases the cumulative cost of time-steps, but leads to higher CVaR returns, as shown in Fig- 31 In addition to superior CVaR, RoML also provides competitive averuge returns this example (Fig. Beh. Finally, accordance with Proposition T
RoML learns significantly faster than CVLR-ML.
8

---

VariBAD
(h) VariBAD
(a) Stum-Mass (VariBAD)
(b) HaliChectahh-Be
1800
1600
1400
VariBAD (1558)
(1581)
(c) CV.
(d) RoML
mass
(c) Hum-Mass (RoML)
vatidation Coittan
(d) HalfCheetah-MMee
Figure 4: MuloCo: (a-c) Sample frames, where tasks vary in mass, head size and damping level. No- tice that in Humanoid, RoML handles the low-return tasks large mass by leaning the center of mass forward, so that gravity pulls the humanoid forward. (d) Average return range of tasks in HalrCheetah- Mass. RoML learns
So11 Ch66-M ML
2006
Seration
(e) Mean
(I) CVaR
Figure 3: Khazad-Duu Dum: (a-d) Sample episodes.
robustly: it is less sensitive to the task, particular performs better high- risk tasks.
(e-f) Test return vs. training iteration, with 95% con- fidence intervals over seeds.
6.2 Continuous Control
We rely on standard continuous control problems from the MuJoCo framework (Todorov et al. 2012]: training cheetah to run (HallCheetabh) and training Humanoid and an Ant to walk. For each of the 3 environments, we create 3 meta-learning versions: (1) Goal or Vel [Finn et al. 2017].
where each task corresponds a different location velocity objective, respectively; (2) Mass, where each task corresponds to different body mass; and (3) Body, where each task corresponds to different mass, head size and physical damping level (similarly to Wang and Van Hoof [2022]) ). In
addition, to experiment with high-dimensional task spaces, randomly draw 10 numeric variables from env.model in HalfCheetah, and them vary between tasks. We define 3 such environments with different random sets of task variables (HalfCheetah 10D-task a,b,c). For each of the 12 environments above, we set a target risk level of a 0.05 and optimize for = 2 episodes per task. Additional implementation details are specified in Appendix D.2 Interestingly, the naive approach of CVaR-ML consistently fails meta-learn in all the cheetah environments. It remains unsuccessful even after large number of steps, indicating difficulty beyond sample inefficiency. possible explanation the effectively decreased batch size of CVaR-ML. PAIRED and CeSoR also fail to adjust to the MRL environments, and obtain poor CVaR returns.
RoML, the other hand, consistently improves the CVaR returns (Table compared its baseline (VariBAD or PEARL., while using the same hyper- sapee-ppraeeters as the baseline. The VariBAD baseline presents better returns and running times than PEARL HalfCheetah, and thus is used for the extended environments (Humanoid and Ant). RoML improves the CVaR return in comparison to the baseline algorithm in all the 18 experiments (6 with PEARL and 12 with VariBAD).
In out 18 experiments, RoML slightly improves the average return compared to its baseline, and not only the CVaR (Table in the appendix). This indicates that low-return tasks sometimes be improved at the cost of high-return tasks, but without hurting average performance. addition, this may indicate that over-sampling difficult tasks forms helpful learning curriculum.
The robustness of RoML to the selected task demonstrated in Fig. 4d In multi-dimensional task spaces, RoML learns to focus the sampling modification the high-impact variables, demonstrated

---

Table 1: CVaRo.0s return over 1000 tasks, for different models and MuJoCo environments. Standard deviation is presented over 30 seeds. Mean returns are displayed in T
HalfCheetah Mass
HalfCheetah 10D-task
Vel
Body
(a)
(b)
(c)
CeSoR
2600 25
902 36
478
637
981 +: 31
664 +
PATRED
-725 + 65
138 + 37
218 + 51
229 + 59
354 + 53
81 + 65
CVVR--ML
-897 + 23 23
38 - 6
76 5
120 +: 11
141 + 11
81 =
PEARL
-1156 23
1115 + 19
800
1140
1623 + 23
1016
VariBAD
-202
1072
835 30
1126
1536
+ 39
988 =
RoML (VariBAD)
-184
1259 + 19
935 135111
1227 + 13
1697
24
999 = 20
RoML (PEARL)
1089 + 31
1186 + 34
808 6
1141 + 27
1657 + 18
1024 -
Humanoid Mass
Ant Mass
Vel
Body
Goal
Body
VariBAD 801 1 10
1283 + 18
1290 +: 19
500 1: 9
1370 + 6
1365 4
RoML (VariBAD)
833 + 4
1378 + 20
1365 + 21
454 + 8
1385 +
1368 + 4
Fig. in the appendix. Finally, qualitative inspection shows that RoML learns to handle larger masses, for example, by leaning forward and letting gravity pull the agent forward, as displayed in
Fig. 4c (see animations GitHub).
6.3 Beyond RL: Robust Supervised Meta-Learning
model MAMI
CVaRe loss
Our work focuses on robustness in MRL. However, the concept of training harder data improve robustness, embod- ied in RoML, is applicable beyond the scope of RL. As preliminary proof-of-concept,
RaML
12
apply RoML toy super- vised meta-learning problem of sine regression, based on Finn [0, 2x),, the desired output is
al. 2017]: The input Asi wI b), and the task is defined by the parameters == (A, w). Similarly to Finn et al. [017]]. the model is fine-tuned for each task via a gradient- -rradeent-eeseen optimization step over 10 samples {( yt) ) and tested on another set of 10 samples. The goal is to find model weights that adapt quickly to new task data.
tuning iteration
Figure 5: Supervised Sine Regression: CVaR loss 10000 test tasks, against the number of tuning gradient-steps at test time. The 95% confidence intervals are calculated over 30 seeds.
We implement CVaR-ML and RoML on top of MAML [Finn et al.) (017]. As shown in Fig. 5 RoML achieves better CVaR losses over tasks than both CVaR-ML and MAML. The complete setting and results are presented in Appendix
7 Summary and Future Work
We defined a robust MRL objective and derived the CVaR-ML algorithm optimize In contrast to its analogous algorithm in standard RL, we proved that CVaR-ML does not present biased gradients, yet does inherit the same data inefficiency. To address latter, we introduced RoML and demonstrated its advantage in sample efficiency and CVaR return.
Future research may address the CEM-related limitations of RoML discussed at the end of Section Another direction for future work is extension of RoML to other scenarios, especially where natural task structure can be leveraged improve task robustness, e.g., in supervised learning and coordinate-bant based regression TTancik et al. 2020].
Finally, RoML is easy to implement operates agnostically meta-algorithm top of existing MRL methods, and can be set to any desired robustness level. We believe that these properties, along with our empirical results, make RoML promising candidate for MRL in risk-sensitive applications.
Acknowledgements: This work has received funding from the European Union's 's Horizon Europe Programme, under grant number 101070568.
10

---

References
Carlo Acerbi and Balazs Szekely. Back-testing expected shortfall. Risk, (7 2014.
Agrawal. Reinforcement learnigg Policy gradicnt methods, 2019. URLphttss//ieor0000 Sttthh/w....cccccccc///////////0
pdf
Anurag Ajay, Abhishek Gupta, Dibya Ghosh, Sergey Levine, and Pulkit Agrawal. Distributionally adaptive meta reinforcement learning. arXiv preprint arXiv:2210.033 2022.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone, Risk-sensitive and robust decision- making: cvar optimization approach. Advances in neural information processing systems, 28, 2015.
Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, and Gal Elidan. Dynamic planning in open- open-ended dialogue using reinforcement learning, 2022. httpsttt:://// org.://.vrrr///oooooooooo abs/
2208 .2294
Liam Collins, Aryan Mokhtari, and Sanjay Shakkottai. Task-robust model-agnostic meta-learning. Advances in Neural Information Processing Systems, 33: 13:18880-18871, 20200.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings the 35th International Conference on Machine Learning, pages 1096- 116-1105, 105, 2018.
P. T. de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial the cross-entropy method. Annals Operations Research, 134:19-67, 2005.
Tito Homem de Mello and Reuven Y. Rubinstein. Rare event estimation for static models via cross-entropy and importance sampling, 2003.
Michael Dennis, Natasha Jaques, Fugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 33: 3049-13061, 2020.
Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A bayesian approach robust reinforcement learning. In Uncertainty in Artificial Intelligence, pages 648-658. PMLR, 2020.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint MwXiv:2008.02599 2020.
Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-paced context evaluation for contextual reinforcement learning. In International Conference on Machine Learning, pages 2948-2958. PMLR, 2021.
Carlo Filippi, Gianfranco Guastaroba, and Maria Grazia Speranza. Conditional value-at-risk beyond finance: survey. International Transactions in Operational Research, 27(3):1277-13199 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of
deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Javier Gareía and Fernando Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16: 13:143371480, 2015.
Mathieu Godbout, Maxime Heuillet, Sharath Chandra, Rupali Bhati, and Audrey Durand. CARL: Conditional--value-ttriss adversarial reinforcement learning. arXiv preprint arXiv:2109.09441 2021.
Micah Goldblum, Liam Fowl, and Tom Goldstein. Adversarially robust few-shot learning: meta-learning approach. Advances in Neural Information Processing Systems, 2020.
Ido Greenberg. Cross entropy method with non-stationary score function. https://pyyp h/tp:://///oo////////// pro wwt.ee//ccrosssetrooo----oo-eeee 2022.

---

Ido Greenberg and Shie Mannor. Detecting rewards deterioration in episodic reinforcement learn- ing. In Proceedings of the 38th International Conference on Machine Learning, volume 139. pages 3842-3853. PMLR, 18 24-24.1 Jul 2021. URL httpttttp//occcedinnggg mlr http://ggs......ees////33.3 press/v139/ greenberg2la. html
Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 2022. URL https:
//arxiv. www. abs/ 22152
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, and Yoshimasa Tsuruoka. Learn- ing robust options by conditional value at risk optimization. NeurlPS, 05 2019.
Audrey Huang, Liu Leqi, Zachary C. Lipton, and Kamyar Azizzadenesheli. On the convergence and optimality of policy gradient for markov coherent risk, 2021a. httptttss///xiii.rrrrtrttet
2103.02827
Kevin Huang, Sahin Lale, Ugo Rosolia, Yuanyuan Shi, and Anima Anandkumar. CEM-GD: Cross- entropy method with gradient descent planner for model-based reinforcement learning, 2021b.
Minqi Jiang, Michael Dennis, Jack Parker-Holde Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1:8441111 188-11987 2021.
Pascal Klink, Carlo D Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33:9216-9227, 2020.
Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 2022.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning, Advances Neural Information Processing Systems, 33:10161-10173. 2020.
Jerry Luo, Cosmin Paduraru, Octavian Voicu, Yuri Chervonyi, Scott Munns, Jerry Li, Crystal Qian, Praneet Dutta, Jared Quiney Davis, Ningjia Wu, Xingwei Yang, Chu-Ming Chang, Ted Li, Rob Rose, Mingyan Fan, Hootan Nakhost, Tinglin Liu, Brian Kirkman, Frank Altamura, Lee Cline, Patrick Tonker, Joel Gouker, Dave Uden, Warren Buddy Bryan, Jason Law, Deeni Fatiha, Neil Satra, Juliet Rothenberg, Mandeep Waraich, Molly Carlin, Satish Tallapaka, Sims Witherspoon, David Parish, Peter Dolan, Chenyu Zhao, and Daniel J. Mankowitz. Controlling commercial cooling systems using reinforcement learning, 2022. URL attps:///rrxla
httgg/ab//221111 07357
Shie Mannor, Reuven Rubinstein, and Yohai Gat. The cross entropy method for fast policy search. Proceedings, Twentieth International Conference on Machine Learning, 2, 07 2003.
Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint arXiv:2006.07178, 2020.
2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy I Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pages 1928-1937 PMLR, 2016.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv: 803. 1347, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 2817-2826. JMLR.org, 2017.
12

---

Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reinforremmen learning with online superviiin.. International Conference on Machine Learning, pages 1781 1-17829. PMLR, 2022.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. ICLR, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- aff-policy meta-reinforcemeen learning via probabilistic context variables. In International conference machine learning, pages 5331-5340. PMLR, 2019.
Frederik Schubert, Theresa Eimer, Bodo Rosenhahn, and Marius Lindauer. Automatic risk adaptation in distributional reinforcement learning. arXiv preprint SxXiv:2106.06311, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Shai Shalev-Shware Shaked Shammah, and Amnon Shashua. Safe, multi agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
Aviy Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. 15AI15, page 2993-2999, 2015.
Matthew Taneik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations, 2020. URL httpstts::///// /attps://arxiv..oom org/ abs 02189
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5035 IEEE, 2012.
J. R. R. Tolkien. The Lord of the Rings: The Fellowship of the Ring. George Allen and Unwin, 1954.
Anthony Tromba and Jerrold Marsden. Vector calculus. Freeman, 1996.
Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning. pages 23055-23011 PMLR, 2022.
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255) 1992.
Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. Prototransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107.14035. 2021.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- meta-learning. In International Conference on Learning Representations, 2019.
13

---

Contents
Introduction
2
Related Work
3
Preliminaries
4
CVaR Optimization in Meta-l.carning
4
Efficient CV
6
Experiments
Summary and Future Work
10
Policy Gradient for CVaR Optimization in Meta RL
15
Proof of Proposition
18
The Cross Entropy Method
19
Experiments: Detailed Settings and Results
19
Supervised Meta-Learning
24
14

---

A Policy Gradient for CVaR Optimization in Meta RL
In this section provide the complete proof for Theorem For completeness, Appendix A. recaps of the analogous proof in standard (non-meta) RL, before moving on to the proof in Appendix A.2 This allows to highlight the differences between the two.
The substantial difference is that in RL, the CVaR is defined directly over the low-return trajectories, and the policy parameter 0 affects the probability of each trajectory in the tail (Eq. (10)). In MRL (Eq. (2), on the other hand, the CVaR is defined over the low-return tasks, whose probability is not affected directly by 0 (Eq. 12 This allows decoupling between and т, which results in Theorem
Another high-level intuition is as follows. In RL, the CVaR-PG gradient is invariant to successful strategies, hence must be explicitly negative for the unsuccessful ones (in order not to encourage them, see Fig. 6). In MRL, within the tasks of interest, the gradient always encourages successful strategies on account the unsuccessful ones (Fig.
A.1 Recap: PG for CVaR in (non-meta) RL
We briefly recap the calculation of Propositions 1 and 2 in Tamar et al. [2015) for CVaR policy gradient under the standard RL settings.
Definition 3 (CVRR return in (non-meta) RL). Consider an MDP (S, A, P. reward (i.e., return) R ~ pe, whose a-quantile is (H). Recall the CVaR objective defined in Section 3
Po) with the cumulative
(2)()) | x P.P"(z) dx
(R) R-
To calculate the policy gradient (H)), we begin with the conservation of probability mass below the quantile oh
(2 pd (x)dr =
Then, using the Leibniz integral rule, we have
(12) Pd (x)dr==
[
a (1)
0 0= V
V P (z)dx
[Po (e. (R) Vogi
(10)
Notice that particular consequence of the conservation rule Eq. (00.. positive gradients of (*)) cause the quantile (R) to decrease. This phenomenon makes the CVaR policy gradient sensitive to the selection of baseline, as visualized Fig. 6b In fact, the quantile df (R) itself is the only baseline that permits unbiased gradients:
(2))
Vo Vo
x. p0(x) dx
2 (M)
10
x V VoP (z)
+ (R) (R)( Vog.
(11)
(R)
(R) (rite) Vop VoP"(x)dx
2. VoP (z)
(R)
(x da (R)) (x)dd,
which gives us Eq.
A.2 PG for CVaR in Meta RL
We now turn to for CVaR optimization in MRI.. We rely on the definitions and notations of Section as well as Definition I and Assumption in Section Notice that S (R) of Eq. (2 can be
15

---

(R)
(R)
R
baseline
baseline
Mean PG
(R))R)
(M)(R)
baseline
baseline
(b) CVaR PG, Eq.
Figure 6: Illustration of the policy gradient estimation in standard RL. (a) In Mean PG, the expected gradient independent baseline: even most the distribution p seems
"pushed upwards", normalization to a total probability of forces the probability of low returns to decrease for that of high returns will increase. (b) In CVaR PG, due to the effect of Eq. 10 any baseline except for 4a (R) leads to biased gradients.
written in integral form
Ja (R) CVaR [En-rg [R] = /
{ x. (x) de dz.
D(z) 5g
In Eq. above, the boundary of the integral over the o-tail is simply the scalar da In MRL, this replaced by the boundary of the set 22g defined in general topological space. Thus, begin by characterizing this boundary.
Lemma (The boundary of 22)). Under Assumption
V2 E an
S x (x -x (V²).
Proof. Since v(z) v is continuous function between topological spaces, by denoting B (-o0, go (V?)) we have
contimuous
0n = du (B) C
( = - = (= (= (V,))
hence V2 E an V = (V.).. Notice that V = - (x) dr.
Finally, we can prove Theorem !l
Proof of Theorem ! First, consider the conservation of probability:
[
D(2)dz a
The gradient of this integral can be calculated using high- dimensional generalization of the Leibniz integral rule, named Reynolds Transport Theorem (RTT, Tromba and Marsden (1996)):
derivative constani
RTT D(2)d=
VeD(2)d2 + / D(=) (m. n) dA
V
120
(12)
VoD(=)=0
D(z) (Vo n) dA,,
aog
where 0) is the nutward-pointing unit vector that is normal to the surface a2 and (2 0) the velocity of the area element the surface with area d.A. Notice that VzD(z) 0 in the last
16

---

(CVaR,
Meta PG [Mean]
Meta PG (CVaR)
shifted baseline)
pT (R)
1
(R)
Fj
( (R)
task
baseline
(R)
p (P)
p (R)
rask Fz
Lask F2
T2
baselliee
pio (
p (R)
task Fg
task Ty
task
R
baseline
beseline
baseline
Figure 7: Illustration of the meta policy gradient estimation. In Mean meta-PG (left, Eq. 3 ). the PG is
estimated for all tasks. In CVaR meta-PG (center, Eq. (6), is only calculated the low-retur tasks. Since the returns distribution decoupled from tasks distribution, shifting baseline (right) does not insert bias the PG, in accordance with Theorem
equality expresses substantial difference from the standard RL settings in Eq. (10): there, have (x))(x), which does not necessarily vanish.
Next, we turn to the meta policy gradient itself, again using Reynolds Transport Theorem:
(R) Va / D(E) { xP dr dz
+ + D(2) ( (x) dr)
Lemma
D(2)
~VoP. (x) dx dz
(vb n) dA
Eg. 12
I D(z)
+ (a(V / D(z) (vs - n) dA
vV 120(1 (x) dx dz
Jong
/ D(2)
V. (x) dx dz.
-00
(13)
17

---

Finally, we show that any 9-independent additive baseline (highlighted in the equation) does not change the gradient calculation:
/ D(2)
(xc (x) dx dz
/ D(2)
/ x. (x) dx dz
[ D(2) / PV)P da dz
1 p (x)d2=)
S/ec ((2) L x. <VoP (x) dr dz
[ ((2) ..)...... b. Vo (ʃ- (e( dx) dz
J D(z) 0
x V (o (z)
dx dz
0 (o).
(14)
which completes the proof. Notice that we used the identity Vo I pe (x)dx = Vol = O; this does
not hold for analogous term in the standard RL settings Eq. Va (2)(()) pa (x))da, whose gradient depends on V Vogh (z) according to the Leibniz integral rule.
B Proof of Proposition 1
Proof. Recall that by Eq. G 1
Ei=l R,sag Em= Him Denoting G, Em= mim and
aN
substituting d% = qh we have
1 N
G = N
Σ
Ri G..
Expectation:
Since [G:} are i.i.d, and using the law of total probability, we obtain
Epla L Sn G.j =a (a -1 -1 Ep[G, Ri S c.l)) + +1 a) (a- EOEE R, > 2.ll)
=Ep G1 Rí S )
By switching the task sample distribution to Dd and using the definition of D simply have
Eng [ar - - 1 R. 50 Gi| = -E [Gi] a "Ep[G Ri aa]-
Together, we obtain Dg [aG] Ep [G] as required.
Variance:
For the original distribution, since {Gi} are i.i.d, have
N Var) VarD(a '1 5gg G1)
Epla 21 RR sog G9) - Ep(a ' sn Gi]²
Eplua 0 R1 < "1 Ep/aa G
G₁ R2 S .12
a -B [G - EDy [G
2 a (Bng [G]] - Eng [G1]3)
=a - (G1).
For the tail distribution Dd however,
N. Var pg (aG) a Var (a 11 G1) Var (G1).
which completes the proof.
18

---

C
The Cross Entropy Method
C.1
Background
The Cross Entropy Method (CEM, de Boer et al. (005 general approach to rare-event sampling and optimization. In this work, D. As described in Section [] the CEM repeatedly samples from the parameterized distribution Dø. and updates & according the f-tail of the sampled batch. Since every iteration focuses on the tail of its former, we intuitively expect exponential convergence to the tail of the original distribution. While theoretical convergence analysis does not guarantee the exponential rate [de Mello and Rubinstein 2003], practically, the CEM often converges within several iterations. For clarity, we provide the In this version, the CEM repeatedly generates samples from the tail of the given distribution D
its sampling version to sample high-risk tasks from the tail of
pseudo-code for the basic CEM in Algorithm 3
Algorithm 3: The Cross Entropy Method (CEM)
Input: distribution ; score function R; target level q; batch size N; CEM quantile B.
f- 6
3 while true do
// Sample
Sample 2 DAY
Wi +- Doo (z)/Do(z)) (1
Print = // Update
q' E max (q, Q8 ((R(2))
8
s + argmaxy Ei=1 wil R()))s log (zi
C.2 Discussion
The CEM is the key the flexible robustness level of RoML (Algorithm Pp: can learn to sample not only the single worst-case task, but all the tasks with the lowest returns.
The CEM searches for task distribution within parametric family of distributions. This approach can handle infinite task spaces, and learn the difficulty of tasks in the entire task space from a mere finite sample of tasks. For example, assume that the tasks correspond to environment parameters that take continuous values within some bounded box (as in Section 6.2]and and Section 6.3). The CEM can fit distribution over an entire subset of the box- from mere finite batch of tasks. This property lets the CEM learn the high-risk tasks quickly and accelerates the meta-training, as demonstrated in Section 6 and Appendix D33
On the other hand, this approach relies on the structure in the task space. If the tasks do not have natural structure like the ones in the bounded box, it not trivial define the parametric family of distributions. This is the case in certain supervised meta learning problems. For example, in the common meta-classification problem [Finn al. 2017], the task space consists of subsets of classes to classify. This is a discrete space without a trivial metric between tasks. Hence, it is difficult for the CEM to learn the a lowest-return tasks from a finite sample. Thus, while RoML is applicable to supervised meta learning as well as MRL, certain task spaces require further adjustment, such as meaningful embedding of the task space. This challenge is left for future work.
D Experiments: Detailed Settings and Results
D.I Khazad Dum
At the end of the hall the floor vanished and fell to an unknown depth. The outer door could only reached by a slender bridge of stone, without kerb or rail, that spanned the chusm with one curving spring of fifty feet. It was ancient defence of the Dwarves against any enemy that might capture the First Hall and the outer passages. They could only pass across it in single file. [Tolkien] 1954
19

---

The detailed settings of the Khazad Dum environment presented in Section 6.1 are as follows. Every task is carried for K 4 episodes T 32 times steps. The return corresponds to the undiscounted sum of the rewards (7- = 1). Every time step has a cost of /" points if the I. L.1-distance of the agent from the target larger than 5; and for distances between and 5, the cost varies linearly between and 1/ I/T. By reaching the destination, the agent obtains reward of 5/T, and has no more costs for the rest of the episode. By falling to the abyss, the agent can no longer reach the goal and is bound to suffer cost of 1/"T for every step until the end of the episode. Every step, the agent observes its location (represented using a soft one-hot encoding, similarly to Greenberg et al. (2022) ) and chooses whether move left, right, up down. If the agent attempts to move into wall, remains in place.
The tasks are characterized by the rain intensity, distributed т Exp(0.1). The rain only affects the episode when the agent crosses the bridge: then, the agent suffers from an additive normally- distributed action noise (0, r²), addition
direct damage translated into a cost of 3 т. For RoML, the CEM is implemented over the exponential family of distributions Exp(() with Φo 0.1 and / = 0.05. In this toy benchmark we use no regularization (v 0).
In addition to the test returns throughout meta-training shown in Fig. 3 Fig. displays the final test returns the end of the meta-training, over 30 seeds and 3000 test tasks per seed.
KhazadDum
KhazadDum
CVaRo.n return
return
-0.1
-0.2
0.2
0.4
-9.3
-0.4
VariBAD
RoML method
CVaR-ML
BAD
RaML method
CVaR-ML
Figure 8: Khazad-Duma Dumm: Mean and CVaR returns over seeds and 3000 test tasks.
D.2 Continuous Control
In all the MuJoCo benchmarks introduced in Section 6.2 each task's 's meta-rollout consists of episodes x 200 time-steps per episode. Below deseribe the task distributions, as well as their parameterization for the CEM. In each benchmark, we used CEM quantile = 0.2 and regularization of 0.2 samples per batch.
HalfCheetah- Vel: The original task distribution is uniform 7 ~ U 7) in HalfCheetah
(the task space [0, 7] was extended in comparison to Zintgraf et al. [2019]. to create more
significant tradeoff between tasks). We rewrite it as 7 77, F ~ Beta(2p. 2 - 26) with do 0.5 (leading to Beta(1, 1), which is indeed the uniform distribution). The CEM learns to modify 6. Notice that this parameterization satisfies E.,, [7] Φ.
Humanoid-Ve Vel: Same as HalfCheetah- Vel, with task space [o, 2.5] instead of [o, 7].
The target location random within circle of radius 5. We represent the target location in polar coordinates, and write r ~ Beta 262)1 and 6 ~ Beta(202. 2-2 (up to multiplicative factors 5 and 2T). The original distribution parameter @o (0.5,0.5) and the CEM learns to modify it.
Ant-Goal:
Humanoid- Ant-Mass: The task + E [0.5, corresponds to the multiplicative factor of the body mass (e.g., 7 is a doubled mass) The original task distribution is uniform over the log factor, i.e., log, re- re-parameterize the uniform distribution as Beta, and learn modify its parameter.
U(-11, 11). Again, we
HalfCheetah-Body, Humanoid-Body Ant-Body: The components of the task correspond to multiplicative factors of different physical properties, and each of them is distributed independently and uniformly log, i.e., V1 < S log, Tj ~ U([-1,1) 1]). We re-parameterize this as 3 independent Bela distributions with parameters
(Φ), dh2, da).
HalfCheetah 10D-task: Again, the task components correspond to multiplicative factors of different properties of the model. This time, there are 10 different properties (i.e., the task space is 10-dimensional), but each of them varies in a smaller range: logo T
20

---

Table Mean return over 1000 test tasks, for different models and MuJoCo environments. Standard deviation is presented over 30 seeds. CVaR returns are displayed in Table
HalfCheetah Mass
HalfCheetah 10D-task
Vel
Body
(a)
(b)
(c)
CeSoR
1316
18
1398
1008
1222 +:
1388 +: 20
1274 32
PAIRED
-545 + 55
662 + 30
192 51
551 + 53
706 36
561 - 65
CVaR-ML
-574 22
113 + 8
193 + 6
263 + 15
250 - 11
192 -
PEARL
534 15
1726 + 13
1655 6
1843 +
1866 - 13
1425 6
VariBAD
-82
1558
1 32
1616
1984 1
1617
12
RoML (VariBAD)
95 +
1581
32
1582 +
1819
1950 20
1616 + 13
RoML (PEARL)
519 + 15
1553 + 18
1437 8
1783 7
1859 + 10
1399 8
Humanoid Mass
Ant Mass
Vel
Body
Goal
Body
VariBAD
880 4
1645 22
1678 + 17
229 3
1473 #
1476 1 1472
RoML (VariBAD)
883 + 4
1580 + 17
1618 + 18
2224 3
1475 + 2
U([-0.5,0.5] Each such property vector, and is multiplied by Тj when executing the task т. The 10 properties are selected randomly, among all the variables of type float ndarray in env.model. We generate such MRL environments with different random sets of 10 lask-variables each. Some examples for properties are inertia, friction and mass.
In the experiments, we rely on the official implementations of VariBAD [Zingrraf et al.) 2019 and PEARL [Rakelly et al. 2019], both published under the MIT license. CVaR-ML and RoML are implemented on top of these baseline, and their running times are indistinguishable from baselines. All experiments were performed on machines with Intel Xeon 2.2 GHZ CPU and NVIDIA's V100 GPU. Each experiment (meta-training and testing) required 12-72 hours, depending on the environment and the baseline algorithm.
Table and Fig. present detailed results for our MuJoCo experiments, addition the results presented in Section 6.2
21

---

HaliChnetahhiel
HaliChostahRody
HallCheetahRedy
HallCheetahRoddy
-50
2000
VeritAD (1610)
return
return
RoML (1582)
-.00
2300
VerIAAD (1616)
1.00
VenBAD (1616)
VariBAD (-82)
1250
HoMiL (1382)
HoMI (1382)
2315
(-95) (-95)
J.g0al
mass
damping
head Size
(a) HalfCheetah- Vel
(b) HalfCheetah--ool (mass, damping and head-size)
Humaaniidvl
HumanoidBady
HumanoidBody
HumancidBody
902-
return
retum
VeritAD (1670)
VariBAD
RoML
1600
VariBAD (1678)
2000
VariBAD (1678)
(3610)
1425
RoML (1618)
0003
d
V_goal
damping
head size
(c) Humanoid- Vel
(d) Humanoid-Boo (mass, damping and head-size)
AntMass
Ant
AntGool
1850
return
2680
return
redunn
200
VariBAD (-224)
-200
VarIBAD (1GA5)
VariBAD (3473)
VoriAAD 1 2222)
2400
RoML (1580)
ROMIL (3475)
RoMil (-224)
theta
mass
MOsS
(c) Hormanoid-Mass
Ant-Mass
(g)
Ant-Goal
Figure 9: Average return per range of tasks in the various MuJoCo environments (global average in parentheses). HalfCheetah-Mass is displayed Fig.
D.3 The Cross Entropy Method
We implemented RoML using the Dynamic Cross Entropy Method implementation of Greenberg 2022 Below we concentrate results related to the CEM functionality in all the experiments:
Learned sample distribution:
One set of figures corresponds to the learned sample distri- bution, as measured via é throughout the meta-training.
Sample returns: A second set of figures corresponds to the returns over the sampled tasks (corresponding to Dz)): ideally, we would like them to align with the o-tail of the reference returns (corresponding to Do.). Thus, the figures present the mean sample return along with the mean and CVaR reference returns (the references are estimated from the sample returns using Importance Sampling weights). In most figures, we see that the sample returns (in green) shift significantly from the mean reference return (in blue) towards the CVaR reference return (orange), at least for part of the training. Note that in certain environments, the distinction between difficulty of tasks can only be made after the agent has already learned basic meaningful policy, hence the sample returns do not immediately deviate from the mean reference.
KhazadDum
KhazadDum
E[task noise]
-0.4
0.4
return
-0.6
Reference mean
0.3
Reference CVaRys
Sample mean
0.1
10
20
30
40
50
0
30
20
30
CEM iteration
iteration
(a) Learned sample distribution
(b) Sample returns
Figure 10: The CEM in Khazad-Dum. Note that the effect of the CEM concentrates at the first half of the meta-training; once the meta-policy learns focus the long path, the agent becomes invariant to the sampled tasks, and the sampler gradually returns the original task distribution do.
22

---

hatchoctatine
Ha Musi
Hamanoo phoass
MiCppe) banotteell
1080
(a) HalfCheetah--ee Vel
(b) HallCheetah--eee Mass
(c) HalfChhetah-
(d) Humanoid- -Mas
Figure 11: Learned sample distribution in MuloCo benchmarks. Notice that for HallCheetah Body, the CEM
has to control 3 different task parameters simultaneously.
PAAECC Ceete Bods
Humanoodttars
Meloor Foir
Refirerte PRASO
Deration
(a) HalfCheetah- Vel
(b) HalfCheetah-Mann
(c) HalfCheetah-Bod
(d) Humanoid-Me -Mass
Figure 12: Sample returns MuJoCo benchmarks.
mreah ampiibude
mos Reeueeee
Score
CVaRos (8)
Sample mean
200
12:0
250
623
300
200
459
1003
-2
480
500
3000
GEM ller 417271
CCM Letaliio
CEM fer alimm
CEM Rltelion
Figure 13: Sine Regression: The 3 left figures show the learned sample distribution, corresponding average amplitudes. phases and frequencies. The CEM immediately learns that the amplitude the strongest effect on test loss, whereas frequency has a moderate effect and phase has none. The right figure shows in green the mean sample scores (in supervised learning these are the negative losses instead of the returns).
D.4 Ablation Test: RoML with Naive Sampler
RoML learns to be robust to high-risk tasks by using the CEM to over-sample them throughout training. The dynamic CEM presented in Section is designed to identify and sample high-risk tasks from a possibly-infiiit task-space, where the task values change with the evolving policy throughout training.
In this section, we conduct an ablation test to demonstrate the importance of the CEM to this task. To that end, implement a naive adversarial task sampler. The first M 100 tasks are sampled from the original distribution D, and the naive sampler memorizes the aM lowest-return tasks, and samples randomly from them for the rest of the training.
As displayed in Table switching to the naive sampler decreases the CVaR returns significantly.
23

---

Table Ablation test: CVaRg.os return, compared the naive sampler baseline.
HalfCheetah Body
HalfCheetah 10D-task
(b)
(c)
VariBAD
835 + 30
1126 = 6
1536 + 39
988 + 13
Naive sampler
839 20
1056 = 34
1340 = 57
978 + 12
RoML (VariBAD)
935 17
1227 =
1697 + 24
999 +
E
Supervised Meta-Learning
Below we provide the complete details for the toy Sine Regression experiment of Section 6.3 The
input in the problem is æ E [0, 2 ) the desired output is y = A (in + b), and the task is defined by the parameters T (A, w.) distributed uniformly over ? [0.1, 5) [0, 2T] [0.3,3] Similarly to Finn et al. [017]] J. the model is tne-tuned for each task via a gradient-descent optimization step
over 10 samples [ y/ )= and is tested on another set of 10 samples. The goal is find model
weights that adapt quickly to new task data.
CVLR-ML. and RoMI. are implemented with robustness level of a 0.05, on top of MAMI. For the CEM of RoML, we re-parameterize the uniform task distribution using Beta distributions (see CEM details below).
As shown Fig. 13 RoML learns to focus on tasks (sine functions) with high amplitudes and slightly increased frequencies, without changing the phase distribution. Fig. d4diplay the test losses over 30 seeds, after meta-training for 10000 tasks. Similarly to the MRL experiments, again RoML achieves better CVaR losses than both CVaR-ML ML and the baseline.
mean loss
4.0
model
MAMIL
RoML
CVaRa.os loss
MAML
CUSR, ML
3.5
12
2.5
tuning iteration
tuning iteration
(a) Mean
(b) CVaRo.05
Figure 14: Sine Regression: Mean and CVaR losses 10000 test tasks, against the number of tuning gradient-steps test time. The 95% confidence intervals are calculated over 30 seeds.
CEM implementation details: In comparison to Finn et al. 2017], we added the sine frequency as third parameter in the task space, since the original problem was to simplistic to pose mean/CVaR tradeoff. The tasks are distributed uniformly. and we reparameterize them for the CEM using the Beta distribution, similarly to Appendix D.2
(Ty}²= Tj ~ Beta (26, 2 2(²)). On top this, we add linear transformation from the Beta distribution domain [o, 1] to the actual task range ([0.1, 5] for amplitude, [0, 2r]] for phase and |0.3, 3] for frequency). Note that the original uniform distribution recovered by do = (0.5, 0.5, 0.5). The parameter d; E [o, 1], which controlled by the CEM, equals the expected task E,-n, [T.]. The other hyper-parameter of RoML are set to 3 = 0.2 (CEM quantile) and v = 0 (no regularization).
24


Reviews:
[{"review_id": "sgvSrv8HWC", "reviewer": "Reviewer_ib31", "summary": " This paper aim to address the limitations in the standard meta reinforcement learning (MRL) methods, where they optimize the average return over tasks and often struggle with high-risk or difficult tasks, hence limiting system reliability. To this end, the authors define a robust MRL objective with a controlled robustness level to overcome these limitations. Meanwhile, this work proposed RoML with provable gradient bias reduction and improvement on the data efficiency. Overall, this paper is well-organized and presented. The proposed method is promising with experimental evidence. They show that RoML achieves robust returns, indicating improved performance and reliability in challenging scenarios. By combining the controlled robustness of the MRL objective and the oversampling approach of RoML, this paper contributes to advancing the field of RL in real-world applications with variable and unknown test tasks.", "questions": "1. Does the rollout length matter to the performance? Do you choose different rollout length for different tasks?\n2. Does over-sampling on the low-return tasks increase the computation complexity?\n3. If the task distribution changes overtime, does the over-sampling strategy still be able to guarantee the robustness?", "limitations": "The author has discussed the limitations of this work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To address the impact of the biased gradient issue in standard RL, this work proposed to use CVaR in the objective and provides the theoretical guarantee (Theorem 1)\n2. The proposed CVaR-ML has improved data efficiency by a factor up to $1/\\alpha$\n3. The conducted experiments on both risk-averse, risk-neutral baselines shows promising results of RoML.", "weaknesses": "1. The proposed algorithm RoML requires the proper set up of parameter $\\alpha$ and $\\beta$, it will be beneficial to discuss more on the choice of those parameters for different tasks? e.g., do you use grid-search for choosing those parameters?  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. Please see our responses below.\n\n**RoML requires the proper set up of parameter alpha and beta.**\n\nThe parameter alpha is part of the problem definition and it is not a hyper-parameter of RoML. It expresses the desired robustness level, which is analogous to the uncertainty-set size in standard robust RL.\n\nBeta is indeed a hyper-parameter and can be set using hyper-parameter tuning on a validation set. In practice, the value of $\\beta=0.2$ (taken from the CEM of Greenberg et al. 2022) worked well for us. We will clarify this issue in the revised version.\n\n**Do you choose different rollout length for different tasks?**\n\nBoth the number of episodes-per-task ($K$) and episode lengths are constant per benchmark in our work. For most benchmarks, we just set them according to the MuJoCo benchmarks of VariBAD (Zintgraf et al, 2019).\n\n**Does over-sampling on the low-return tasks increase the computation complexity?**\n\nThe additional complexity is negligible. We oversample low-return tasks **instead** of other tasks, so  RoML does not require additional samples (this is the key advantage of RoML over CVaR-ML). The task selection itself relies on the CEM, whose additional running-time overhead is negligible (see Line 304).\n\n**If the task distribution changes overtime, does the over-sampling strategy still be able to guarantee the robustness?**\n\nTo some extent. The CEM searches for low-return tasks in a given task distribution $D$. It yields the CEM’s sample-distribution $D_\\phi$. You could change $D$ to some $D’$ in the middle of training, and the CEM would adapt to $D’$ in time (via the importance-sampling weights) – as long as $D$, $D’$ and $D_\\phi$ all have the same support. Notice that in Appendix D, we indeed defined all the CEM distribution-families to have constant supports. Finally, if $D$ changes repeatedly, the CEM may still catch up if the change is slower than the CEM’s convergence rate. This is an interesting problem for future work, and we will add this discussion to the last section. Thanks!\n"}, {"review_id": "ab1YQF4yyz", "reviewer": "Reviewer_CM4Y", "summary": "The paper presents a novel approach to robust Meta-Reinforcement Learning (MRL), aiming to improve the consistency of returns across tasks, crucial for many RL applications. The authors introduce Conditional Value-at-Risk (CVaR) optimization to MRL, and propose a new method called Robust Meta RL algorithm (RoML). This approach overcomes issues of biased gradients, common in CVaR optimization in RL, and increases sample efficiency by identifying and over-sampling lower return tasks. Experimental results support the effectiveness of RoML, showing improved robustness over baseline algorithms in various domains.", "questions": "1. Could you elaborate on the conditions under which RoML improves the average return? Does this improvement come at the cost of performance in other aspects?\n\n2. Could you provide more clarity on how task selection during training works with the RoML algorithm?", "limitations": "This paper includes a few sentences of the main limitations. I suggest authors discuss more on this. \nI don't find any obvious societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths**:\n\n1. The use of CVaR optimization in Meta RL is a novel idea that addresses the issue of robustness, a common challenge in reinforcement learning applications.\n\n2. The theoretical proof that MRL is immune to biased gradients, a common issue in CVaR optimization in RL, is a significant strength.\n\n3. Experimental results highlight the robustness and efficiency of the RoML method across various domains.", "weaknesses": "**Weaknesses**:\n\n1. The RoML algorithm assumes that tasks can be selected during training. This might not always be feasible in real-world applications, potentially limiting its general applicability.\n\n2. While the paper mentions that average return is improved in certain experiments, it does not provide a clear discussion on this, leaving questions on how RoML impacts average return across different applications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. Please see our responses below.\n\n**Could you elaborate on the conditions under which RoML improves the average return?**\n\nThis result is empirical, and we specify it mostly to show that optimizing the mean and the CVaR are not necessarily contradicting. That is, sometimes low-return tasks can be improved at the cost of high-return tasks, but without hurting average performance. This leaves the question: why does the mean improve? We hypothesize that focusing on hard tasks forms a learning curriculum, which has been shown to help learning sometimes. We will discuss this hypothesis in the manuscript. Thanks!\n\n\n**Could you provide more clarity on how task selection during training works?**\n\nTask sampling relies on the Cross-Entropy Method (CEM). It is discussed in Section 5 and we also provide a simplified presentation including pseudo code in Appendix C1. \nThe main idea is that we sample $N$ tasks and generate rollouts for them. Then, we take the $\\beta\\cdot N$ tasks with worst returns, and update the sample distribution accordingly (we define a class of task-distributions in advance, and fit a distribution from the class to these worst-returns tasks). Since $\\beta>\\alpha$, fitting $\\beta\\cdot N$ tasks is less prone to overfit. By repeatedly fitting the worst tasks of the *current* distribution, we eventually reach the extreme alpha-tail tasks of the *original* distribution, as desired.\n\n**I suggest authors discuss more on the limitations.**\n\nAs suggested in the review, we will discuss the limitations of the methods in more detail. In particular, we will connect the discussion in Lines 242-246 with Lines 374-379, for a more coherent presentation of the limitations. Thanks!\n"}, {"review_id": "Y2LHoaaySp", "reviewer": "Reviewer_vMWn", "summary": "This paper focuses on the meta-reinforcement learning setting, where the agent learns a meta-policy that adapts to new tasks with few observations. However, traditional meta-RL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty. This may make the agent fail to generalize to risk tasks. In this paper, the authors introduce CVaR into the meta RL framework to solve this problem. They theoretically prove that an unbiased gradient estimator is available in the meta RL setting, which allows a general critic module. They then propose the Robust Meta RL algorithm (RoML) algorithm to generate a robust version of any given MRL algorithm, by identifying and oversampling harder tasks throughout training. Experiments on the gym locomotion tasks demonstrate the advantages of the proposed method.", "questions": "Points 1 and 2 of the weaknesses are my questions.", "limitations": "The limitation of the proposed method is discussed in the conclusion section. The algorithm requires control over the selection of training tasks. This assumption is acceptable since it is possible to define and control different tasks in both simulation and real-world cases.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written and easy to follow. The figures are clear and illustrative for understanding the idea.\n2.\tThe theoretical analysis shows evidence to support the advantages of the proposed method.\n3.\tThe experiments also support the claim that the proposed method achieves better adaptation to unseen tasks.\n", "weaknesses": "1.\tI am a little confused about the setting of robust meta-reinforcement learning. What does robustness mean in meta RL? Is there any disturbance to the dynamic system or observation? It seems that the setting mentioned in this paper is that some tasks are harder than others.\n2.\tThe main contribution of this paper is extending the CVaR-PG framework to meta RL setting and eliminating the limitation of the biased gradient estimator. However, the intuition behind the difference between using CVaR in RL and meta-RL settings is not clear. If the task space is not discretized (as mentioned in the caption of Figure 1), the boundary of RL and meta RL will be unclear. So we can still have both easy and hard situations in the RL setting and assume there is an unknown distribution of the difficulty. I think the authors should give some hints in the introduction section about why using CVaR in meta RL is better than in RL. Does the advantage come from the discretized task indicator?\n3.\tThe experimental evaluation is not thorough enough. First, the experiments do not use standard meta-RL benchmarks, such as Meta World. Second, the experimental setting is not consistent with the motivation “Standard MRL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty.” There are no particular high-risk or difficult tasks in the environments （a high-risk task is defined as the high mass in the mujoco environment）. The example in Figure 1 looks good but is not mentioned in the experiment part. Third, potential simple baselines are missing. For example, oversampling the hard task according to the return. This could be a general trick for all meta-learning methods, so I think it should be a fair baseline.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and helpful feedback. Please see our responses below, as well as the new experiments we added following the review.\n\n**What does robustness mean in meta RL?**\n\nWe defined the robustness of an agent in meta RL as the CVaR return of the agent. High CvaR returns express robustness to the selection of tasks, as it results in a reasonable return even under adversarial tasks. This is analogous to robustness in RL, where we aim to perform well under adversarial deviation of the underlying model. In addition, interpreting the CVaR as robust optimization is supported by Chow et al. 2015: they showed that optimizing CVaR is equivalent to robust RL under a certain uncertainty set. To clarify our suggested notion of robustness, we will add this discussion to the intro, thanks!\n\n**Potential simple baselines are missing.**\n\nThe submission contained two sample-based baselines, PAIRED [Dennis et al. 2020] and CeSoR [Greenberg et al. 2022]. Following this comment, **we now conducted experiments with an additional baseline, which will be added to the manuscript.** The naive sampler baseline memorizes the alpha% tasks of lowest returns, among the first batch of 100 sampled tasks, and then focuses the training on these lowest-return tasks. As presented below, this resulted in significantly lower CVaR returns:\n\n|                |     Cheetah-Body     |  Cheetah-10D-task (a) |  Cheetah-10D-task (b) | Cheetah-10D-task (c) |\n|:--------------:|:------------:|:-------------:|:-------------:|:------------:|\n| VariBAD        | $835 \\pm 30$ |  $1126 \\pm 6$ | $1536 \\pm 39$ | $988 \\pm 13$ |\n| Naive sampler  | $839 \\pm 20$ | $1056 \\pm 34$ | $1340 \\pm 57$ | $978 \\pm 12$ |\n| RoML           | $935 \\pm 17$ | $1227 \\pm 13$ | $1697 \\pm 24$ | $999 \\pm 20$ |\n\nWe explain the advantage of RoML by addressing the following challenges:\n* Generalization to unseen tasks: in MuJoCo’s continuous task spaces, with infinite tasks, memorization of tasks is inherently limited. By contrast, the CEM modeling of RoML captures the structure of the task space, and learns which tasks are harder - even for tasks that have not been observed yet.\n* Adaptation to the ever-changing policy: the lowest-return tasks at the beginning of training do not necessarily remain so throughout the whole training. RoML’s CEM adapts continuously to the observed returns, and also keeps sampling some high-return tasks - in case that their returns will drop at some point.\n\nWe can extend the experiments beyond the 4 environments presented above, if the reviewer considers it beneficial.\n\n**The experiments do not use standard meta-RL benchmarks, such as Meta World.**\n\nOur primary experiments follow and extend the standard MuJoCo MRL benchmarks from Zintgraf et al, 2019 (VariBAD). As discussed above, these MuJoCo benchmarks present infinite, continuous, but structured task spaces, which pose an additional challenge, as the low-return tasks cannot be learned by naive memorization. Thus, we found these environments more suitable for testing of RoML.\n\n**The experimental setting is not consistent with the motivation.**\n\nFirst, the toy Khazad-Dum benchmark does follow a disaster-prevention motivation, aiming not to fall off the bridge. Second, our general motivation is to minimize the risk of very low returns. RoML learns where this risk comes from. In some experiments, poor returns turned out to be caused by high body-mass, thus RoML learned to focus on such tasks, which we see as consistent with the motivation.\n\n**The difference between using CVaR in RL and meta-RL settings is not clear… Does the advantage come from the discretized task indicator?**\n\nIn RL, when the policy changes, this affects both (a) the return of high-risk situations, and (b) the probability to get to these situations.\n\nIn MRL, when the policy changes, this only affects (a) the task return, while (b) the task probabilities remain the same (since the task distribution does not depend on the policy).\n\nSince (b) is the cause for the gradient bias in RL (as illustrated in Figure 5 in Appendix A), the bias disappears in MRL (illustrated in Figure 6). \n\nThis is the substantial difference between robustness in RL and in MRL, and it holds for both discrete and continuous task spaces.\nWe will stress this root difference in the front manuscript, in addition to Appendix A.\n"}, {"review_id": "Ii24J9ABGy", "reviewer": "Reviewer_FerX", "summary": "This is an interesting paper that harnesses a risk-averse optimization from RL and applies the trick to Meta RL (called RoML). It has a theory section that proves that biased gradients in ordinary robust RL objectives are no longer biased when applied to meta RL, an interesting result. there is also a data inefficiency problem that arises in robust RL objectives because out of a batch of N trajectories, they can only make use of a small proportion (alpha) of them. The authors importance sample the set of tail tasks, recovering sample efficiency. Altogether, the authors discuss several key problems that are present in vanilla RL which disappear in meta RL. \n\nThe authors demonstrate experimentally on a variety of tasks the benefits of RoML over baselines.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper discovers the cool trick that whereas CVaR has problems associated with its biased estimate of gradients in ordinary RL, these problems are resolved in Meta RL, which is a cool finding backed by theory. The intuition is easy to understand. The experiments are solid.", "weaknesses": "I have no concerns. The authors demonstrate strong performance on a variety of tasks, demonstrating its versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our work, and are encouraged that the reviewer found it cool, intuitive and solid."}]

Author rebuttal:
We are excited that all the reviewers recommended accepting the paper, and thank them for their helpful comments.

The reviewers found that our work **“contributes to advancing RL in real-world application”** (ib31) and is **“crucial for many RL applications”** (CM4Y). They consider our **theoretical guarantees** in providing unbiased gradients a **”significant strength”** and a **”cool finding”** (CM4Y, FerX). The **improved sample efficiency** is also highlighted (ib31, CM4Y, FerX). The reviewers found that our experiments are **“solid”** (FerX) and **“highlight the robustness and efficiency”** (ib31, CM4Y), and concluded that our method **”achieves better adaptation”** (vMWn) – on **“a variety of tasks”** (CM4Y, FerX) including **”challenging scenarios”** (ib31). Finally, the reviewers have found the paper **“well-written”** (ib31, vMWn).

Our responses to the reviewers’ comments are below.

Meta-review: **Summary:** This paper presents a novel formulation of robust meta-RL, where the standard objective of expectation (of the value) over the tasks is replaced by a CVaR objective. This, however, immediately poses computational challenges.  In particular, if one has to use policy optimization methods, then a well-defined policy gradient that can be estimated from trajectory samples is needed. The first main contribution of this paper is such a meta policy gradient theorem. The paper then proposes a variance reduction scheme, and a practical algorithm called Robust Meta RL (RoML). The paper presents experimental results that illustrates the performance of the proposed algorithm. 

**Comments:** The review scores are 6, 6, 6, 6, and average is 6.  The paper is technically solid, and gives meta policy gradient theorem, a variance reduction approach, and a practical algorithm called Robust Meta RL (RoML). The paper presents experimental results that illustrates the performance of the proposed algorithm by comparing it against a non-robust meta RL algorithms, and (non-meta) risk averse algorithms. Most of the reviewer comments were regarding the experiments and baseline, and the authors have given detailed explanation during rebuttal, including the addition of another baseline. During the discussion period, the reviewers did not raise any additional comments. 

Please revise the paper by including the all the changes you promised during the rebuttal, such as additional simulations, experiments with new baselines, more clarification about certain aspect, and a discussion about the limitations. 

Thank you!

AC
Decision: Accept (poster)