Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design
Authors: Matthew Thomas Jackson
Minqi Jiang
Jack Parker-Holder
Risto Vuorio
Chris Lu
Gregory Farquhar
Shimon Whiteson
Jakob Nicolaus Foerster
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Meta-Learning
Meta-RL
Meta-Optimization
Policy Meta-Optimization
Environment Design
Unsupervised Environment Design
Auto-Curricula
TL;DR: Regret-based environment design improves the performance of meta-optimizers when generalizing to unseen environments.

Abstract: The past decade has seen vast progress in deep reinforcement learning (RL) on the back of algorithms manually designed by human researchers. Recently, it has been shown that it is possible to meta-learn update rules, with the hope of discovering algorithms that can perform well on a wide range of RL tasks. Despite impressive initial results from algorithms such as Learned Policy Gradient (LPG), there remains a generalization gap when these algorithms are applied to unseen environments. In this work, we examine how characteristics of the meta-training distribution impact the generalization performance of these algorithms. Motivated by this analysis and building on ideas from Unsupervised Environment Design (UED), we propose a novel approach for automatically generating curricula to maximize the regret of a meta-learned optimizer, in addition to a novel approximation of regret, which we name algorithmic regret (AR). The result is our method, General RL Optimizers Obtained Via Environment Design (GROOVE). In a series of experiments, we show that GROOVE achieves superior generalization to LPG, and evaluate AR against baseline metrics from UED, identifying it as a critical component of environment design in this setting. We believe this approach is a step towards the discovery of truly general RL algorithms, capable of solving a wide range of real-world environments.

Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design
Matthew Jackson" University of Oxford
Minqi Jiang UCL
Jack Parker-Holder Google DeepMind
Risto Vuorio University of Oxford
Chris Lu University of Oxford
Gregory Farquhar Google DeepMind
Shimon Whiteson University of Oxford
Jakob N. Foerster
University of Oxford
Abstract
The past decade has vast progress in deep reinforcement learning (RL) the back of algorithms manually designed by human researchers. Recently, it has been shown that it is possible meta-learn update rules, with the hope of discovering algorithms that can perform well on wide range of RL tasks. Despite impressive initial results from algorithms such Learned Policy Gradient (LPG), there remains generalization gap when these algorithms are applied to unseen environments. In this work, we examine how characteristics of the veea-trainnng distribution impact the generalization performance of these algorithms. Motivated by this analysis and building on ideas from Unsupervised Environment Design (UED), we propose a novel approach for automatically generating curricula to maximize the regret of meta-learned optimizer, in addition novel approxima- tion of regret, which we name algorithmic regret (AR). The result is our method, General RL Optimizers Obtained Via Environment Design (GROOVE). In se- ries of experiments, we show that GROOVE achieves superior generalization LPG, and evaluate AR against baseline metrics from UED, identifying as a critical component of environment design in this setting. We believe this approach is step towards the discovery of truly general RL algorithms, capable of solving a wide range of real-world environments.
GROOVE over LPG (Grid-World Atari transfer)
10a
Percentage Improvement
104
-10
100
110
Figure 1: Out-of-distribttion performance on Atari after meta-training exclusively on World levels, our method (GROOVE) significantly outperforms LPG on Atari. Improvement is measured as percentage of mean return over 5 seeds.
"Correspondence to jackson@robots. Connepponeeeettoccccceeerbbbbb
ax.ac. uk.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
The past decade has seen vast progress in deep reinforcement learning [Sutton and Barto
1998 RL].
paradigm whereby agents interact with environment maximize scalar reward. In particular,
deep RL agents have learned to master complex games (Silver et al.] 20166] 2017 Berner et al. 2019
control physical robots [OpenAl et al., 2019 Andrychowicz al. 2020 Miki al.) 2022 and
increasingly solve real-world tasks [Degrave et al. 2022]. However, these successes have been driven algorithms, which have been refined over many years to
by the development of manually-designn tackle new challenges in RL. As a result, these methods do not always exhibit the same performance
when transferred to new tasks [Henderson al. 2018 Andrychowicz al. 2021] and are limited by our intuitions for RL.
Recently, meta-learning has emerged promising approach for discovering general RL algorithms in a data-driven manner [Beck et al. [2033]] J. In particular, Oh et al.] [2020) introduced Learned Policy Gradient (LPG), showing is possible meta-learn an update rule on toy environments and transfer it zero-shot to train policies on challenging, unseen domains. Despite impressive initial results, there remains significant generalization gap when these algorithms In this work, we seek to learn general and robust RL algorithms, by examining how characteristics of the meta-training distribution impact the generalization of these algorithms.
applied to unseen environments.
Motivated by this analysis, our goal is to automatically learn a meta- veeaa-tainnng distribution. We build on ideas from Unsupervised Environment Design [Dennis al. 2020 UED], paradigm where a student agent trains on an adaptive distribution of environments proposed by a teacher, which seeks to propose tasks which maximize the student's regret. UED has typically been applied to train single RL agents, where it has been shown to produce robust policies capable of zero-shot transfer challenging human-designed tasks. Instead, we apply UED to the meta-RL setting of meta-learning policy optimizer, which we refer to as policy meta- melaa-ptimmizttonn (PMO). For this, we propose algorithmic regret (AR), novel metric for selecting meta-training tasks, in addition to method building on LPG and ideas from UED. We name our method General RL Optimizers Obtained Via Environment Design, or GROOVE.
We train GROOVE on an unstructured distribution of Wordd environments, and rigorously variety of unseen tasks- ranging from challenging fridd-srrdds to
examine its performance Atari games. When evaluated against LPG, GROOVE achieves significantly improved generalization compare AR against prior environment design
performance all of these domains. Furthermore, metrics proposed in UED literature, identifying it as a critical component for environment design in this setting. We believe this approach is step towards the discovery of truly-general RL. algorithms, capable of solving wide range of real-world environments.
We implement GROOVE and LPG in JAX [Bradbury et al. 2018), resulting in meta-training time of 3 hours on single V100 GPU. As well as being the first complete and open-source implementation of LPG, achieve major speedup against the reference implementation, which required 24 hours on a 16-core TPU-v2. This will enable academic labs to perform follow-up research in this field, where compute constraints have long been limiting factor.
Updato optimizer
Agents
Levels
Sampler
Unseon environment
Unsean agent
Meta- optimizer
Curator
Optimizer
aptimizor
G
R
Figure 2: GROOVE meta-training (left) and meta-testing (right). During meta-training, levels are sampled from both the level curator and sampler. Agents are trained by an optimizer for multiple updates, before a meta-optimizer updates the optimizer based agent return. At the end of agent's lifetime, its regret is calculated and the level curator updated. During meta-testing, the trained optimizer is applied to previously unseen environments and agent architectures.

---

Our contributions are summarized as follows:
In order distinguish this problem setting from traditional meta-RL, provide novel formulation of PMO using the Meta-UPOMDP (Section 2.1).
We propose AR (Section 3.2). a novel regret approximation for PMO, and GROOVE (Section 3.3), a PMO method using AR for environment design.
We analyze how features of the meta-training distribution impact generalization in PMO
Section 4.2) and demonstrate AR as proxy for task informativeness (Section 4.3).
We extensively evaluate GROOVE against LPG, demonstrating improved in-distribution robustness and out-of-distribution generalization (Section 4.4).
We perform an ablation of AR, demonstrating the insufficiency of existing methods (PLR and LPG) without AR, as well as the impact of the antagonist agent in AR (Section 4.3).
2 Problem Setting and Preliminaries
2.1 Formulating Policy Meta-Optimization for Environment Design
Policy Meta-Optimization In this work, consider subproblem of meta-RL which refer to as policy meta mntta-optinization (PMO). PMO is a bilevel optimization problem. In the inner loop, collection of agents each interact with their associated environments and are updated with a policy optimizer. Following series of inner-loop updates, the outer loop updates the policy optimizer in order to maximize the performance of these agents. PMO only trains the policy optimizer the outer loop, while the initial agent parameters for each task are generated by static initialization function.
Unsupervised Environment Design In UED DDnnis et al. 2020). Teacher is given the problem of designing an environment distribution which is maximally ly useful for training a student agent. UED formalizes this problem setting with the Underspecified Partially Observable Markov Decision Process (UPOMDP), an extension of the POMDP with additional free parameters 6 that parameterize those aspects of the environment which the teacher can modify throughout training. In prior work, this paradigm has been used to adapt environment distributions to facilitate the learning of robustly transferable policy [Jiang al. 2021b Parker-Holder Holder al. 2022a]. However, our problem setting of PMO, the central focus is on learning the update rule itself, with the goal of transferring the update rule new environments.
Problem Formulation We therefore introduce the Meta-UPOMDP. an extension of the UP- OMDP, to account for this difference. Formally, the Meta-UPOMDP defined by the tuple (A, SA, T,I, R,y, n, 2, O). The first components correspond to the standard UPOMDP, where is the action space, is the state space, O is the observation space, and T : S A x + S is the transition function. Upon each transition, the student agent receives an observation according to the observation function I S + O and a reward according to the reward function R S' A H R. Here, the free parameters 6 E & control the variable aspects of the environment, such as the 2, y y-positions of obstacles in 2D maze
The Meta-UPOMDP models PMO, in which an optimizer F. ① x T + θ learns to update an agent's s parameters θ given the sequence of states, actions, rewards and termination flags corresponding the agent's 's past experience T in the environment. This update is performed after every transition over the agent's 's lifetime of N environment interactions, resulting in sequence of parameters (On, ... N.). The Meta-UPOMDP extends the UPOMDP include agent parameters θ and appends the agent lifetime N to the free parameters é, making a controllable feature of tasks.
For an initialization of agent parameters fy and free parameters ó, we define the value of the optimizer to be (J) Exe,y DE 7/re), which is the expected return of the trained agent new on the environment specified by ø at the end of its lifetime. Given an optimizer Fr, with meta-parameters n, we reformulate the PMO objective from Oh et al. 2020) to
[(n) = Ex-p(() Eeg-n(e.) (V.0, (F,)).
where p()) and p(Ou) are distributions of free parameters and initial agent parameters.
'The terms "tasks" and "levels" used interchangeably refer settings of free parameters.
3

---

2.2 Learned Policy Gradient
Learned Policy Gradient [Oh et al. 2020, LPG] is a PMO method which trains a generalization of the actor-critic architecture |Barto al., 1983] This replaces the critic with a generalization of value functions from RI., that we refer to as bootstrap functions. Whilst value functions are trained predict the expected discounted return from given state, bootstrap functions predict an n-dimensional,, categorical bootstrap vector, the properties of which are meta-learned by LPG.
LPG uses reverse-LSTM [Hochriter and Schmidhuber, 1997 to learn policy update for each agent transition, conditioned on all future episode transitions. For a single update to agent parameters 0 at time-step t, LPG outputs targets Ut. R. ( It = (r,. di, 7i To(a (ao(st), yo (s()., yo (8(+))))] is a vector containing reward rr.
FT where flag de, discount factor 7 probability of the chosen action ne Te(ar and bootstrap vectors for the current and next states (*x) and yo (81+)). The targets ý and म update the bootstrap function and policy respectively, giving the update rule
A x [V log By (voll())))))
2.3 Learning Robust Policies via Minimax-Regre UED
A trivial example of UED is domain randomization [Jakobi 1997 DR], in which the teacher generates an environment distribution by uniformly sampling free parameters from the UPOMDP. By sampling randomly, DR often fails to generate environments with interesting structure: they may be trivial, impossible to solve, or irrelevant to downstream tasks interest. An alternative approach is to train an adversarial minimax teacher, whose objective is generate environments which minimize the agent's 's return. This has the benefit of adapting the environment distribution to the agent's current capability, by presenting it with environments which it performs poorly. However, by naïvely minimizing return, the teacher is incentivized to generate environments which are impossible for the agent solve.
In contrast to this, Dennis et al. 2020 propose the minimax-regret objective for UED, in which the teacher aims maximize the agent's ' regret, the difference between the achieved and maximum return. Unlike return minimization, this disincentivizes the teacher from generating unsolvable levels where regret would be O. However, since computing the maximum return is generally intractable. methods implementing this objective have proposed approximations of regret. PAIRED [Dennis et al.) 2020] co-trains an antagonist agent with the original (protagonist) agent, estimating regret as the difference in their performance. PLR [Jiang et al. [2021baa curates a buffer of high-regret levels, rather than training generative model produce them. In this, range of regret approximations evaluated, with positive value loss, I.1 value loss, and maximum Monte Carlo achieving consistent performance across the evaluated domains.
3 Adversarial Environment Design for Policy Meta-Optimization
In this section, introduce GROOVE, novel method for PMO. We begin by formulating the minimax-regee objective implemented by GROOVE, followed by AR, a novel regret approximation designed for PMO. Finally, discuss existing approaches to environment design and motivate the selection of a curation-based approach.
3.1 Minimax- as a Meta-Objective
Our method trains optimizer F, over adversarial distribution of environments, in which the UED adversary's 's objective is to maximize the regret of the meta-learner, defined as
REGRET Φ) Vo(n Vn(n).
Here, ni denotes the optimal meta-parameters for updating the student's policy on an environment instance Φ i.e., 7 = arg max, Eg Eeg-n(0a) (0or-((0)\Voee,, (F))). The function Va H. + R defines the expected return of the student policy on á at the end of the lifetime (after N updates) using the update rule parameterized by H, when trained from initialization 0g p(0o).

---

3.2 Approximating Minimax-Regra
As in the traditional UED setting, regret is generally intractable to compute. A range of scoring functions have been proposed approximate regret, often deriving the approximation from value loss (Jiang et al. [0][bba]. Two consistently high performing metrics are LI value loss and positive value loss, which are equal the episodic mean of absolute and positive GAE [Schulman al. 2016) terms respectively.
In order exploit the structure of our problem setting, propose alternative approximation which we refer to as algorithmic regret (AR). In this, we co-train an antagonist agent using a manually designed RL algorithm A (e.g., A2C [Mnih al.) 2016), PPO [Schulman et al. (2077] in parallel to the protagonist agent trained by GROOVE. AR then computed from the difference in final performance against the antagonist,
REGRET Ø) Vo(A) (T)
(4)
where (AA) denotes the expected return of the antagonist policy when trained with AA.
We evaluate AR against both LI value loss and positive value loss Section 4.4, demonstrating improved generalization performance on Min-Atar.
3.3 Environment Design
Following the dual curl-curri design paradigm from Jiang et al. (2021a), a large class of UED methods be represented combination of two teachers: curator and generator. Here, the level generator is a generative model that is optimized to produce regre-mmaxmmizinn levels, whilst the level curator maintains set of previously -iisttdd high-regret levels to be replayed by the agent. The generator provides a slowly adapting mechanism for environment design, allowing the method to design levels without random sampling, whilst the curator provides quickly-adaptinn replay buffer of useful levels.
In PMO, single sample of environment regret requires agent to be trained to convergence and subsequently evaluated on that environment. This is significantly less sample efficient than traditional UED, where regret is measured from a single rollout of the current policy. Furthermore, generator- based methods for UED (e.g. PAIRED) have been shown to achieve lower performance and sample efficiency than curation-baat based approaches [Jiang et al.) 202 Parker- Holder al.) [022aa] Due to this, we design GROOVE using PLR (Section 2.3) which curates randomly generated levels, avoiding the need to train level generator.
The meta-training loop for GROOVE presented in Algorithm and Figure 2
Algorithm 1 GROOVE meta-training
input: Environment set 4 agent parameter initialization function p(0) initialize: Meta-parameters 7), PLR level buffer A, agent-environment lifetimes {0, d) repeat
for all lifetimes () do for update <- 1 to do
Rollout agent Ta environment Update A with F,, (Equation 2}
end for
Compute meta-gradient for v) with updated 0 (Equation if lifetime over then
Evaluate regret approximation R + REGRET" (1,, Φ) with final 0 Update level buffer A with R and Reinitialize lifetime (0,)) (0)) x A
end if end for Update 77 with accumulated meta meta-gradients until 7) converges
5

---

Experiments
Our experiments are designed to determine (1) how the meta-training distribution impacts OOD generalization in PMO, (2) how well AR identifies informative levels for generalization, and (3) the effectiveness of GROOVE at generating curricula for generalization using this metric. To achieve this, we first manually examine how informative and diverse meta-trast aata-training distributions improve the generalization performance of LPG (Section 4.2). We then evaluate curricula generated by AR against those generated by randomly sampling and handcrafting (Section 4.3p, demonstrating the ability to identify informative levels. Following this, we evaluate GROOVE against LPG (Section 4.4). demonstrating the impact of environment design for improving both in in-distribution robustness and generalization performance. Finally, we evaluate GROOVE with AR against baseline metrics from the UED literature (Section 4.5), showing only AR consistently improves generalization.
4.1 Experimental Setup
Training Environment For meta-training, we use a generalization of the tabular Grid- World environment presented by Oh et al. [2020]. In this environment distribution, a task is specified by the maximum episode length, grid size, wall placement, start position. and number of objects, whilst the objects themselves vary in position, reward, and probabilities of respawning or terminating the episode. This space contains tasks encapsulating thematic challenges in RL, including exploration, credit assignment, and stochasticity.
However, these challenges are notably sparse over the environment distribution. For instance, maze- like arrangements of walls induce hard exploration challenge by creating anomalously long shortest- path lengths to objects, but are rare under uniform sampling. This captures the need for environment design, in order discover complex and informative structures required for generalization.
Testing Environments The purpose of our evaluation is to determine the generalization perfor- mance of the algorithms we consider, i.e., the expected return on real-world RL tasks. In order to approximate this, we evaluate on Atari [Bellemare et al. 2013). an archetypal RL benchmark, as well as its simplified counterpart Min-Atar Young and Tian 2019 for our intermediate results.
Model Architecture and Implementation For our learned optimizer, we use the model architecture proposed in LPG. Since GROOVE is agnostic to the underlying optammppiiinttion method, we select LPG due its state-of-the-arr generalization performance analysis of training distribution performed on LPG, which we build upon in [Section 4.2) Further comparison to prior meta-optimization methods is presented in [Section 5| Our experiments were executed on two to five servers, containing eight GPUs each (ranging in performance from 1080-Ti to V100). Model hyperparameters can be found in the supplementary materials and the project repository is available at httpe//bb.oooo//mppyyaass//////e
unseen tasks, in addition to the prior
4.2 Designing Mraannrg Distributions for Generalization
Unlike in standard RL, the impact of meta-training distributions OOD generalization in PMO has not been explored in depth. One attempt at this came from Oh et al. [2020), who evaluate the generalization performance of LPG after meta-training both the number of tasks and the environments that the tasks are sampled from. By demonstrating improved transfer Atari, the authors claim that two factors improve generalization performance:
three different environment sets, varying
1. Task diversity (the number of training tasks), and
2. How informative the tasks are for generalization
Whilst these results suggest relationship between the meta-training distribution and generalization performance, the strength of these claims is limited by the number of environment sets (three) and confounding of these factors.
"Quote: "specific types of training environments... improved generalization performance." We refer to these types of environments as being informative for generalization.

---

To analyze the impact of task diversity, we sam- ple Grid- World subsets of various sizes, before meta-training LPG on each of these and evaluat- ing on Min-Atar (Figure 3). By generating envi- ronment sets with L..dd sampling from the Grid- World environment distribution, we remove the informativeness of tasks confounding fac- tor. We observe significant (p < 0.05) positive correlation in performance with number of lev- els in the aggregate task return, supporting the first claim. Furthermore, when considering the per-task breakdown of results (see supplemen- tary materials). observe a significant positive correlation on three of the four tasks, with the re-
Random
Max-AR
Handcrafted
AZC-Normalized Retum
0.5
0.4
0.3
0.2
0.1
0.0
r= 0.86
maining task having weak positive correlation, thereby supporting the first claim.
2
2
2
26
2
2
Number of Grid-World training levels
We investigate the second factor using set of handcrafted Frid-World configurations pro-
Figure 3: Aggregate performance Min-Atar, with standard error shaded over independent runs runs-PMCC PMCC given Random levels. A per-task breakdown provided the supplementary materials.
posed by Oh et al.] [2020) (see supplementary materials). These are manually designed to em- phasize stochasticity and credit assignment, two key challenges in RL,, making them more informative for generalization than randomly sampled Grid-World configurations. At the same number of levels, observe improvement in perfor- mance from meta-training on handcrafted levels against random levels. Moreover, training on five handerafted levels exceeds the performance of 26 64 random levels, demonstrating the need for task distributions to contain tasks that are both informative and diverse.
4.3 Algorithmic Regret Identifies Informative Levels for Generalization
hypothesize that AR identifies informative levels for meta-training. Before evaluating auto-curricula generated with this metric (Section 4.4), we evaluate static curricula generated by AR against random and handerafted curricula. To achieve this, train an LPG instance to convergence and collect a buffer of 10k unseen levels, ranked by the final AR of the model. We then train new LPG instance on the highest-scoring levels, controlling for task diversity by subsampling variable-sized level sets from the buffer.
In Secciion 3.2
At all sizes of training environment set, we observe improved OOD transfer performance when training on high-AR levels compared to training on the same number random levels (Figure 3). Furthermore, training with high-AR levels outperforms training the same number of handcrafted levels, and far exceeds the performance increased, without requiring any human curation. These results validate AR effective metric for automatically generating informative curricula.
the fixed handcrafted set as the number of levels is
However, we note that the performance gap between high-AR and random levels decreases as the number of levels grows. This is not surprising, since the high-AR levels are generated by ranking and selecting the levels with the highest AR for each training set size, leading to natural dilution in average AR. This also highlights the need for automatic curriculum generation throughout meta- training rather than training static curricula, which we examine further in Section 4.4
IQM
Optimality Gap
LPG
GROOVE
0.100
0.125
0.78
0.84
0.90
HumannNormalized Score
Figure 4: Aggregate performance metrics on Atari-shaded area shows 95% stratified bootstrap confidence
interval (CI) over 5 seeds, following methodology from (Agaaal et al. 202T. Higher score better for IQM lower score better for optimality gap.
7

---

4.4 Adversarial Environment Design Improves Generalization
We now evaluate the impact of environment de- sign on generalization performance by compar- ing GROOVE LPG, thereby evaluating the same meta-optimizer with and without environ- ment design. After meta-training both methods on Grid-World, we first evaluate randomly- sampled, unseen Grid- Worlds Figure 5). In addition to GROOVE achieving higher mean performance, we observe increased robustness with GROOVE consistently outperforming LPG on their lower-scoring half of tasks. Notably, LPG fails to achieve greater than 75% A2C- normalized return on 187% more tasks than GROOVE. Despite this, GROOVE and LPG achieve comparable performance on their higher- scoring half of tasks, suggesting that GROOVE increases in-distribution robustness without re- ducing performance on easy tasks.
LPG GROOVE
1.00
Fraction of runs with score
0.75
0.50
0.25
0.00
0.0
0.4
0.8
A2C-normalized score (T)
Figure 5: A2C-normalized score distribution unseen Grid-World levels, shaded area shows 95% CI over 10 seeds.
To evaluate generalization to challenging environments, we evaluate GROOVE against LPG on the Atari benchmark. GROOVE achieves superior per-task performance to LPG, achieving higher mean score on 39 vs. 17 tasks (Figure 1). with equal performance on one task (Montezuma's 's Revenge). Comparing aggregate performance, we observe significant increases in both IQM and optimality gap from GROOVE against LPG (Figure 4). While both of these methods achieve inferior performance to sate-oofthheartt manually-designe RL algorithms (Hessel et al. 2018). the improvement from GROOVE highlights the importance of the meta-training distribution and potential for UED-based approaches when generalizing complex and unseen environments.
4.5 Algorithmic Regret Outperforms Existing Metries
Finally, we evaluate the quality of our proposed environment design metric, AR, against existing metrics from UED (Figure 6). We compare to LI value loss and positive value loss (Section 3.2) due to their consistent performance in prior UED work, as well as regret against an optimal policy (as is analytically computable Grid-Wor World), which serves we also evaluate uniform scoring, which is equivalent to domain randomization (i.e. standard LPG).
upper bound regret. As baseline,
MinnAtrr
Breakoul-MMnnAtar
Freeway- Freeway-MinAlar MinnAar
Spacelnvaders-Minne
Uniform
Value
Positiva Value
Optimal-Palley Regret Algortthm
9
12
3.2
4.0
4.8
6
9
75
90
105
Return
Figure 6: Evaluation of our proposed environment score function, AR, against uniform scoring, optimal-policy 10 random seeds are shown.
regret and baselines from UED literature. Mean return and standard
AR achieves the highest performance on all tasks, significantly outperforming optimal-policy regret on all tasks and each of the other baselines at least two out of four tasks. Furthermore, the value-loss metrics only significantly outperform uniform scoring on a single task, with positive value loss underperforming it on all others. This failure to identify informative levels for generalization highlights the challenge in transferring existing UED methods to PMO and the effectiveness of AR.
Whilst it is surprising that aptimal-policy regret, which uses privileged level information, underper- forms AR with an A2C antagonist, we hypothesize that the non-optimal performance and generality of handcrafted algorithms is a benefit for identifying informative levels. aptimal-police regret does not account for training time, making equivalent to an antagonist optimizer which always returns the optimal policy parameters, a setting likely to identify artificially difficult levels. To investigate this, we perform an comparison A2C, PPO, random and expert antagonist agents (see supplementary materials), finding that using A2C or PPO antagonists outperforms random or expert agents.
8

---

5 Related Work
In this work, we examine the impact of training distribution for meta-learning general RL algorithms. More broadly, work aligns with the "AI-generating algorithm" [Clune] 2019 AI-GA] paradigm, building upon two of the three pillars (learning algorithms and environments). In this section, we outline prior work in each of these fields and explain their relation this work.
Policy Optimnatinnnatton A common approach to PMO is to optimize RL algorithm components via meta-gradien meta-gradients (Xu et al. 2018 2020]. In particular, ML [Bechtle et al., 2021] uses meta- gradients to optimize domain-specific loss functions that transfer between similar continuous control tasks. MetaGenRL (Kirsch et al. 2020 expands upon this to optimize general loss functions that transfer to unseen tasks. LPG Oh et al., 2020] discovers a general update rule that transfers from simple toy tasks to Atari environments. We choose to focus on LPG since it displays radical out-of-distribution transfer and imposes minimal structural bias on the learned update rule.
An alternative approach uses Evolution Strategies [Rechenberg. 1978, Salimans et al. 2017 to optimize RL objectives. EPG (Houthooft al., 2018] evolves an objective function parameterized by a neural network that transfers to similar MuJoco environments, whilst DPO [Lu et al.) [022] evolves an objective that transfers from continuous control tasks MinAtar environments. Other approaches to PMO symbolically evolve RL optimization components such as the loss function [Co-Reyes et al.
2021
Garau-Luis et al. 2022 or curiosity algorithms (Alet al. 2020 PMO is instance of
Auto-RL RI. Parker-Holder et al. 2022b), which automates the discovery of learning algorithms.
Alternative Approaches to Meia-Reinforeement Learning PMO belongs to the subclass of many- shot meta-RL. algorithms, which pose the setting of learning-to-leam given substantial number
of inner-loop environment interactions. An alternative class of approaches to this learn "intrinsic rewards", which augment the RL objective in order improve learning. Alet et al. [2020) achieve this by meta-lcarning a program to transform the agent's 's objective, whilst Veeriah et al. [022 propose a hierarchical method which meta-learns transferable options.
However, the majority of work in meta-Re RL has instead been on few-shot learning [Beck et al. 2023b RL3 (Duan et al. 2016 Wang et al. 2016) use a black-box model for this, by representing both the policy and update rule with a recurrent neural network. A range of extensions to RL have been proposed, which augment the original model with auxiliary task-inference objectives (Humplik et al.) 2019, Zinngraf et al.] 2020]. additional exploration policies (Liu et al.] 202T] and hypernetwork- based updates [Beck et 2023a Parameterized policy gradients
alternative approach, with Model-Agnostic Meta-Learning [Finn et al.) 2017] MAML] being the seminal method. MAML meta-learns shared neural network initialization, such that it rapidly adapts to new tasks when optimized with policy gradients in the inner loop. Follow up work to this has proposed partitioning parameters (Zintgraf et al. 2019), modulating parameters for multimodal distributions (Vuorio et al. 2019] and investigated the reasons for its effectiveness [Raghu et al. 2019].
Unsupervised Environment Design Unsupervised Enviromment Design (UED) was first proposed by Dennis al. [0200 with the introduction of PAIRED, which trains level generator for a single agent with minimax regret. GROOVE is based on PLR [Tiang et al.) [021b al. which builds upon this objective by instead curating a buffer of high-regret levels. PLR remains one of the state of the art UED al algorithms, which has been extended to consider multi-agent settings (Samvelyan et al., 2023). curriculum-induced covariate shift [Jiang et al., [022] and more open-ended environment generators [Parker-Holder et al., 2022a]. Aside from regret, environments can also be selected to induce diversity in a population of agents [Brant and Stanley 2017). most famously in the POET algorithm [Wang et al. 2019 which evolves a population of highly capable specialist agents.
UED falls more broadly into the field of open-endedness [Soros and Stanley 2014), which attempts to design algorithms that continually produce novel and interesting behaviours. We take step towards more open-ended algorithms by combining UED with meta-learning, thus discovering both algorithms and environments in a single method. Previous works combining these two AI-GA pillars include Team et al. (2023 who introduce a memory-based agent capable of human-timescale adaptation, and OpenAl et al. 2019 who train a policy capable of sim-to-real transfer to control Rubik's Cube. Unlike our work, neither of these meta-learn general RL. algorithms that can transfer to far out- distrribbtt environments such as Atari.

---

Conclusion and Limitations
In this paper, we provide the first rigorous examination of the relationship between meta-training distribution and generalization performance in policy meta- vetaaoppimization (PMO). Based on this analysis we leverage ideas from Unsupervised Environment Design (UED) and propose GROOVE, novel method for PMO, in addition to novel environment design metric, algorithmic regre! (AR). Evaluating against LPG, we demonstrate significant gains in generalization from Grid- Worlds to Grid-Worlds
Atari, as well as increased robustness to challenging in-distribution tasks. Finally, we identify AR as a critical component for applying environment design to PMO, demonstrating its effectiveness on Min-Atar, where prior UED metrics fail to outperform random sampling.
We acknowledge limitations in our work, largely driven by computational constraints. Given the huge number of environment interactions during meta-training, we heavily leverage our GPU-based Grid-World implementation in lowering training time. This limits our analysis to these fast but simple environments, meaning our conclusions cannot be guaranteed to generalize to more complex environments. Due to the cost of meta-testing on the large-scale Atari benchmark, our evaluation also limited in variety of benchmarks, however we believe the diversity of the tasks in this domain sufficiently captures a range of challenges RL.
By releasing our implementation which is capable of meta-training these models on single GPU in hours, rather than days-we hope to spawn future work this area from academic labs. In particular, these results may be scaled to training distributions with more complex and diverse environments, leading to the discovery of increasingly general RL, algorithms.
Acknowledgments and Disclosure of Funding
The authors thank Junhyuk Oh, Tim Rocktäschel and the anonymous NeurIPS reviewers for their helpful feedback that improved our paper. Matthew Jackson funded by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems, and Amazon Web Services.
References
R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 14:29304-29320, 2021. 4
Alet, M. Schneider, T. Lozano-Pérez, and L. P. Kaelbling. Meta-learning curiosity algorithms. CoRR, abs/2003.05325, 2020. BRLpttt::///arx/ org/ abs 2003. 05325
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021.
O. M. Andrychowicz, B. Baker, M. Chociej. R. Józefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics
Research, 39():320 2020. []
A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5): 834-846, 1983. doi: 10.1109/TMMC.1933.6311000
2.2
Bechtle, A. Molchanov, Y. Chebotar, F. Grefenstette, Righetti, G. Sukhatme, and F. Meier. Meta learning via learned loss. In 2020 25th International Conference Pattern Recognition (ICPR), pages 4 4161-4168. IEEE, 2021.
J. Beck, M. T. Jackson, R. Vuorio, and S. Whiteson. Hypernetworks meta-reiinforcement learning. In Conference on Robot Learning, pages 1478-1487. PMLR, 2023a. 5
Beck, R. Vuorio, E. Liu, 7. Xiong, L. Zintgraf, C. Finn, and S. Whiteson. survey of meta-reinforeement learning. arXiv preprint arXhv:2301.08028 2023b. 05
10

---

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279,
2013. 41
C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, Tang, F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. CoRR.
abs/1912.066800, 2019. T
J. Bradbury, R. Frostig. P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman- -Milne,, and Q. Zhang. JAX: composable transformations of Python+NumPy programs. 2018.
thub. com com/google/ jax
J. C. Brant and K. O. Stanley. Minimal criterion coevolution: A new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '17, page 67-74, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 9781450349208. doi: 10.1145/3071178.00 186.. 5
Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. CoRR, abs/ 19985 2019. 5
J. D. Co-Reyes, Y. Miao, D. Peng, E. Real, S. Levine, Q. V. Le, H. Lee, and A. Faust. Evolving reinforcement learning algorithms. arXiv preprint urXiv:2101.03956 2021. 5
J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdol-
maleki, D. Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay,
A. Merle, J.-M. Moret, and M. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602:414-411 02 2022. [
M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems,
2020. 2.1 2.3
Y. Duan, Schulman, X. Chen, L. Bartlett, I. Sutskever, and P. Abbecl. 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277 2016.
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on muchine learning, pages 1126-1135 PMLR, 2017. 5
J. Gurau-Luis, Y. Miao, D. Co-Reyes, Parisi, J. Tan, E. Real, and A. Faust. Multi-ohjective evolution for generalizable policy gradient algorithms. arXiv preprint arXiv:2204.04292 2022.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. AAAI, 2018. !
M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 4.4
Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98)):1335-1700. 1997. 2.2
R. Houthooft, Y. Chen, P. Isola, B. Stadie, F. Wolski, O. Jonathan Ho, and P. Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018.
J. Humplik, A. Galashov, L. Hasenclever, P. A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning as task inference. arXiv preprint 1rX1::1905.06424 2019. 5
N. Jakobi. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive behavior, (2):325-368, 1997. 2.3
M. Jiang, M. Dennis, J. Parker-Holded J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay-
guided adversarial environment design. Advances in Neural Information Processing Systems, 34: 1884-1897, 202 2.3
3.2
3.3 5

---

M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In International Conference on Machine Learning. pages 19940-4950 PMLR, 2021b.
M. Jiang, M. Dennis, Parker-Holder, A. Lupu, H. Küttler, E. Grefenstette, Rocktäschel, and
J. Foerster. Grounding aleatoric uncertainty for unsupervised environment design. In S. Koyejo,
S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32868-328888 32881. Curran Associates, Inc., 2022. S
L. Kirsch, S. van Steenkiste, and J. Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Representations, 2020.
E. Z. Liu, A. Raghunathan, P. Liang, and C. Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International conference on machine learning. pages 6925-6935. PMLR, 2021.
C. Lu, J. Kuba, A. Letcher, L. Metz, C. Schroeder de Witt, and J. Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455-16468, 2022. 5
T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62)xeabk2822, 2022. [
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016. 3.2
Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. P. van Hasselt, S. Singh, and D. Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33: 1060-1070, 2020.
C
OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's 's cube with a robot hand. CoRR, abs/1910.07113, 2019.
5
Parker-Holder, M. Jiang, M. D. Dennis, M. Samvelyan, N. Foerster, Grefenstette, and T. Rock- täschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01333 2022a. 2.1
Parker-Holder, R. Rajan, Song, Biedenkapp. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, et al. Automated reinforcement learning (autorl): A survey and open
problems. Journal of Artificial Intelligence Research, 74:517-568, 2022b. 5
A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. arXiv preprint arXiv: (oXi::990000157 2019. [
I. Rechenberg. Evolutionssttrategien.. In B. Schneider and U. Ranft, editors, Simulationsmethodden in der Medizin und Biologie, pages 83-114 Berlin, Heidelberg, 1978. Springer Berlin Heidelberg. ISBN 00----642-81288-- 5
T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies scalable alternative to reinforcement learning. CoRR, 2017. 5
M. Samvelyan, A. Khan, M. D. Dennis, M. Jiang, J. Parker-Holder J. N. Foerster, R. Raileanu, and T. Rocktäschel. MAESTRO: Open-ended environment design for multi-agent reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023, S
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Y. Bengio and LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. httpttt: http:///xxvvvvv.. org// abs/ 1506. 02438 3.2
Schulman, F. Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 3.2
12

---

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanetot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484 489, 2016. [
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Ku- maran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. Science, 2017. [
L. Soros and K. Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In ALIFE 14: The Fourleenth International Conference on the Synthesis and Simulation of Living Systems, pages 793-800. MIT Press, 2014.
R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA,
USA, 1st edition, 1998. ISBN 0262193981.
A. A. Team, Bauer, Baumli, Baveja, F. Behbahani, A. Bhoopchand, Bradley-Schmieg.
M. Chang, N. Clay, A. Collister, al. Human- Gemaantiimescale adaptation in open-ended task space. arXiv preprint XXX::2301.07608, 2023. 5
V. Veeriah, T. Zahavy, M. Hessel, Z. Xu, J. Oh, I. Kemaev, H. P. van Hasselt, D. Silver, and S. Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861-2998
2021. 5
R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in neural information processing systems, 32, 2019. 5
X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv preprint 16Xiv:6611.00 1.11,6763, 2016. 5
R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint
2019. 5
Z. Xu, H. P. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems, 31, 2018. 55]
Z. Xu, H. P. van Hasselt, M. Hessel, J. Oh, S. Singh, and D. Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 33:15254-15264, 2020. 5
K. Young and T. Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: /903.003333111 03676,11110 2019.
L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta- learning. In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: a very good method for bayes-adaptive deep rl via meta-learning. Proceedings of ICLR 2020, 2020. 5
13

---

A
Summary of Notation
Table 1: Summary of notation used in this work.
Symbol
Definition
Meta-UPOMDP
O,
Action, state and observation spaces. Transition, observation and reward functions. Discount factor.
T.I.R
Φ E +
Free parameters of the environment. Agent parameters, shared between actor and critic/bootstrap function.
0 E
θ
T
Sequence transitions (O A x R x O), denoted task "experience".
Policy Meta-Optimization
0x
Agent optimizer.
7 7EH
Agent optimizer parameters.
Expected return of Fr the end of training, given task e and agent initialization 0g.
(F.
(Shorthand) Expected return of Fr the end of training task Φi over a distribution of agent initializations p(0)).
Vo(T)
Learned Policy Gradient [Oh et al. 2020
To : A, []
Agent policy.
Agent bootstrap function- generalization of value critics from RL, outputting a vector with semantics determined by the learned optimizer. LPG target function, outputting bootstrap function and policy targets ù and at time step I, conditioned on future transitions.
yo O + [0, 1]"
Un T 00, 1|"d x
14

---

B Hyperparameters
B.1 GROOVE
Hyperparameters shared between GROOVE and LPG were tuned using LPG on Grid-World, then transferred to GROOVE without further tuning. The additional GROOVE hyperparameters (regarding the level buffer) were then tuned separately on Grid-World.
Table 2: GROOVE/LPG hyperparameters
Hyperparameter
Value
Optimizer
Adam
Learning rate
0.0001
Discount factor
0.99
Policy entropy coefficient (3))
0.05
Bootstrap entropy coefficient (31)
0.001
L.2 regularization coefficient for (32)
0.005
L2 regularization coefficient for (3g)
0.001
Level buffer size
4000
Replay probability
0.5
Number of interactions per agent update
20
Number agent updates per optimizer update
Number of parallel lifetimes
512
Number of parallel environments per lifetime
64
Algorithmic regret baseline algorithm
A2C
B.2 Agents
Agent hyperparameters based tuned A2C agents, before being fine-tuned with LPG. Since we meta-train on continuous distribution of Grid-World environments, we do not use the agent hyperparameter bandit proposed by Oh et al. 2020 for meta-training.
Table 3:
Agent hyperparameters- architecture descriptions D(N) and C(NV respectively refer to dense and convolutional layers of size N; ReLU activations are used throughout.
Environment
Hyperparameter
Grid-World
Min-Atar
Atari
Architecture
Tabular
D(64)-D(64)
C(32)-C(61)-C(44)-D6622)
Optimizer
SGD
Adam
Adam
Learning rate
40
0,0005
0,0005
Bootstrap KL coefficient (ay)
0.5
0.5
0.5
Train steps
2500
100,000
100,000
Agent seeds per I.PG seed
16
15

---

C Handcrafted Environments
For our handcrafted environment set, we use the five tabular Grid-World configurations from Oh et al. 2020]. Grid-World objects are defined by (r, Geem: Crespawnl, where represents the reward when collected, Gem is the spisode-terminane probability and Gresaww the probability of the object respawning each step after collection.
C.1 Dense
Property
Value
Size 11 x 11
Objects
[1,0, 0.05), [-1, [-,, 0.5, [-1,0,
Maximum episode length
500
C.2 Sparse
Property
Value
Size 13 13
Objects [1, 1, [],,
Maximum episode length
50
C,3
Long Horizon
Property
Value
Size 11 x 11
Objects x [1,0, 0.011, 2 [-1,0.5, 1]
Maximum episode length
1000
C.4 Longer Horizon
Property
Value
Size 9 x 9
Objects x 0., 0.01), [-1,0.8,11 1]
Maximum episode length 2000
Note: size increased from 7 x 9 for consistency with our generalized Grid- World distribution.
C.5 Long Dense
Property
Value
Size 11 11
Objects
x [1, 0.005]
Maximum episode length
2000
16

---

D Atari Training Curves
1.60
0.75
0.57
0.23
2006
176k
100k
50C Essounte
10ak
DOK
220k
Seam
1.3
50
25
50C
S3K
320k
SCK
Compoede
Clayy Climbar
Durmer Attacc
1.0
0.
txt Bestyy
0.5
200k
12ck
100k
100k
D3K
120k
SEK
LOOK
Craviuur
Jamsubond
-1.0
-2.0
50c
100k
53X
S4k
dling Marter
Masmouuma Revinng
Rocmon
Plaenix
petul
1.0
-2.4
2.0
1n4
1.0g
-3.2
ad
10
SPC Solara
Dixx
10:K
555
3TR Sumound
Soccueact
Scuce
-0.905
-0.355
SDC Banukktton
SIC Nitwar
SOE Venturr
100K
2.0
1.0
0.3
10
txz
0.15
0.50
0.22
GROOOV
SCK
200K
S3K
130K
50c
100K
Figure 7: Atari training curves environment names are highlighted according to highest evaluation return, asterisks (*) denote significant differences in evaluation return (5 seeds, p 0.05).
17

---

E Min-Atar Per-Task Performance
As expected, observe increased noise when breaking down performance by individual Min-Atar lasks, however, the results from the majority of tasks support our carlier conclusions. Firstly, we observe significant positive correlation between the number of random training levels and return on three of the four Min-Atar tasks, again demonstrating the impact of task diversity on generalization. When controlling for the number of levels, we observe improved performance after training on handcrafted, rather than random, levels on three of the four Min- MiarAAtar tasks. Furthermore, on Asterix, training on handcrafted levels results in higher performance than the largest set of 2 1024 random levels, supporting our conclusion about level informativeness.
After training on high-AR levels, observe improvement against random levels least three of the four Min-Atar tasks for all sizes of training environment set up to 2 = 64 levels. Beyond this, random and high-AR levels outperform each other on equal number tasks, however the dilution in mean AR for larger training sets makes this convergence unsurprising. Furthermore, high-AR levels are competitive with handcrafted levels at the same training set size and quickly outperform the fixed handcrafted set more high-AR levels are added, demonstrating the effectiveness of AR at identifying informative curricula.
Handcrafted
Asterix-MinAtar
Breakout-Minntar
Fresway-Minntar
Spacelnvaders-MinAta
15
20
100
7.5
Retum
10
5.0
10
50
2.5
0
0.0
0.64
F'm
2
24
2
2
24
2
2
24
2
2
2
2
Number of Grid-World training levels
Figure 8: Generalization performance on Min-Atar, after meta-training LPG on variable-sized sets of Grid-World levels -eeel) are selected through uniform-random sampling of all levels ("Random")), from the highest-regret levels of a previous LPG instance ("Max-AR"), or from a set of five handcrafted levels ("Handcrafted"). Pearson correlation coefficient is given for Random levels; significant positive correlations are marked with an asterisk (*).
F GROOVE vs. LPG Procgen Evaluation
After meta-training on Grid-World, we observe superior GROOVE performance on 2 out of Procgen environments, superior LPG performance on environment, and no difference on the remaining environment. We note that A2C is very weak on Procgen, failing to learn on the majority of environments, so we selected a subset of Procgen levels that A2C managed learn in preliminary experiments. Procgen poses a robustness challenge that has required an extensive amount of further research to solve, using components not found in LPG GROOVE.
Chaser
Dodgeball
Fruilbat
Plunder
1.5
1.00
7.5
Return
0.75
5.0
1.0
0.50
2.5
0.5
0.25
0.0
-2.5
25K
50K
25K
50K
25K
50K
0
25K
50K
Train step
Figure 9: GROOVE and LPG training curves on Procgen (test performance, 5 seeds).
18

---

G Algorithmic Regret Antagonist Comparison
In order investigate the impact of the antagonist agent on the performance of AR, we evaluated the performance of GROOVE with a range of antagonists (Figure 10p. On Min-Atar, using random or optimal agent as the antagonist for AR results in lower performance than using A2C or PPO on all environments. Furthermore, using A2C achieves higher performance than PPO on all environments.
Asterix-MinAtar
Kreakout-MinAlen
Freeway-MinAter
Spacelnvaders-
AR (PPO) AR (A2C) (Expert)
4.0
4.5
3
80
100
Return
Figure 10: GROOVE Min-Atar performance after Grid- World meta- meta-training, using random, expert, A2C and PPO agents as the algorithmic regret antagonist mean return over 10 random seeds is marked, with standard error shaded.
To further investigate this result, we evaluated PPO and A2C on both random and difficult, handcrafted Grid-Words levels. PPO achieves lower performance than A2C on Grid-World with a larger gap difficult, handcrafted Grid- World levels. This explains the previous results, as PPO will inferior at identifying difficult levels when used as the AR antagonist. Furthermore, the update parameterized by LPG capable of representing A2C, but not PPO. This implics that levels solvable by A2C should also be solvable by LPG, making them useful for training. In contrast, PPO may identify levels that cannot be solved without components found in PPO (clipping, mini-batch iterations) but not LPG.
Random
Handorafted
18
16
14
Return
2.1
12
2.0
10
1.9
A2C
PPO
1.8
Train step
Train step
Figure A2C and PPO training curves on random and handcrafted Worid-Wordd levels we observe larger performance gap on harder, handcrafted levels (10 seeds).
19


Reviews:
[{"review_id": "CpaGE58THU", "reviewer": "Reviewer_8DkV", "summary": "This paper proposed a new framework, GROOVE, to solve a new Meta-UPOMDP problem. It also introduces algorithmic regret (AR) to approximate the regret to update the curator and generator (sampler?).  The results show that the meta-optimizer learned by GROOVE can be used to improve the games in Atari.", "questions": "1. How do you select algorithm A in equation 4? Including the hyperparameter of the algorithm. \n2. The claim in 209 seems similar to [1]. Please explain the difference. \n3. What is the probability of using the generator and the curator?\n\n\n\n[1] Li, Y., & Zhang, C. (2022). On Efficient Online Imitation Learning via Classification. Advances in Neural Information Processing Systems, 35, 32383-32397.", "limitations": "Mentioned in weakness.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed problem Meta-UPOMDP is important. \n2. The experiments show that the method can work even from grid-world to atari. \n3. The AR idea is novel. \n4. The results of AR in Fig. 6 is great.", "weaknesses": "1. Several definitions are absent, including the sampler in Fig2 and the symbol \\eta mentioned in line 74.\n2. It would be beneficial to introduce PMO earlier. The headline and abstract might lead readers to expect traditional RL and few-shot meta-learning.\n3. It is recommended to conduct tests on additional environments, such as Procgen. Even just for evaluating the meta-optimizer learned in Grid-World.\n4. Since AR constitutes a crucial aspect of the work, further insights should be provided.\n5. The majority of the work involves combining existing works (LPG and PLR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your kind words regarding the problem’s importance, the novelty of algorithmic regret (AR), and the demonstrations of its performance. We encourage the reviewer to **read the edits and new results outlined in the author rebuttal**. We respond to each of your comments below:\n\nWeaknesses:\n1. Thank you for pointing these out, we have now added the definition of \\eta (**see line 74 edit**). The use of “sampler” rather than “generator” (see Section 3.3) reflects that this component generating random levels in PLR, rather than learning a generative model. We will highlight this distinction in the caption.\n2. We agree that the distinction between PMO and traditional meta-RL is important. No formal distinction between the two settings has been proposed in prior work, which is why we introduce PMO as a novel problem formulation in this work. Since readers will be unfamilar with this terminology, we use existing terminology - such as “discovering RL algorithms” in the title and “meta-learn update rules” in the abstract - to describe the setting before mentioning it in the introduction and formalizing it in Section 2.1.\n3. This is a good suggestion and we have now added your suggested experiment of GROOVE vs. LPG on Procgen (**see new results**). We chose to prioritize Atari for our original evaluation to be consistent with the environments used in the original LPG paper, but we hope this provides the additional insights you hoped for.\n4. We have now added a further experiment for AR, evaluating the impact of different antagonist agents on generalization performance (**see new results**). In addition to this, we believe Figure 3 provides strong insight into AR as a proxy objective for the informativeness of levels, whilst Figure 6 directly compares it to existing metrics, demonstrating its effectiveness.\n5. Whilst it is true that GROOVE builds on LPG and PLR, the primary message of our paper is that these existing methods fail to work together without our novel component, AR. This is demonstrated by our ablation study in Section 4.5, which shows the failure of existing metrics and success of AR. To make this clearer, we have added a list of contributions, including this point, to the introduction (**see manuscript edits**).\n\nQuestions:\n1. For generality, we define AR to be agnostic to the underlying RL algorithm. In our experiments, we selected A2C due to it being used by LPG at meta-train time, using existing hyperparameters for A2C from prior work. We detail these in Table 2 of the supplementary material, but will add a clarification of the source of our hyperparameters there.\n2. Thank you for referring us to this work, the similarity is very subtle. The foremost difference is problem setting: they study algorithms for online imitation learning, rather than policy meta-optimization algorithms. In addition to this, they analyze sample complexity in the number of interaction rounds, expert annotations and oracle calls, whilst we fix the number of environment interactions (samples) and control the number of training environments.\n3. This can be found in Table 1 of the supplementary material. The probability of using the curator (known in UED literature as “replay probability”) is a tunable hyperparameter of the algorithm. We use 0.5, which we selected from preliminary tuning and prior PLR implementations.\n\nIn summary, we hope that the referenced edits have resolved Weaknesses 1 and 5, our new results resolve Weaknesses 3 and 4, and that we have provided sufficient clarification for Weakness 2 and your questions. We thank you again for your praise of the importance, novelty, and evaluation of this work. In light of our response, would you consider increasing your score to an accept? Thank you!"}, {"review_id": "IgJqe44ocd", "reviewer": "Reviewer_qWd8", "summary": "The authors present a meta-learning method to enhance an RL algorithm's performance across diverse RL tasks. They utilize the concept of unsupervised environment design (UED) methods for training RL agents and adapt this approach for the task of meta-learning a policy optimizer. The authors propose a novel algorithmic regret (AR) to facilitate UED and confirm its superior performance to L1 value loss and positive value loss through experiments. The experiments verify the effectiveness of proposed method.", "questions": "Please refer to the 'Weaknesses' section above for questions and clarifications.\n", "limitations": "No.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and easy to understand.\n2. A thorough review of related studies is provided.\n3. The proposed idea, while simple, appears logically sound. However, as I am not intimately familiar with the domain of reinforcement learning, I am unable to evaluate its novelty.", "weaknesses": "1. Could the authors further elaborate why the proposed Algorithmic regret is better than the approximation method before? Some intuitive discussions are necessary in addition to only experimental results. Moreover, AR is compared to L1 value loss and positive value loss in Sec. 4.5. It seems there are also some other loss  in Lines 16-25. Could the authors clarify the reason that the comparison between AR and these loss is not conducted?\n\n2. It would be informative if the authors could discuss the scenarios depicted in Fig. 1 where Groove performs less effectively than LPG.\n\n3. \"The result is our method\" in abstract could be rephrased.\n\n4. It is mentioned that \"By examining how characteristics of the meta-training distribution impact the generalization of these algorithms\". It is unclear to me where it is reflected in the paper. Could the authors further illustrate it?\n\nMinor:\nLine 50 Section 4.4 -> Section 4.5", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review of our method, evaluation, and clarity. In response to your comments, we have provided **new results in the author rebuttal**. Please find our response to each of your comments below:\n1. We have now added a comparison of A2C, PPO, expert, and random antagonist agents in AR, in order to further explain the role of the antagonist on AR’s performance (**see new results**). Additionally, we hope the simplicity of the method and the interpretation of AR as a proxy for informative levels (see Section 4.3) allows the reader to form a strong intuition for how it achieves its performance. Beyond this, we do not provide speculative interpretations for why AR succeeds in the paper, but we hypothesize that levels identified by AR will reflect the biases of the antagonist agent. Therefore, using generalizable algorithms (A2C, PPO) rather than specialized (expert) or weak (random) algorithms as the antagonist should identify levels with common properties.\nRegarding the omitted regret metric (maximum Monte Carlo) mentioned on line 124, we do not evaluate against this baseline since it is only a minor adaptation of positive value loss, designed to handle sparse-reward environments [1]. Since we meta-train on Grid-Worlds, where objects frequently respawn and can be reached in a small number of steps, we do not require this adaptation and therefore use the original form of positive value loss.\n2. Understanding the properties of optimizers learned by GROOVE and LPG is an interesting and important line of work. Since the application of environment design to PMO is an entirely novel topic, the focus of this work is to extensively demonstrate its potential to improve generalization performance. To achieve this, we compare both in-distribution robustness (Figure 5) and out-of-distribution performance on Min-Atar (Figure 6), Atari (Figures 1 and 4) and now Procgen (**see new results**), providing a comprehensive demonstration of GROOVE’s effectiveness. Since these benchmarks are not designed to test interpretable capabilities of agents, we were unable to find any common theme between the settings on which GROOVE outperformed LPG. However, we plan on performing this analysis in future work.\n3. Thank you for pointing this out, we see how it could be unclear. We will rephrase this sentence.\n4. We perform this analysis in Section 4.2, “Designing Meta-Training Distributions for Generalization”. In this, we evaluate the OOD performance of LPG whilst manually controlling the diversity and informativeness of meta-training tasks, thereby relating characteristics of the meta-training distribution to the algorithm’s generalization performance.\n\nWe hope that we have clarified each of your questions and will be sure to edit the sentence discussed in Q3. Please let us know if any of these points remain unclear, or if you have further questions. If not, in light of these responses, our new results, and your praise of our method, evaluation, and clarity, would you consider upgrading your score above a borderline? Thank you!\n\n[1] M. Jiang, M. Dennis, J. Parker-Holder, J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay guided adversarial environment design. Advances in Neural Information Processing Systems, 34: 1884–1897, 2021"}, {"review_id": "W4saKTPDB5", "reviewer": "Reviewer_TbjH", "summary": "The paper presents an approach, i.e. GROOVE, to learning an update rule for generalization on unseen tasks automatically,  based on the idea of Unsupervised Environment Design, where a student agent is trained on an adaptive distribution of environments proposed by a teacher and the teacher seeks to propose tasks which maximize the student’s regret. The authors also introduce a new concept of algorithmic regret, which is used to approximate regret and automatically generate curricula. The results of experiments comparing GROOVE and LPG demonstrate the superiority of GROOVE in terms of generalization.", "questions": "1. Have the authors tried other RL algorithms as the baseline in the Algorithmic Regret? if so, how do they perform?", "limitations": "as stated in the weakness", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well-motivated and the authors provide a clear explanation of the underlying concepts.\n2. GROOVE, especially the use of algorithmic regret, is simple to understand and effective that is well-supported by the experimental results and analysis.", "weaknesses": "1. The paper could benefit from listing its contributions, e.g. in the first chapter, which helps readers capture its novelty faster.\n2. More detailed discussion of the connection between GROOVE and related works in the field of RL and meta-learning (i.e. in the part \"Alternative Approaches to Meta-Reinforcement Learning\")  will be helpful for readers to address this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and overall positive review of our work. We have provided **edits and new results in our summary response**, which we encourage you to read. Please find our response to the weaknesses below:\n1. We thank you for this suggestion and agree that listing contributions would be an effective format, so have **added a contribution list in our edits**.\n2. Since GROOVE is based on ideas from PMO and UED, we discuss its connection to each of these fields in the “Policy Meta-Optimization” and “Unsupervised Environment Design” sections of the Related Work. Here, we highlight GROOVE’s improved robustness and generalization performance on OOD tasks. Regarding the wider field of meta-RL, we contrast this with the PMO problem setting as a whole, rather than just GROOVE, in the “Alternative Approaches to Meta-Reinforcement Learning” section. This avoids us directly comparing GROOVE to methods outside of PMO, which aim to solve different problems.\n\nIn response your question, suggesting that we evaluate alternative RL algorithms for Algorithmic Regret (AR), we have now performed and included this experiment (**see new results**). We thank you for this suggestion, as we believe it demonstrates interesting properties of AR: that providing the optimal policy or a random agent is insufficient and can select artificially difficult or easy levels, whilst the choice of RL algorithm and its capability on the meta-training distribution also influences generalization.\n\nWe hope that the addition of your suggested experiment has provided the insights you hoped for, and that our edit and clarifications have sufficiently addressed both of your concerns. If so, would you consider increasing your rating and/or contribution scores? Thank you!"}, {"review_id": "9HHN1UQKD9", "reviewer": "Reviewer_4F7T", "summary": "The paper \"Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design\" proposes an automatic curiculum approach for meta-learning of RL optimizers. It is based on a notion of regret of the optimizer, to choose environment parameters that are at a good level for the optimizer to leverage feedback from RL training steps. Since the notion of regret is intractable to be exactly computed, authors introduce a notion of algorithmic regret, which is an approximation. Results show intersting properties of the method. ", "questions": "see weakness above", "limitations": "..", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Meta-learning of RL optimizers is an important problem\n- The proposal looks to contain innovative components\n- Interesting results", "weaknesses": "- The paper is very hard, not to say impossible, to follow, as many many different components are not sufficiently defined (when not defined at all). To be useful, a research paper must contain every formalization and definition to allow an educated researcher to understand and reproduce the contribution. While I am not fully familiar with meta-learning of optimizers, I have a strong background in RL and I am still unable to put everything together in that paper to understand the proposal. I really think the paper should be fully rewritten before considering publication anywhere.  Here is a (non-exhaustive) list of things that are really difficult to understand: \n\n            - U_\\eta is introduced in 2.1 with y_t,\\pi_t=U_\\eta(x_t|x_t+1...x_T). I understand it is an LSTM on sequence x_t+1...x_T but why giving x_t| here ? Is it a distribution of x_t given the sequence ? This does not fit with the fact to feeding this to variables y and \\pi. Is it the result of a LSTM step after feeding x_t in the sequence ? but x_t contains y_\\theta(s_t) so it does not make sense to me. \n\n             - In 2.1 we do not know how is trained U_\\eta (\\eta discussed only in 3.1) so it is very diffcult to understand what it captures.  \n\n            - what is y_\\theta and why does it subscripts \\theta as the policy \\pi does ? policy and boostrap  share same parameters ? what is it supposed to capture ? \n\n            - in 2.2 V gets a meta-optimizer F as input, in 3.1 it takes parameters and in 3.2 an RL algorithm\n\n            - G in 3.1 is not defined : is it the regret ? \n\n            - \\cal{H} is not defined\n\n            - F_\\eta not well defined, we do not really know what controls \\eta\n\n            - Algo 1 : \"Compute meta-gradient for \\eta with updated \\theta (eq 2)\" ==> eq 2 is an expectation of expected return, not a gradient\n\n            - Algo 1 : \"update \\Lambda with R and phi\" => what does it concretely mean ? You store the regret associated with phi in the buffer ? \n\n            - Algo 1: What means initialize lifetimes? How is it done ? \n\n \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words about our problem, method and results. We appreciate the time you have taken to give such extensive feedback regarding our formalization. We address each of your concerns below and refer you to the **edits described in our summary response**, which we believe will resolve much of the stated confusion. If you have time, we encourage you to revisit the manuscript’s formalism after reading our edits and clarifications, which we hope will lead you to the same clarity and ease of understanding described by the other reviewers.\n\n* Thank you for spotting this, there is a mistake where the bootstrap vectors are not subscripted with \\theta, which is likely the source of the confusion (**see line 77 edit**). However, there’s a minor mistake in your question, since the output of LPG is “\\hat{y}\\_t, \\hat{\\pi}\\_t”, not “y\\_t, \\pi\\_t”. Hopefully, this answers your questions since, as we state, “LPG outputs targets \\hat{y}\\_t, \\hat{\\pi}\\_t” for y\\_t and \\pi\\_t (targets being denoted by the hat), which are distinct from the agent outputs contained in x\\_t, being the “probability of the chosen action \\pi\\_theta(a\\_t|s\\_t) and bootstrap vectors for the current and next states y\\_\\theta(s\\_t), y\\_\\theta(s\\_{t+1})”.\n* Regarding the | notation in U_\\eta(x\\_t|x\\_{t+1}...x\\_T), you are correct that this is done to reflect the reverse-recurrent processing of tokens in LPG. Without this constraint, this function could be generalized to U\\_\\eta(x\\_t,x\\_{t+1},...x\\_T). However, we believe this notation is instructive for understanding the working of LPG, which processes each token individually based on the recurrent state from all future tokens.\n* We have now included the definition of \\eta, in order to make clear that it parameterizes LPG (**see line 74 edit**). On lines 68-71, we present an intuitive interpretation of the information learned by \\eta, followed by formally defining the function it parameterizes on lines 74-78. The loss for \\eta is presented immediately after in the following subsection (Equation 2) once we have formally defined the PMO problem setting in the context of UED. We hope this makes \\eta clearer; if not, is there any other information you believe would do so?\n* This confusion also likely results from our omitted definition of \\eta (**see line 74 edit**). y\\_\\theta is defined as the function producing “bootstrap vectors for the current and next states y\\_\\theta(s\\_t) and y\\_\\theta(s\\_{t+1})”, which share the “agent parameters \\theta” with \\pi\\_theta. On line 68-71, we provide an explanation of bootstrap vectors and their relation to critics in RL, making clear the information they capture.\n* Whilst this is a common abuse of notation, we already make a point of explaining the overloading of function V each time it is performed (lines 135-136 and 149) to avoid any confusion.\n* This is a good spot, thank you. We have added a definition of G to our edits (**see line 135 edit**).\n* \\cal{H} is a capital \\eta, denoting the space of the variable \\eta. Whilst this use of capitalization for parameter space is common, we acknowledge that capital \\eta is uncommon so we have added an explicit definition on line 135 in our edits to avoid confusion (**see line 135 edit**).\n* \\cal{F} is defined as “a meta-optimizer \\cal{F}: \\Theta \\times \\bb{T} -> \\Theta” on line 95. The statement “the update rule parameterized by \\eta” on lines 136-137 and our new definition of \\eta (**see line 74 edit**) should now make its meaning clear.\n* Algorithm 1 is presented in pseudocode and intended to give a high-level overview of GROOVE’s meta-training, much like Figure 2. This is done to improve the reader’s ease of understanding the method since exhaustively defining all operations here would make the algorithm opaque. All algorithm components are individually detailed in greater depth elsewhere in the manuscript. With this in mind, we respond to each of your related comments below:\n    * It is true that Equation 2 is not a gradient. However, this reference is intended to contextualize a line of non-formal pseudocode rather than formally define the gradient computation, and it trivially follows that the meta-gradient is the gradient of the meta-objective given in Equation 2, so we do not believe that restating it separately adds clarity.\n    * That’s right, the update primarily stores the regret associated with phi in the buffer. However, the entire PLR procedure is more involved than that and would make the algorithm significantly more verbose to include in its entirety (see the original PLR work, [1]). Therefore we define \\Lambda as the “PLR level buffer” in the algorithm rather than outlining the entire PLR procedure here.\n    * Initializing a lifetime involves sampling the level \\phi from the “PLR level buffer \\Lambda” and the agent parameters \\theta from their “initialization function p(\\theta)”. This is presented in the algorithm, with the line “Reinitialise lifetime (\\phi, \\theta) ~ \\Lambda \\times p(\\theta)”.\n\nWe thank you again for taking the time to find these oversights and hope that our edits and clarifications resolve any remaining confusion. In light of these and the opinion of the other reviewers that the paper is “well-structured/organized” (qWd8, f1yc), “easy to understand” (qWd8), and “provides a clear explanation of the underlying concepts” (TbjH), we wonder if you would be willing to increase your score in order to reflect the contributions that you mention?\n\n[1] M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940–4950. PMLR, 2021\n"}, {"review_id": "vruZTN5lGL", "reviewer": "Reviewer_f1yc", "summary": "This paper focuses on the problem of learning update rules in reinforcement learning. By combining the advantages of the Learned Policy Gradient (LPG) algorithm and the Unsupervised Environment Design (UED) technique, the authors propose a meta RL algorithm, namely, GROOVE (General RL Optimizers Obtained Via Environment Design), to address the generalization problem for meta RL algorithms are applied to unseen environments. Accounting for the curricula generation of the meta-learned optimizer, the authors also design a metric named algorithmic regret (AR) to evaluate the regret during the meta-training, then be used to guide the learning of meta-learner. A series of experiments also present that GROOVE outperforms baselines on the Atari games.", "questions": "Whether different antagonist agent will affect the performance?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized, and the authors have a throughout introduction for the related work.\n2. Experiments show that the designed AR outperforms existing metrics.", "weaknesses": "1. The approximation of algorithmic regret seems heavily relies on the choice of an oracle algorithm as UEDs, i.e., the choice of antagonist agent, which would be a performance bottleneck for GROOVE. Thus, there would be better if the authors demonstrate an ablation study on using different antagonist agents, i.e., training different RL algorithms.\n2. The novelty is limited, as GROOVE is a direct combination of UED and LPG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback about our writing and results. We have introduced **new results and edits in our summary response**, which we encourage you to read. Please find our response to each of your comments below:\n\n1. Thank you for this suggestion, we agree that this is an interesting idea and have now performed this experiment, comparing random, expert, A2C, and PPO antagonists (**see new results**). The results verify the need for general RL algorithms as the AR antagonist and demonstrate how the choice of RL algorithm impacts generalization.\n2. Whilst it is true that GROOVE builds on LPG and PLR, the primary message of our paper is that combining these existing methods is insufficient without our novel component (AR). This is demonstrated by our ablation study in Section 4.5, which shows the failure of existing metrics and the success of AR. To make this clearer, we have added a list of contributions, including this point, to the introduction (**see manuscript edits**).\nWe also note that our PMO formalism, the application of environment design to PMO, and our analysis of how meta-training distributions impact generalization in this setting are entirely novel.\n\nGiven the addition of your suggested experiment, along with our clarification and edit highlighting the paper’s novelty, we hope that you will consider raising your score to a stronger accept? Thank you!\n"}]

Author rebuttal:
Thank you to all of the reviewers for their detailed and insightful feedback.

We appreciate the positive comments describing our problem setting as **important** (8Dkv, 4F7T) and **well-motivated** (TbjH), in addition to our proposed method containing **innovative components** (4F7T), with Algorithmic Regret (AR) being described as **novel/new** (8DkV, qWd8, TbjH). Furthermore, we are thankful that **all reviewers provide positive feedback regarding our results**, describing them as great (8DkV) and interesting (4F7T), whilst showing the method is well-supported (TbjH), achieves superior performance to existing metrics (qWd8, f1yc) and can even [transfer] from Grid-World to Atari (8DkV). Finally, we thank the reviewers for their kind words regarding the paper’s writing, specifically that it is **well-structured/organized** (qWd8, f1yc) and **easy to understand** (qWd8), providing a **clear explanation** of the underlying concepts (TbjH) and a **thorough introduction** to related work (qWd8, f1yc).

## Manuscript edits
* Reviewers 8DkV and 4F7T pointed out omissions in our formalism. As a result, we have **added the following correction and new definitions** to rectify their concerns,
    * *Line 74*: Define \eta as the meta-parameters of LPG,
    * *Line 77*: “y(s_t) + y(s_{t+1})” => “y_\theta(s_t) + y_\theta(s_{t+1})”,
    * *Line 135*: Define G as expected return and capitalized \eta as the space of the meta-parameters, \eta.
* Reviewers 8DkV and f1yc raised a concern over our proposed method, GROOVE, containing existing components from UED and PMO. Firstly, the application of UED to PMO is already a novelty. Furthermore, our results underscore that combining existing methods (PLR and LPG) is insufficient without our novel component (AR). In addition to this, reviewer TbjH suggested that we list contributions in the introduction to help readers capture this and other novelties faster. Therefore, we add the following **list of contributions** to the end of the introduction (*line 58*):
    * In order to distinguish this problem setting from traditional meta-RL, we provide a novel formulation of PMO using the Meta-UPOMDP (Section 2).
    * We propose AR (Section 3.2), a novel regret approximation for PMO, and GROOVE (Section 3.3), a PMO method using AR for environment design.
    * We analyze how features of the meta-training distribution impact generalization in PMO (Section 4.2) and demonstrate AR as a proxy for task informativeness (Section 4.3).
    * We comprehensively evaluate GROOVE against LPG, demonstrating improved in-distribution robustness and out-of-distribution generalization on Min-Atar, Atari, and Procgen (Section 4.4).
    * We perform an ablation of AR, our novel component, demonstrating the insufficiency of existing methods (PLR and LPG) without AR, as well as the impact of the antagonist agent in AR (Section 4.5).
    * We release our implementation of GROOVE, PLR and LPG, capable of single-GPU meta-training in 3 hours.

## New results (see PDF)
* Reviewers TbjH and f1yc suggested an **ablation of the antagonist agent in AR** to determine its impact on the method’s effectiveness, whilst reviewer 8DkV asked for further insights about AR. In response to both of these, we have performed the suggested ablation and will add it to Section 4.5.
    * *Figure 9*: On Min-Atar, using a random or optimal agent as the antagonist for AR results in lower performance than using A2C or PPO on all environments. Furthermore, using A2C achieves higher performance than PPO on all environments.
    * *Figure 10*: PPO achieves lower performance than A2C on Grid-World, with a larger gap on difficult, handcrafted Grid-World levels. This explains the previous results, as PPO will be inferior at identifying difficult levels when used as the AR antagonist. Furthermore, the update parameterized by LPG is capable of representing A2C, but not PPO. This implies that levels solvable by A2C should also be solvable by LPG, making them useful for training. In contrast, PPO may identify levels that cannot be solved without components found in PPO (clipping, mini-batch iterations) but not LPG.
* Reviewer 8DkV suggested we further evaluate **GROOVE vs LPG on Procgen**, which we have now included and will add to Section 4.4.
    * *Figure 11*: After meta-training on Grid-World, we observe superior GROOVE performance on 2 out of 4 Procgen environments, superior LPG performance on 1 environment, and no difference on the remaining environment. We note that A2C is very weak on Procgen, failing to learn on the majority of environments, so we selected a subset of Procgen levels that A2C managed to learn in preliminary experiments. Procgen poses a robustness challenge that has required an extensive amount of further research to solve, using components not found in LPG or GROOVE.

In summary, we have introduced edits to address the reviewers’ concerns regarding writing and novelty, in addition to new results to provide further insights about our method. In light of these, we welcome any further feedback and hope the reviewers will consider raising their original scores.

Meta-review: This paper is concerned with enhancing the performance of reinforcement learning algorithms across diverse tasks using a meta-learning approach. The proposed method, called GROOVE, incorporates unsupervised environment design and introduces algorithmic regret to facilitate task generalization, resulting in superior performance compared to LPG in terms of generalization ability. The idea looks interesting and novel, and the proposed method is sound with good experimental support.
Decision: Accept (poster)