SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Aggregate performance on Min-Atar, with standard error shaded over 5 independent runs—PMCC is given for Random levels. A per-task breakdown is provided in the supplementary materials.
4.3 Algorithmic Regret Identifies Informative Levels for Generalization
In Section 3.2, we hypothesize that AR identifies informative levels for meta-training. Before evaluating auto-curricula generated with this metric (Section 4.4), we evaluate static curricula generated by AR against random and handcrafted curricula. To achieve this, we train an LPG instance to convergence and collect a buffer of  $10\mathrm{k}$  unseen levels, ranked by the final AR of the model. We then train a new LPG instance on the highest-scoring levels, controlling for task diversity by subsampling variable-sized level sets from the buffer.
At all sizes of training environment set, we observe improved OOD transfer performance when training on high-AR levels compared to training on the same number of random levels (Figure 3). Furthermore, training with high-AR levels outperforms training over the same number of handcrafted levels, and far exceeds the performance of the fixed handcrafted set as the number of levels is increased, without requiring any human curation. These results validate AR as an effective metric for automatically generating informative curricula.
However, we note that the performance gap between high-AR and random levels decreases as the number of levels grows. This is not surprising, since the high-AR levels are generated by ranking and selecting the levels with the highest AR for each training set size, leading to a natural dilution in average AR. This also highlights the need for automatic curriculum generation throughout metatraining rather than training on static curricula, which we examine further in Section 4.4.
Figure 4: Aggregate performance metrics on Atari—shaded area shows  $95\%$  stratified bootstrap confidence interval (CI) over 5 seeds, following methodology from Agarwal et al. [2021]. Higher score is better for IQM and lower score is better for optimality gap.
4.4 Adversarial Environment Design Improves Generalization
We now evaluate the impact of environment design on generalization performance by comparing GROOVE to LPG, thereby evaluating the same meta-optimizer with and without environment design. After meta-training both methods on Grid-World, we first evaluate on randomly-sampled, unseen Grid-Worlds (Figure 5). In addition to GROOVE achieving higher mean performance, we observe increased robustness with GROOVE consistently outperforming LPG on their lower-scoring half of tasks. Notably, LPG fails to achieve greater than  $75\%$  A2C-normalized return on  $187\%$  more tasks than GROOVE. Despite this, GROOVE and LPG achieve comparable performance on their higher-scoring half of tasks, suggesting that GROOVE increases in-distribution robustness without reducing performance on easy tasks.
Figure 5: A2C-normalized score distribution on unseen Grid-World levels, shaded area shows a  $95\%$  CI over 10 seeds.
To evaluate generalization to challenging environments, we evaluate GROOVE against LPG on the Atari benchmark. GROOVE achieves superior per-task performance to LPG, achieving higher mean score on 39 vs. 17 tasks (Figure 1), with equal performance on one task (Montezuma's Revenge). Comparing aggregate performance, we observe significant increases in both IQM and optimality gap from GROOVE against LPG (Figure 4). While both of these methods achieve inferior performance to state-of-the-art, manually-designed RL algorithms [Hessel et al., 2018], the improvement from GROOVE highlights the importance of the meta-training distribution and potential for UED-based approaches when generalizing to complex and unseen environments.
4.5 Algorithmic Regret Outperforms Existing Metrics
Finally, we evaluate the quality of our proposed environment design metric, AR, against existing metrics from UED (Figure 6). We compare to L1 value loss and positive value loss (Section 3.2) due to their consistent performance in prior UED work, as well as regret against an optimal policy (as is analytically computable on Grid-World), which serves as an upper bound on regret. As a baseline, we also evaluate uniform scoring, which is equivalent to domain randomization (i.e. standard LPG).
Figure 6: Evaluation of our proposed environment score function, AR, against uniform scoring, optimal-policy regret and baselines from UED literature. Mean return and standard error over 10 random seeds are shown.
AR achieves the highest performance on all tasks, significantly outperforming optimal-policy regret on all tasks and each of the other baselines on at least two out of four tasks. Furthermore, the value-loss metrics only significantly outperform uniform scoring on a single task, with positive value loss underperforming it on all others. This failure to identify informative levels for generalization highlights the challenge in transferring existing UED methods to PMO and the effectiveness of AR.
Whilst it is surprising that optimal-policy regret, which uses privileged level information, underperforms AR with an A2C antagonist, we hypothesize that the non-optimal performance and generality of handcrafted algorithms is a benefit for identifying informative levels. Optimal-policy regret does not account for training time, making it equivalent to an antagonist optimizer which always returns the optimal policy parameters, a setting likely to identify artificially difficult levels. To investigate this, we perform an comparison of A2C, PPO, random and expert antagonist agents (see supplementary materials), finding that using A2C or PPO antagonists outperforms random or expert agents.
In this work, we examine the impact of training distribution for meta-learning general RL algorithms. More broadly, our work aligns with the "AI-generating algorithm" [Clune, 2019, AI-GA] paradigm, building upon two of the three pillars (learning algorithms and environments). In this section, we outline prior work in each of these fields and explain their relation to this work.
Policy Meta-Optimization A common approach to PMO is to optimize RL algorithm components via meta-gradients [Xu et al., 2018, 2020]. In particular,  $\mathrm{ML}^3$  [Bechtle et al., 2021] uses metagradientsto optimize domain-specific loss functions that transfer between similar continuous control tasks. MetaGenRL [Kirsch et al., 2020] expands upon this to optimize general loss functions that transfer to unseen tasks. LPG [Oh et al., 2020] discovers a general update rule that transfers from simple toy tasks to Atari environments. We choose to focus on LPG since it displays radical out-of-distribution transfer and imposes minimal structural bias on the learned update rule.
An alternative approach uses Evolution Strategies [Rechenberg, 1978, Salimans et al., 2017] to optimize RL objectives. EPG [Houthooft et al., 2018] evolves an objective function parameterized by a neural network that transfers to similar MuJoco environments, whilst DPO [Lu et al., 2022] evolves an objective that transfers from continuous control tasks to MinAtar environments. Other approaches to PMO symbolically evolve RL optimization components such as the loss function [Co-Reyes et al., 2021, Garau-Luis et al., 2022] or curiosity algorithms [Alet et al., 2020]. PMO is an instance of Auto-RL [Parker-Holder et al., 2022b], which automates the discovery of learning algorithms.
Alternative Approaches to Meta-Reinforcement Learning PMO belongs to the subclass of many-shot meta-RL algorithms, which pose the setting of learning-to-learn given a substantial number of inner-loop environment interactions. An alternative class of approaches to this learn "intrinsic rewards", which augment the RL objective in order to improve learning. Alet et al. [2020] achieve this by meta-learning a program to transform the agent's objective, whilst Veeriah et al. [2021] propose a hierarchical method which meta-learns transferable options.
However, the majority of work in meta-RL has instead been on few-shot learning [Beck et al., 2023b]. RL² [Duan et al., 2016, Wang et al., 2016] use a black-box model for this, by representing both the policy and update rule with a recurrent neural network. A range of extensions to RL² have been proposed, which augment the original model with auxiliary task-inference objectives [Humplik et al., 2019, Zintgraf et al., 2020], additional exploration policies [Liu et al., 2021] and hypernetwork-based updates [Beck et al., 2023a]. Parameterized policy gradients are an alternative approach, with Model-Agnostic Meta-Learning [Finn et al., 2017, MAML] being the seminal method. MAML meta-learns a shared neural network initialization, such that it rapidly adapts to new tasks when optimized with policy gradients in the inner loop. Follow up work to this has proposed partitioning parameters [Zintgraf et al., 2019], modulating parameters for multimodal distributions [Vuorio et al., 2019] and investigated the reasons for its effectiveness [Raghu et al., 2019].
Unsupervised Environment Design Unsupervised Environment Design (UED) was first proposed by Dennis et al. [2020] with the introduction of PAIRED, which trains a level generator for a single agent with minimax regret. GROOVE is based on PLR [Jiang et al., 2021b,a], which builds upon this objective by instead curating a buffer of high-regret levels. PLR remains one of the state of the art UED algorithms, which has been extended to consider multi-agent settings [Samvelyan et al., 2023], curriculum-induced covariate shift [Jiang et al., 2022] and more open-ended environment generators [Parker-Holder et al., 2022a]. Aside from regret, environments can also be selected to induce diversity in a population of agents [Brant and Stanley, 2017], most famously in the POET algorithm [Wang et al., 2019] which evolves a population of highly capable specialist agents.
UED falls more broadly into the field of open-endedness [Soros and Stanley, 2014], which attempts to design algorithms that continually produce novel and interesting behaviours. We take a step towards more open-ended algorithms by combining UED with meta-learning, thus discovering both algorithms and environments in a single method. Previous works combining these two AI-GA pillars include Team et al. [2023] who introduce a memory-based agent capable of human-timescale adaptation, and OpenAI et al. [2019] who train a policy capable of sim-to-real transfer to control a Rubik's Cube. Unlike our work, neither of these meta-learn general RL algorithms that can transfer to far out-of-distribution environments such as Atari.
In this paper, we provide the first rigorous examination of the relationship between meta-training distribution and generalization performance in policy meta-optimization (PMO). Based on this analysis we leverage ideas from Unsupervised Environment Design (UED) and propose GROOVE, a novel method for PMO, in addition to a novel environment design metric, algorithmic regret (AR). Evaluating against LPG, we demonstrate significant gains in generalization from Grid-Worlds to Atari, as well as increased robustness to challenging in-distribution tasks. Finally, we identify AR as a critical component for applying environment design to PMO, demonstrating its effectiveness on Min-Atar, where prior UED metrics fail to outperform random sampling.
We acknowledge limitations in our work, largely driven by computational constraints. Given the huge number of environment interactions during meta-training, we heavily leverage our GPU-based Grid-World implementation in lowering training time. This limits our analysis to these fast but simple environments, meaning our conclusions cannot be guaranteed to generalize to more complex environments. Due to the cost of meta-testing on the large-scale Atari benchmark, our evaluation is also limited in variety of benchmarks, however we believe the diversity of the tasks in this domain sufficiently captures a range of challenges in RL.
By releasing our implementation—which is capable of meta-training these models on single GPU in hours, rather than days—we hope to spawn future work in this area from academic labs. In particular, these results may be scaled to training distributions with more complex and diverse environments, leading to the discovery of increasingly general RL algorithms.
The authors thank Junhyuk Oh, Tim Rocktäschel and the anonymous NeurIPS reviewers for their helpful feedback that improved our paper. Matthew Jackson is funded by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems, and Amazon Web Services.
R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021. 4
F. Alet, M. F. Schneider, T. Lozano-Pérez, and L. P. Kaelbling. Meta-learning curiosity algorithms. CoRR, abs/2003.05325, 2020. URL https://arxiv.org/abs/2003.05325. 5, 5
M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021. 1
O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020. 1
A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5): 834-846, 1983. doi: 10.1109/TSMC.1983.6313077.2.2
S. Bechtle, A. Molchanov, Y. Chebotar, E. Grefenstette, L. Righetti, G. Sukhatme, and F. Meier. Meta learning via learned loss. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4161-4168. IEEE, 2021. 5
J. Beck, M. T. Jackson, R. Vuorio, and S. Whiteson. Hypernetworks in meta-reinforcement learning. In Conference on Robot Learning, pages 1478-1487. PMLR, 2023a. 5
J. Beck, R. Vuorio, E. Z. Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023b. 1, 5
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484-489, 2016. 1
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. Science, 2017. 1
L. Soros and K. Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In ALIFE 14: The Fourteenth International Conference on the Synthesis and Simulation of Living Systems, pages 793-800. MIT Press, 2014. 5
R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981. 1
A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023. 5
V. Veeriah, T. Zahavy, M. Hessel, Z. Xu, J. Oh, I. Kemaev, H. P. van Hasselt, D. Silver, and S. Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861-29873, 2021. 5
R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in neural information processing systems, 32, 2019. 5
J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016. 5
R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019. 5
Z. Xu, H. P. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems, 31, 2018. 5
Z. Xu, H. P. van Hasselt, M. Hessel, J. Oh, S. Singh, and D. Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 33:15254-15264, 2020. 5
K. Young and T. Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019. 4.1
L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019. 5
L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: a very good method for bayes-adaptive deep rl via meta-learning. Proceedings of ICLR 2020, 2020. 5
Table 1: Summary of notation used in this work.
Symbol	Definition
Meta-UPOMDP
A,S,O	Action, state and observation spaces.
T,I,R	Transition, observation and reward functions.
γ	Discount factor.
φ ∈ Φ	Free parameters of the environment.
θ ∈ Θ	Agent parameters, shared between actor and critic/bootstrap function.
T	Sequence of transitions (O × A × R × O), denoted task “experience”.
Policy Meta-Optimization
Fη: Θ × T → Θ	Agent optimizer.
η ∈ H	Agent optimizer parameters.
Vφ,θ0(Fη)	Expected return of Fη at the end of training, given task φ and agent initialization θ0.
Vφ(η)	(Shorthand) Expected return of Fη at the end of training on task φ, over a distribution of agent initializations p(θ0).
Learned Policy Gradient [Oh et al., 2020]
πθ: A, O → [0,1]	Agent policy.
yθ: O → [0,1]n	Agent bootstrap function—a generalization of value critics from RL, outputting a vector with semantics determined by the learned optimizer.
Uη: T → [0,1]n × R	LPG target function, outputting bootstrap function and policy targets帽子 and帽子 at time step t, conditioned on all future transitions.
B Hyperparameters
B.1 GROOVE
Hyperparameters shared between GROOVE and LPG were tuned using LPG on Grid-World, then transferred to GROOVE without further tuning. The additional GROOVE hyperparameters (regarding the level buffer) were then tuned separately on Grid-World.
Table 2: GROOVE/LPG hyperparameters
Hyperparameter	Value
Optimizer	Adam
Learning rate	0.0001
Discount factor	0.99
Policy entropy coefficient (β0)	0.05
Bootstrap entropy coefficient (β1)	0.001
L2 regularization coefficient for π (β2)	0.005
L2 regularization coefficient for帽子 (β3)	0.001
Level buffer size	4000
Replay probability	0.5
Number of interactions per agent update	20
Number of agent updates per optimizer update	5
Number of parallel lifetimes	512
Number of parallel environments per lifetime	64
Algorithmic regret baseline algorithm	A2C
B.2 Agents
Agent hyperparameters were based on tuned A2C agents, before being fine-tuned with LPG. Since we meta-train on a continuous distribution of Grid-World environments, we do not use the agent hyperparameter bandit proposed by Oh et al. [2020] for meta-training.
Table 3: Agent hyperparameters—architecture descriptions  $D(N)$  and  $C(N)$  respectively refer to dense and convolutional layers of size  $N$ ; ReLU activations are used throughout.
Hyperparameter	Grid-World	Environment
Min-Atar	Atari
Architecture	Tabular	D(64)-D(64)	C(32)-C(64)-C(64)-D(512)
Optimizer	SGD	Adam	Adam
Learning rate	40	0.0005	0.0005
Bootstrap KL coefficient (αy)	0.5	0.5	0.5
Train steps	2500	100,000	100,000
Agent seeds per LPG seed	64	16	1
C Handcrafted Environments
For our handcrafted environment set, we use the set of five tabular Grid-World configurations from Oh et al. [2020]. Grid-World objects are defined by  $[r,\epsilon_{\mathrm{term}},\epsilon_{\mathrm{respawn}}]$ , where  $r$  represents the reward when collected,  $\epsilon_{\mathrm{term}}$  is the episode-termination probability and  $\epsilon_{\mathrm{respawn}}$  is the probability of the objectRespawning each step after collection.
C.1 Dense
Property	Value
Size	11 × 11
Objects	2 × [1,0,0.05], [−1,0.5,0.1], [−1,0,0.5]
Maximum episode length	500
C.2 Sparse
Property	Value
Size	13 × 13
Objects	[1, 1, 0], [−1, 1, 0]
Maximum episode length	50
C.3 Long Horizon
Property	Value
Size	11 × 11
Objects	2 × [1,0,0.01], 2 × [−1,0.5,1]
Maximum episode length	1000
C.4 Longer Horizon
Property	Value
Size	9 × 9
Objects	2 × [1,0.1,0.01], 5 × [−1,0.8,1]
Maximum episode length	2000
Note: size is increased from  $7 \times 9$  for consistency with our generalized Grid-World distribution.
C.5 Long Dense
Property	Value
Size	11 × 11
Objects	4 × [1,0,0.005]
Maximum episode length	2000
D Atari Training Curves
Figure 7: Atari training curves—environment names are highlighted according to highest evaluation return, asterisks (*) denote significant differences in evaluation return (5 seeds,  $p < 0.05$ ).
E Min-Atar Per-Task Performance
As expected, we observe increased noise when breaking down performance by individual Min-Atar tasks, however, the results from the majority of tasks support our earlier conclusions. Firstly, we observe a significant positive correlation between the number of random training levels and return on three of the four Min-Atar tasks, again demonstrating the impact of task diversity on generalization. When controlling for the number of levels, we observe improved performance after training on handcrafted, rather than random, levels on three of the four Min-Atar tasks. Furthermore, on Asterix, training on handcrafted levels results in higher performance than the largest set of  $2^{10} = 1024$  random levels, supporting our conclusion about level informativeness.
After training on high-AR levels, we observe an improvement against random levels on at least three of the four Min-Atar tasks for all sizes of training environment set up to  $2^{6} = 64$  levels. Beyond this, random and high-AR levels outperform each other on an equal number of tasks, however the dilution in mean AR for larger training sets makes this convergence unsurprising. Furthermore, high-AR levels are competitive with handcrafted levels at the same training set size and quickly outperform the fixed handcrafted set as more high-AR levels are added, demonstrating the effectiveness of AR at identifying informative curricula.
Figure 8: Generalization performance on Min-Atar, after meta-training LPG on variable-sized sets of Grid-World levels (5 seeds)—levels are selected through uniform-random sampling of all levels ("Random"), from the highest-regret levels of a previous LPG instance ("Max-AR"), or from a set of five handcrafted levels ("Handcrafted"). Pearson correlation coefficient is given for Random levels; significant positive correlations are marked with an asterisk (*).
F GROOVE vs. LPG Procgen Evaluation
After meta-training on Grid-World, we observe superior GROOVE performance on 2 out of 4 Procgen environments, superior LPG performance on 1 environment, and no difference on the remaining environment. We note that A2C is very weak on Procgen, failing to learn on the majority of environments, so we selected a subset of Procgen levels that A2C managed to learn in preliminary experiments. Procgen poses a robustness challenge that has required an extensive amount of further research to solve, using components not found in LPG or GROOVE.
Figure 9: GROOVE and LPG training curves on Procgen (test performance, 5 seeds).
G Algorithmic Regret Antagonist Comparison
In order to investigate the impact of the antagonist agent on the performance of AR, we evaluated the performance of GROOVE with a range of antagonists (Figure 10). On Min-Atar, using a random or optimal agent as the antagonist for AR results in lower performance than using A2C or PPO on all environments. Furthermore, using A2C achieves higher performance than PPO on all environments.