OMNI: Open-endedness via Models of human Notions of Interestingness
Authors: Jenny Zhang
Joel Lehman
Kenneth Stanley
Jeff Clune
Venue: ICLR 2024 poster (2024)
Keywords: Open-endedness
Auto-Curriculum Learning
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: Open-endedness via Models of human Notions of Interestingness (OMNI) leverages foundation models to improve open-ended learning by focusing on tasks that are both learnable and interesting, advancing self-improving AI and auto-curricula.

Abstract: Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent can learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (e.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also $\textit{interesting}$ (e.g., worthwhile and novel). We propose solving this problem by $\textit{Open-endedness via Models of human Notions of Interestingness}$ (OMNI). The insight is that we can utilize foundation models (FMs) as a model of interestingness (MoI), because they $\textit{already}$ internalize human concepts of interestingness from training on vast amounts of human-generated data, where humans naturally write about what they find interesting or boring. We show that FM-based MoIs improve open-ended learning by focusing on tasks that are both learnable $\textit{and interesting}$, outperforming baselines based on uniform task sampling or learning progress alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus on next (i.e., auto-curricula), and could be seen as AI selecting its own next task to learn, facilitating self-improving AI and AI-Generating Algorithms.

Published as a conference paper at ICLR 2024
OMNI: OPEN-ENDEDNESS VIA MODELS OF HUMAN NOTIONS OF INTERESTINGNESS
Jenny Zhang! Joel Lehman
Kenneth Stanley* Jeff Clune 1.255
l Department of Computer Science, University of British Columbia Veccor Institute
3 Stoohasttic Labs *Maven "Canada CIFAR AI Chair
ABSTRACT
Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (c.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also interesting (e.g., worthwhile and novel). We propose solving this problem by Open-endedness via Models of human Notions of Interestingness (OMNI). The insight is that can utilize foundation models (FMs) as model of interestingness (Mol), because they already internalize human concepts of interestingness from training on vast amounts of guman--geerattee data, where humans naturally write about what they find interesting or boring. We show that FM-based Mols improve open-ended learning by focusing on tasks that are both learnable and interesting, outperforming baselines based alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus next (i.e., auto-curriclla,, and could be seen as AI selecting its own next task to learn, facilitating self- eeff-improving Al and AI-Generating Algorithms!
uniform task sampling learning progress
INTRODUCTION
Provided that the real, significant challenges of Al safety and existential risk can be solved (Critch & Krueger 2020 Bostromm 2002; Turchin & Denkenberger 2020, Ecoffet et al. (020). there are tremendous gains to be had by creating more powerful Al or even AGI. A great hope for AI is that one day it can produce breakthroughs that fundamentally improve the human condition. These so-far uniquely human advancements and discoveries are the hallmark of civilization, from the invention of the wheel, farming, vaccines, computers, and even rock and roll. Perhaps someday, Al could achieve such major breakthroughs automatically. What does Al need to possess to discover such new paradigms, only humans have until now?
Much discussed open-endedness research (Stanley et al. 2023). the ephemeral fuel behind civi- lization's 's prodigious output is the human intuition for interestingness. Drawing upon eons of human experience, we can sense potential even when we don't 't precisely know where it leads. Conventional
Reinforcement Learning (RL) tools (e.g., intrinsic motivation (Aubret et 2019 Pathak et al. 2017
Osband et al. 2018 Colas et al. 2022 Oudeyer et al.) 2007) and learning progress (Kanitscheider
al., 2021 Matüien et al. 2019 Portelas et al. [020 Graves al.) 2017 Kovač et al.] 2022]
Baranes & Oudeyer 2013) are so far only shadows of what such a human sense could do. However,
with the rise of foundation models (FMs) (Bommasani et al. 202T), such as large language models
(Radford et al. 2018), an intriguing prospect has arisen trained on vast troves of human experience, perhaps FMs have the potential to grapple for the first time with the critical question what is actually interesting to explore.
Open-ended learning algorithms, which could leverage such notion of interestingness, seek to create Al agents that, like humans, continuously learn variety of different skills within vast, complex,
'We recommend the version on arXiv (https: //wp///aaa/v///////////// org/ abs/ /2711.0171)) which is slightly longer and thus able to explain things more clearly and more fully discuss the implications of this work. Project wehsite: https: // www.. jennyzhangzt, cmni/

---

Published as a conference paper at ICLR 2024
ever-changing environment. The challenge addressed by interestingness is that, in such environments, there are an infinite number of possible tasks, requiring some method to choose which tasks to try to learn next at every point in training. Handcrafting curricula for training agents in open-ended environments can be extremely challenging due to the sheer number of tasks and the need to adapt to the agent's skill level and learning progress. In pursuit of an algorithm that is applicable in any domain and enables perpetual learning, handerafting curricula proves to be an impractical solution. Learning progress methods are a type of auto-curriculum approach that estimates which tasks are at appropriate difficulty levels for the agent to learn from (Kanitscheider et al., 2021; Matiisen et al. 2019; Portelas et al. 2020] Graves et al.) 2017 Kovač et al. 2022] Baranes & Oudeyer| 2013). However, such methods can be distracted by learnable yet uninteresting tasks. For example, an agent could be bogged down indefinitely with rearranging silverware in slightly new configurations, hindering it from trying other interesting tasks. Even after filtering for tasks that the current agent can learn, countless learnable yet uninteresting tasks may persist (c.g.. slight variations of previously learned tasks). A key challenge in ppen-endedness research is the inability to quantify and thus focus on tasks that are not only learnable but also interesting. There have been many attempts to quantify interestingness, but, we detail in Section [ such simple, hand-crafted formulas consistently fall short of truly capturing the essence of interestingness, creating crippling pathologies. This paper proposes different path forward.
To borrow from Newton, modern Al sees further by standing on the shoulders of giant human datasets. Training on vast amounts of human- -ennanneeeertedd data has proven powerful in many cases, such as text generation (e.g., GPT-3 (Brown et al.) (0200), image generation (e.g.. DALL-E (Ramesh et al.
202I)), and representation learning (e.g.. CLIP (Radford al. 202T) ). We propose Open-endedness via Models of human Notions of Interestingnest (OMNI). OMNI leverages the power of FMs that have already been trained on extensive human- human-generated data and have an inherent understanding of human notions of interestingness (Brown et al., 2020 Open 2023). OMNI utilizes FMs as a model of interestingness (Mol) to focus on tasks that are: (1) learnable, at appropriate difficulty levels for the agents to learn from, and (2) interesting, roughly meaning worthwhile to learn and sufficiently novel. The concepts of "interestingness" "worthwhile". and "novelty" are challenging to explicitly define, let alone quantify, which is precisely what OMNI addresses. Humans can intuitively assess these qualities despite their elusive and abstract nature, echoing Justice Potter Stewart's 's sentiment of "I know it when see it" (Stewart 1964). The goal of OMNI is emulate this human capacity for nuanced interestingness judgement in open-ended learning. We evaluate OMNI on three challenging domains, Crafter (Hafner 202T (a 2D version of Minecraft), BabyAl (Chevalier-Boisvertt 2018) (a 2D grid world for grounded language learning), and A/2-THOR (Kolve et al.) (017) (a 3D photo-realistic embodied robotics environment). OMNI outperforms baselines based on uniform task sampling or learning progress alone. Overall, OMNI has the potential to significantly enhance the ability of to intelligently select which tasks to concentrate on next for endless learning and marks step towards self-improving Al and AI-Generating Algorithms (Clune 2019).
al.
RELATED WORK
2.1
AUTO-CURRICULUM LEARNING
Training neural networks with a curriculum has been extensively studied (Bengio et al. 2009).
auto-currrculum learning has emerged as a promising research area in RL. (Kanitscheider et al.
2021 Matiienn al.] 2019 Portelas al., 2020 Graves et al., 2017: Kovač al. 2022 Baranes
& Oudeyer 2013 Lehman & Stanley| 201 la
Eysenbach et al. 2018) Wang et al. 2019 2020
Akkaya al. 2019 9 Florensa et al. 2018 Zhang et al. 20200] Campero et al.] 2220 OpenAl et al.
2021 Dennis et al. 2020 Gur et al., 202 Jiang et al. 2021] Dharna et al. 20222, with approaches
based on success probabilities and reward thresholds (Wang et al. [0||2019 9 2020 Akkaya al. 2019
Campero et al.] 2020; Tan et al. 2023), regret (Dennis et al. [2200] Gur et al.) [222]] et al.
M et al. 2019 Portelas et al.) 20200 Graves et al. 2017 Kovač et al. [022]|Barnneee Baranes & Oudeyer 2013). Static threshold-based approaches provide a straightforward method for curriculum design. These approaches involve setting fixed criteria for tasks based on their difficulty or complexity. An agent progresses to the subsequent task in a predefined order only after mastering a simpler one. To handcraft an effective curriculum, one would have to understand the relative difficulty of each task and identify tasks of suitable difficulty corresponding to each phase of the agent's 's learning trajectory. Doing this in a vast task space is extremely difficult or even impossible. Regret-based methods compute per-task regret by
2021), or learning progress (Kanitscheider et 2001| Matiisen
P
2

---

Published as a conference paper at ICLR 2024
taking the difference between the maximum known return and the average return over multiple
rollouts. Regret-basec methods typically select tasks with high regret. under the assumption that these tasks still offer substantial learning opportunities (Dennis al. 2020; Gur et al. 2021 Diang et al. 202T). However, in stochastic environments, this approach may favor more stochastic and less learnable tasks instead of less stochastic and more learnable ones (Kanitscheider et al.) (021).
curricula have the potential to mitigate these issues by monitoring the agent's progress and adapting the task selection accordingly (Kanitscheider et al. 202T] Matiisen
Leamning-progres--aaee
et al. 2019: Portelas et al.
2020 Graves et al., 2017 Kovač et al.] 2022 Baranes & Oudeyer
2013). Kanitscheider et al. (021)) demonstrated that learning progress can be measured reliably
and that fearning-progress. progress- seasgress-bbeedd based curricula can be applied to hard RL problems at scale. Our work
extends the Serniing-progress based curriculum proposed by Kanitscheider et al. (202T). A notable limitation of existing auto-curricula approaches is their inability to distinguish between interesting and uninteresting tasks. Despite filtering for learnable tasks, open-ended environments may still contain infinite learnable but uninteresting tasks. This paper proposes novel method for identifying and filtering interesting tasks and integrates it with a
baaded auto-curriculum.
2.2 ATTEMPTS TO QUANTIFY INTERESTINGNESS
Many prior research papers have tried to encourage predefined metric of novelty, diversity, explo- ration, or open-endedness, but doing so requires quantifying these ineffable qualities. The problem is that optimizing these quantitative measures often leads to undesirable or pathological outcomes, resulting in output that conforms to the defined metrics, rather than achieving the intended goal
(Aubret et al. 2019; Pathak et al., 2017; Osband et al., 2018) Colas et al. 2022; Oudeyer et al. al. [2020] Lehman & Stanley |201 la Mouret 2011] Mouret & Clune 2015
2007
Etcheverry
Lehman & Stanley| |0T et al. 2018 Bellemare et al. 2016]] Ecoffet et al. 2019 2021 I
Mendonea al. [223]Leemmn Lehman & Stanley [2012] Lehman al. 2020 Nguyen et 2l.5520155 Auerbach
& Bongard 2010 Zhou et 2023] Cal et al.) 2023). As Goodhart's ' law posits, "when a measure becomes target, it ceases to be a good measure" (Strathern 1997). For example, an agent might exploit a novelty measure by generating many superficially different but ultimately trivial solutions, thus undermining the goal of discovering genuinely interesting outcomes (Lehman Stanley 201 Ia). Similarly, based on how intrinsic motivation is measured, an agent could be biased towards certain types of solutions, leading to a narrow exploration of the problem space rather than developing diverse and valuable insights and innovations (Aubret et al.) 2019). Attempting to manually specify a criteria for what constitutes an interesting learning challenge is unlikely to yield satisfactory results. Instead, this paper proposes harnessing FMs to model ineffable human notions of interestingness, gleaned from large text corpora of existing numan-generann data (c.g. training on the Internet).
2.3 PRE-TRAINED FOUNDATION MODELS IN OPEN-ENDEDNESS
Large language models have recently shown a remarkable ability to capture rich knowledge on an extensive array of subjects from large-scale text corpora. They achieve impressive performance across a wide range of natural language processing tasks (Brown et al. 2020) OpenAT [023 Kenton
& Toutanooa | [Lu et al., 2019; Min et al.] 2021 [i et al. 2022 Colas et al. 2023) and display profound understanding of complex concepts such as physics. Consequently. they are utilized in
many robotics domains (Huang et al. 2022bb at Ahn et al. 2022 Yang et al. 2023 Lynch & Sermanet
2020 Sharma et al., 2021; Kant et al.J 2022, Kwon et al. 2023 Du et al., 2023] Driess et al.
2023). There has been growing interest in using them for task selection or generation. Some studies
have investigated the application of FMs in breaking down high-level instructions into a sequence
of sub-goals, which can be executed by an agent in zero-shot manner (Huang et al. 2022b a
Ahn et al. 2022; Yang et al.] 2023) Colas et al., 2023 Zhu et al. (023) or used to train modular sub-policies (Lynch & Sermanet 2020) Sharma et al. 2021). Kant et al. (2022) queries FMs for
zero-shot commonsense priors and apply them to planning task. Other studies have utilized FMs to
estimate success rates for given task or desired behavior (woon et al. 2023 Du et al., 2023 Colas
et al. 2023 Wang et al. 2023a b). Moreover, FMs have been employed to generate or explain tasks,
enabling structured exploration in various environments (Du et al.) 2023 Colas et al.: 2023 Wang
al. 2023a by Yuan al. 2023). OMNI differs from Du et al. (202)) by considering an agent's ' past
successes and employing FMs' commonsense knowledge for adaptive task selection. Unlike Wang et al. (2023a), which employs a code API generated by FMs. OMNI promotes direct action learning via environment interaction, demanding potentially higher computational resources but bypassing the need for and, critically, limitations of, dopain-specific code APls. While While et al. (2023) use
3

---

Published as a conference paper at ICLR 2024
deterministic environments and binary reward signals for trajectory success, OMNI adopts more nuanced approach in stochastic settings, recognizing that agents often improve over time and may not always achieve consistent success rates.
METHODS
3.1
PROBLEM FORMULATION
We train lask-conditioned agents, and formulate the RL. problem as a partially observed Markov decision process (Kaelbling al. 1998 defined by tuple AS, R, O, ?2, Observations E S depend on the new environment states 8 E S and actions taken a E A via O a). The task which the agent conditioned on is part of the environment state s. T(s' a) describes the dynamics of the environment. R(s, a) is the environment's 's reward function. 7 is discount factor. OMNI focuses on generating learnable and interesting tasks condition the RL agent on.
3.2 LEARNING PROGRESS CURRICULUM
The task pool in open-ended environments can be very large and diverse, making it challenging for an agent to learn effectively through uniform sampling. Most randomly sampled tasks are likely to be impossible (or least currently too hard for the agent learn.. To automatically identify tasks the frontier of the agent's 's capabilities, we extend the learning-progress, progress-bbseed curriculum (without the dynamic exploration bonus) from Kanitscheider et al. (2021).
The curriculum predominantly samples tasks with high learning progress, defined as an agent's recent change in task success probability. During training. the agent is periodically evaluated, and recent success probability estimate Precent is calculated by applying an exponential moving average (EMA) to the evaluated task success rates. Precent smoothed with a second, identical EMA to obtain a slower-to-dle change reflection Paradual of the success probability. Since tasks with low success probabilities are more likely to be novel and are harder to learn because the agent observes fewer successes, Precent and Paradual are reweighted to magnify the learning progress in tasks with low success probabilities and reduce the learning progress in tasks with high success probabilities. This reweighting also compensates for the temporal delay caused by the EMA (Figure 4 Bidirectional learning progress, the absolute difference between the reweighted Precent and Pgradual. is used to also focus learning on tasks where performance is degrading due forgetting. Sampling of training tasks is biased towards those that score the highest on this bidirectional learning progress measure. We propose extension to the approach from Kanitscheider et al. (202T), normalizing the task success rates with the success rates achieved by random action policy (Appendix A)
3.3
MODELING WHAT HCMANS FIND INTERESTING
An LP curriculum can be distracted by endless variations of uninteresting tasks. To address this challenge, Model of Interestingness (Mol) selects interesting tasks that offer substantial learning value. Humans often intuitively know what might be useful for learning new skills or achieving goals much later (Stanley & Lehman 2015). This is evident in children playing to unknowingly acquire skills, or scientists exploring new areas to uncover unexpected and beneficial knowledge for future endeavors. This paper presents two (of many possible) instances of the OMNI principle: in finite task spaces (Section (.3). and one in an infinite task space (Section 5.I). This section describes OMNI the former, first outlining the process of using an FM to determine which tasks are interesting. and then describing how the interestingness predictions are utilized to obtain task sampling weights.
Determining Interesting Tasks. This paper capitalizes on the capabilities of autoregressive FMs to emulate human notions of interestingness. FMs are pretrained on vast and diverse text corpora, enabling them to amass significant amount of world knowledge. We prompt the FM in few-shot manner by providing it with examples of choosing which tasks are interesting. It takes into account the agent's 's existing proficiency on a given set of tasks and suggests what humans would typically find interesting to learn next. Davinci GPT-3 (Brown et al. 2020)) was utilized for the Crafter experiments because it was the state-of-the -ftte-of-hhe-rt language model available when the experiments were run. GPT-4 used for the BabyAl experiments, which were conducted later. Appendices B and C show the full prompts.
(OpenAl 2023)
Sampling Weights. OMNI aims to improve open-ended learning by focusing on tasks that both learnable and interesting (Figure[H) The full OMNI algorithm is summarized in Algorithm Task sampling rates are first assigned based on the I.P curriculum, with higher rates for tasks with
4

---

Published as a conference paper at ICLR 2024
Then, FM- Mol predicts which tasks are interesting (Appendix!]) D.. Boring tasks have their sampling weights reduced by multiplying by 0.001. Finally, task sampling rates are normalized to probabilities that sum to 1.
higher learning progress (Section 3.2)
4 EXPERIMENTS IN A FINITE TASK SPACE
Agent
Diloctttion
Ball
Calect Wood
Caliet siar
Hake Blae
Key
Waddiverd
Hiae
Box
Aicafurmec
Calied
Door
Unlocked
111
Masa Seod
Histtes
Collec Damond
Open Door
Figure Crafter and BabyAl environments. (Left) Agent view in a procedurally generated Crafter world, showing terrain types, resources, and the agent's s inventory. (Middle) The 15 tasks considered interesting for Crafter analyses. Arrows indicate which tasks in the technology tree must be completed, often multiple times, along the way to perform more challenging tasks. (Right) Bird's-eye view a randomly generated Baby Al environment, showing different object types, colors, locations, and states. The agent is the red triangle and its view (sometimes occluded) is highlighted in light grey. In this example, the agent starts from the bottom right room, and is tasked to "go to a red ball". To succeed, the agent must open the green door (sometimes locked) reach the red ball,
4.1 CRAFTER ENVIRONMENT
We evaluate OMNI on Crafter (Hafner 2021), a 2D version of Minecraft that enables collecting and creating set of artifacts organized along technology tree. This means that certain tasks need to be completed, often multiple times, as prerequisites for other more challenging tasks (Figure []. Agents receive RGB pixel observations (64 64 resolution) of 9 x grid area surrounding their position within a 64* x 64 grid landscape that varies with each episode, offering complex and engaging testing ground. The agent is provided target task (represented with bag-of-word, encoding) as part of its observation and rewarded +1 upon successful completion of the conditioned task. We modify the game focus on gathering and crafting skills by eliminating the survival component. This removes the need for the agent to learn and continually apply survival tactics against enemies or for food gathering. The "sleep" and "place plant" actions are important for survival in the original game and have been omitted due to their reduced relevance in our modified context, which excludes the survival aspect. The original game consists 22 tasks, of which, the 15 tasks unrelated to survival are selected and considered interesting.
To investigate our hypothesis that focusing on interesting tasks with high learning progress will improve performance, we dilute the 15 interesting tasks with 90 "boring" tasks and 1023 "extremely approaches. Boring
challenging" tasks that serve as potential distractors for learning-progres--bee tasks are generated as numerical repeats of interesting tasks, e.g., "collect N wood" where N E [2, 10), analogous to how minor numerical variations of real-world tasks are less interesting than tasks that differ qualitatively. See Appendix K for the full list of boring tasks. Extremely challenging tasks represent tasks that are too difficult for the agent to complete at its current state learning, serving tasks that uniform sampling will waste time on, but that praminngprogress-beee based methods should successfully ignore. The agent is assumed to always fail at these extremely challenging tasks and hence is always assigned success rate of 0 for them. By analogy, consider the futility of attempting to cook 5-course meal before learning the basic skill of cutting a vegetable.
4.2 BABYAI ENVIRONMENT
We also evaluate OMNI on BabyAl (Chevalier-Boisvertt et al.) 2018), a readily available benchmark domain characterized by its partially observable 2D grid world environment (Figure D.. We test on the MiniBossLevel. While BossLevel is the most challenging level in BabyAL, we choose MiniBossLevel it has the same features BossLevel but with smaller room and lower probability of locked rooms, speeding up training. For each episode, the room layout and item configuration are randomly generated (using off off-thf-shelf configurations from Chevalier- Boisver et al. (018)). The grid world
5

---

Published as a conference paper at ICLR 2024
can have objects in six colors (red, green, blue, purple, yellow, grey), and of four types (key, ball, box, door). The agent randomly spawned at location in the 9 x 9 grid world, containing four 3 3 rooms. The agent's observation includes one-hot encodings of each of the 7 grid cells in front of the agent (observations are set to a special symbol if occluded), and a description of the task in natural language (embedded with look-up table and GRU, Appendix M for more details). The agent receives reward, proportional to the number of steps it took to finish, only when it has successfully completed the given task. While the Baby Language grammar (Chevalier-Boise Boissvrt et al. 2018) is limited to sequential tasks with a maximum of 2 instructions, we expanded this by permitting tasks with up to 5 instructions, resulting in 1364 unique tasks. Each task is sequence of instructions (GoTo, PickUp, OpenDoor, PuiNex(To), linked by the ordering constraint then. Object placements are randomized each episode. Tasks with same sequence of instructions but different object instances are considered the same when sampling (e.g., "go to a blue ball" and "go to a red key" are considered the same task "go to <object>").
4.3 RESULTS
Average Tosk Success Rates
OMNI LP Moi
Uniform
Success Rutes +02
No. of
25
training
(million)
training steps
Figure 2: Results in Crafter. (Left) Conditional success probabilities of all tasks in Crafter. Tasks are organized from simple to complex based on the prerequisite tasks that must be accomplished before completing the target task. Task names (left of each row) are readable in a digital format with zoom. (Right) Performance in Crafter on all tasks. While OMNI biases training towards interesting tasks, achieves higher average task success rates and learns more tasks than uniform sampling or choosing tasks based on learning progress alone, even across all tasks.
Both Crafter and Baby AI RL agents are trained with PPO (Schulman et al. 2017), a standard RL algorithm. Policy details and hyperparameters for the Crafter and BabyAl settings are in Appendices L and We compare the performance agents trained with: (1) Uniform sampling. (2) Learning Progress (LP) only, and (3) OMNI: Learning Progress with additional filtering by Model of Interestingness (OMNI: LP + Mol). Uniform sampling, the control, samples all tasks with equal probabilities. Uniform sampling the most naive and samples tasks that are too easy or difficult for the agent most of the time. LP samples tasks based on the calculated learning progress weights (Section 3.2) but distracted by the many boring tasks. OMNI: LP Mol focuses the subset of tasks with high learning progress that are also interesting (Section 3.3 All experiments are run for 100 million time steps and are repeated 10 times with different random seeds. Each experiment takes about 33 hrs for Crafter and 60 hrs for Baby AI 24GB NVIDIA A10 GPU with 30 virtual CPUs.
We evaluate our methods with two metrics: (1) the average task success rate, and (2) the number of tasks with success rates exceeding a predetermined threshold a. This study sets = 0.2, consistent with the selections made in related literature (Kanitscheider et al.) 2021 Team et al., (023). The first metric reflects the agent's average performance across all tasks, while the second metric captures the extent to which the agent is a generalist that has decent competency on many different tasks. These metrics calculated on the full task set (Figures]) 2 and [7. Metrics calculated on interesting tasks only are shown in Appendix(0) O All confidence intervals given are 95% median bootstrap confidence intervals obtained by resampling 1000 times. Confidence intervals reported with the following

---

Published as a conference paper at ICLR 2024
notation: stat (CI: lower upper) where stat is the median across runs. Shaded areas in graphs also indicate the 95% median bootstrap confidence interval obtained by resampling 1000 times.
Uniform Sampling. As expected, the results with uniform sampling are poor. Worse, the agents did not improve over time as most tasks sampled are too difficult or too easy for the agent and successes extremely sparse (Figures and (7. The agent is considered to have learned task if its conditional success probability on that task is at least 0.2. In Crafter, the agent learns 4 (CI: 4-6) 6) tasks (interesting or boring) and only 3 (CI: 2-3 interesting tasks. The agent achieves an average task success rate of 0.030 (CI: 0.026- 0.033) on interesting and boring tasks, and O. 103 (CI: 0.087 - 0. 0.120) on interesting tasks only. In BabyAI, the agent learns only (CI: 0- 1) task and achieves an average task success rate of 3.7c-3 (CI: 4.6e-3 5.0e-3) on all tasks.
Learning Progress Curriculum. By focusing on tasks with suitable difficulty, the agent learns to do a lot more tasks with higher success rates than uniform sampling. In Crafter, the agent learns 55 (CI: 54 56) tasks (interesting or boring) and 9 (CI: 9 11) interesting tasks. The agent achieves an average task success rate of 0.42 (CI: 0.41 - 0..3) on interesting and boring tasks, and 0.52 (CI: 0.50-0.56 on interesting tasks only. In Baby AI, the agent learns (CI: 4 - 6) tasks and achieves an average task success rate of 5.9e-3 (CI: 5.5e-3 6.2e-3) on all tasks. Across all metrics and in both domains, the differences in performance between LP and Uniform at 25%, 50%, 75%, and 100% of the way through training statistically significant (all p 1c-3, Mann Whitney U test), showing
that IP significantly outperforms uniform sampling (Figures 2 and 7
I.P samples tasks that are 7 8 14 When task's conditional success
at the frontier of the agent's capabilities (Figures 2
probability changes, LP focuses more on it. Hence, there will be more rollouts where the task is the given goal and thus more positive examples from which the agent can learn to solve the conditioned task. However, LP is distracted by boring tasks (Figures and [4p. When the conditional success probabilities of boring tasks change, LP allocates higher sampling weights them even though they are similar to other sampled tasks and might not expand the agent's 's range of skills.
Learning Progress a Model of Interestingness. To automatically select and focus on interesting tasks, an FM is prompted in a few-shot manner to predict which tasks are interesting. By combining L.P with an Mol, OMNI focuses on the subset of high learning progress tasks that are interesting. In Crafter, the agent learns 82 (CI: 80 - 87) tasks (interesting or boring) and 14 (CI: 14- 14) interesting tasks. The agent achieves an average task success rate of 0.56 (CI: 0.54 - 0.58) on interesting and boring tasks, and 0.78 (CI: 0.76- 0.80) interesting tasks only. In Baby the agent learns (CI: 7 - 10) tasks and achieves an average task success rate of 7.5e-3 (CI: 7.3e- - 7.7e-3) on all tasks. Across all metrics and in both domains, the differences in performance between OMNI and I.P at 25% 50%, 75% and 100% of the way through training are statistically significant (all p < 1e-3, Mann Whitney U test), showing that OMNI significantly outperforms L.P-only curriculum (Figures andd] 7). OMNI is not distracted by uninteresting yet learnable tasks, and focuses on the interesting tasks only (Figures S and 14). The trained agent not only achieves higher average task success rates, but also learns more challenging tasks faster (Figures and 7p.
OMNI:
We thus know OMNI performs better than LP alone. but how good is at predicting interesting tasks? To address this, we created an oracle for the Mol, termed the Oracle Model of Interestingness (OMol). Impressively, the performance of the FM- Mol is nearly on par with the oracle, suggesting that OMNI is highly effective in identifying interesting tasks for the agent to learn on (Appendix PP).
EXPERIMENTS IN AN INFINITE TASK SPACE
In truly open-ended settings, there are an infinite number of possible tasks. This section demonstrates OMNI in such setting. Essential to training an agent capable of handling any task in such an open- ended lcarning framework is the development of a universal reward function, which can evaluate if any task has been completed or not. This section proposes an instantiation of OMNI that solves that problem by not only having FMs propose new, interesting tasks, but also by having the FM generate the code for a reward function that determines to what extent each proposed task has been performed.
5.1 METHODS
In an infinite task space, it is impossible to evaluate every possible task to determine the agent's s learning progress. Hence, instead of using a predefined set of tasks, use pretrained autoregressive FM, GPT-4 2023). to generate learnable and interesting tasks throughout training. The LP curriculum then produces task sampling rates over this growing task set (Section 3.2). We input tasks
7

---

Published as a conference paper at ICLR 2024
that the agent can do well and tasks that the agent cannot do yet, then prompt GPT-4 in zero-shot manner to suggest the next learnable and interesting tasks. Tasks done well are those completed with success rates greater than a predefined threshold (0.6 in AI2-THOR experiments). We also ask GPT-4 to output sequence of environment states (in code format) that can be used to check whether or not the task has been successfully completed during training and evaluation. Appendix [D shows the full prompt and Appendix
[ shows an example output.
There are existing approaches that use FMs to generate code reward functions (Kwon et al. 2023 Wang et al. 2023a Yu et al., 2023). This version of OMNI integrates the generation of the task and the code requirements for task completion into a single output. This integrated approach ensures that every generated task comes with a comprehensive definition of what constitutes its completion (in code format). This approach can work for any domain in which one can run code to make queries about the underlying state. We apply OMNI to complex, embodied robotics kitchen domain, A12-THOR (Kolve et al., 2017), and show that OMNI is not only able to continuously generate learnable and interesting tasks, but also learns more tasks over time than controls.
5.2 A12-THOR ENVIRONMENT
Performance on UHOR domain
Slice the apple with the knife
OMNI linteresting added)
Fill the cup with water from the faucet
Egocentric View
LP (random tasks added)
Turn on the coffee machine
Unifarm
See and up wine bottle
See a pan
23-
See pick up sponge
No. of Taske with
Pick fork
Open a microwave
Pick egg
Bird's Eye View
Open drawer
training steps
Led
See a coffee machine
See and pick up spoon
Go an apple
Identify a visible knife
See and open cabinet
up visible knife
See and pick up a potato
Figure 3: A12-THOR environment and results. (Left) Agent's 's egocentric view and bird's-eye view in an A12-THOR kitchen environment. (Right) OMNI learns more tasks than the Learning Progress and Uniform sampling baselines. Example tasks learned by OMNI are shown in gray boxes.
2017 is an embodied 3D domain characterized by its near photo-realistic A12-THOR (Kolve et al.
environment (Figure 3). We train our methods on an A12-THOR kitchen floorplan. The environ- ment contains many objects commonly found in a real kitchen, such as food (e.g., apple, bread), appliances (e.g., coffee machine, microwave), and tools (e.g.. mug, pan). The agent has 13 discrete actions: MoveAhead, RotateRi ght, Rotateleft, LookUp, LookDown, Pickup, Put, Open, Close, Toggle0n Toggl eOff, 51 ice and Wiithhittiqqi We simplify the action mechanics that require a target object as an argument (c.g., the Pickup action, which re- quires a target object like Cup). Rather than force the agent to specify one of an infinite number of possible objects, instead, if the object mentioned in the current task is visible and requires the action be applied to it to complete the task, it is automatically designated as the target object. If not, the target defaults to the visible object nearest to the agent. The agent's 's observation includes 300 x 300 RGB pixel observations of a fiefield of view, and description of the task in natural language (embedded with look-up table and GRU, Appendix No. The agent receives a +1 reward, with a small penalty of 0.001 for each time step. when it has successfully completed the given task. A task can be described in natural language or by sequence of environment states. For the agent to complete a given task, it needs to sequentially achieve a list of environment states (specified in code). For example, if the task is "Pick up an apple, then put it down", the correspond- ing code format could be [ [[objaattrbuut (" ("Apple" ",, "i "isPickedUpp achieve the first environment state where the apple is picked up. then achieve the second environment state where the apple is not picked up. The task space is infinite, as there is restriction the number of attributes to check for in each environment state, or the length of environment states to be achieved sequentially when specifying each task.
( ("Apple", " "isPickedUp" False) ]I, whereby the agent has to
True) 1,
bebb_attribute
8

---

Published as a conference paper at ICLR 2024
The complexity and variability of tasks and interactions in A12-THOR are significant, yet represent only a fraction of the possibilities of a Darwin Complete environment generator, meaning one that can create any possible learning environment Clune (2019). By demonstrating OMNI in this infinite A12-THOR task space, we mark a step towards that ultimate, lofty goal of generating learnable and interesting tasks in search space that includes any conceivable environment.
5.3 RESULTS
A12-THOR RL agents are trained with PPO (Schulman et al. 2017 ) a standard RL algorithm. Policy details and hyperparameters are in Appendix N We compare the performance of agents trained with: (1) Uniform sampling (Appendix J.I (2) LP, the Learning Progress curriculum over growing task where random tasks are added (Appendix J.2). and (3) OMNI, which is the Learning Progress curriculum applied over growing task set where interesting and learnable tasks suggested by the FM are added (Section[5.]) Uniform sampling, the control, uniformly samples any task within the task space. Uniform sampling is naive and samples tasks that are too difficult for the agent most of the time, hurting learning (before even factoring in whether the tasks are worth learning). LPP samples tasks based on the calculated learning progress weights (Section (32). but most tasks added to the task set are too difficult. OMNI automatically generates learnable and interesting tasks for the agent to learn on. All experiments are run for I million time steps and are repeated 10 times with different random seeds. Each experiment takes 24 hrs on a 24GB NVIDIA A10 GPU with 30 virtual CPUs.
In this vast landscape of infinite potential tasks, it is impossible evaluate every conceivable task. Hence, each method is only evaluated on tasks that have ever been sampled before. We measure our methods by the number of tasks completed at success rate greater than predetermined threshold (here, 0.6). All confidence intervals are 95% median bootstrap confidence intervals obtained by resampling 1000 times. Confidence intervals lower- - upper) where stat is the median across runs. Shaded areas in graphs also indicate the 95% median bootstrap confidence interval obtained by resampling 1000 times.
reported with the following notation: stat (CI:
Uniform Sampling, As expected, the results with uniform sampling are poor. The agent trained with Uniform sampling learns (CI: 0 2) tasks (defined here and other treatments conditional success probability of at least 0.6).
Learning Progress Baseline. Although the LP curriculum allows the agent focus on the learnable tasks within the task set, because the tasks added to the task set are often too difficult, the agent does not learn many tasks either. The agent trained with LP learns (CI: 0 3) tasks.
OMNI. To automatically generate and learn interesting tasks, an FM is prompted in zero-shot manner to suggest the next new learnable and interesting tasks, augmenting the task set for the agent to train on. The agent trained with OMNI learns 13 (CI: 11 - 17) tasks. The difference in performance between OMNI and both baselines (Uniform sampling and I.P baseline) at 25% 50%, 75% and 100% of the way through training statistically significant (all p Ie-3, Mann Whitney U test), showing that OMNI significantly outperforms both Uniform sampling and the LP baseline (Figure 3).
DISCUSSION, FUTURE WORK, AND CONCLUSION
In conclusion, our work demonstrates the potential of using an Mol significantly enhance auto- curricula and the quest for open-ended learning algorithms by intelligently focusing on learnable and interesting tasks. OMNI addresses the Achilles Heel of open-ended systems, which lies in defining and quantifying interestingness, as previous attempts have resulted in pathologies when optimizing against such definitions and quantifications. OMNI mitigates this problem by leveraging human notions of interestingness to guide AI systems. There are numerous ways to implement the principles of this new paradigm, and exploring different versions presents an exciting avenue for future research (Appendix [.. The generality and applicability of OMNI to other open-ended domains with vast task spaces further underscores its significance. In the long run, hints at a synergy between FMs and open-endedness that simultaneously addresses looming challenges for both: how will FMs ultimately rise to the level of creativity seen in the best of human innovation, and how will open- open-endedness overcome the trap of diverging into vast space of uninspiring mediocrity? By playing off each other's strengths, FMS can perhaps someday become essential engines of open-ended discovery and begin participate in the creative dance that has defined civilization since its inception.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program, grant from Schmidt Futures, an NSERC Discovery Grant, and generous donation from Rafael Cosman. We also thank Andrew Dai, Cédric Colas, and members in our lab the University of British Columbia, namely Aaron Dharna, Ben Norman, and Shengran Hu, for insightful discussions and feedback.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint 270112200.01661, 2022.
1lge Akkaya, Marein Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint urXiv:1910.071 13, 2019.
Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. OPTICS: Ordering points to identify the clustering structure. ACM Sigmod record, 28(2):49 60, 1999.
Arthur Aubret, Lactitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. arXiv preprint arXiv: 1908.069766 2019.
Joshua Auerbach and Josh C Bongard. Evolving CPPNs to grow three-dimensional physical structures. In Proceedings of the 12th annual conference Genetic and evolutionary computation, pp. 627-634, 2010.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from Al feedback. arXiv preprint (rXiv:2212.08077) 2022.
Adrien Baranes and Pierre-Y Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 4 41-48, 2009.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0723 2021.
Nick Bostrom. Existential risks: Analyzing human extinction scenarios and related hazards. Journal of Evolution and technology. 9, 2002.
Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-Diverstty through AI Feedback. arXiv preprint arXiv:2310.1303 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,20 2020.
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 13734-13744, 2023.
Andres Campero. Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. /2/22, 2020.
10

---

Published as a conference paper at ICLR 2024
Maxime Chevalier-Boisee Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272. 082700000.22... 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jeff Clune. AI-GAs:
Al-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv: 10985, 2019.
Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre- Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: short survey. Journal of Artificial Intelligence Research, 74:1 1111119999999 2022.
Cédric Colas, Laetitia Teodorescu, Fies Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté. Augmenting Autotelic Agents with Large Language Models. arXiv preprint arXiv:2305. 12487, 2023.
Andrew Critch and David Krueger. Al research considerations for human existential safety (ARCHES). arXiv preprint arXiv:2006.04948 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:1 13309-130611 2020.
Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (GRU) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp. 1597-1600. IEEE, 2017.
Aaron Dharna, Amy K Hoover, Julian Togelius, and Lisa Soros. Transfer dynamics in emergent evolutionary curricula. IEEE Transactions on Games, 2022.
Li Ding, Jenny Zhang. Jeff Clune, Lee Spector, and Joel Lehman. Quality Diversity through Human Feedback. arXiv preprint wrXii:2310.121100 2023.
Danny Driess, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
Yuging Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding Pretraining in Reinforcement Learning with Large Language Models. arXiv preprint 230.022202. 2023.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv: /901. 10995, 2019.
Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity. In ALIFE 2020: The 2020 Conference Artificial Life, pp. 27-35. MIT Press, 2020.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-58, 2021.
Mayalen Etcheverry. Clément Moulin-Frier and Pierre- Yves Oudeyer. Hierarchically organized latent modules for exploratory search in morphogenetic systems. Advances in Neural Information Processing Systems, 33:4846-4852 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint
2018.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International conference on machine learning, pp. 1515-1528. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2024
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukeuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Izzeddin Gur, Natasha Jaques, Kevin Malta, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Ad- versarial environment generation for learning to navigate the web. arXiv preprint arXiv:2103.0199 2021.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.0670 2021.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pp. 9118-9147. PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 2022b.
David Yu-Tung Hui, Maxime Shevaller-Boisvert Dzmitry Bahdanau, and Yoshua Bengio, BabyAI 1.1. arXiv preprint arXiv:2007. 12770, 2020.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950. PMLR, 2021.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)999-134, 1998.
Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task cur- riculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106. 14876, 2021.
Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal. Housekeep: Tidying virtual households using commonsense reasoning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIX, pp. 355-373. Springer, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08830 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2, 2019.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, al. A12-thor: An interactive 3d environment for visual AI. arXiv preprint 17200:1712 2017.
Grgur Kovač, Adrien Laversanne-Finot and Pierre-Y Oudeyer. Grimgep: learning progress for robust goal sampling in visual deep reinforcement learning. IEEE Transactions on Cognitive and Developmental Systems, 99: 19: 2022.
Minae Kwon. Sang Michael Xic, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint ar arXiv:2303.0000 2023.
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-22) 201 la.
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 21 1-218, 20.
12

---

Published as a conference paper at ICLR 2024
Joel Lehman and Kenneth O Stanley. Beyond :een-endedness: Quantifying impressiveness. In ALIFE 2012: The Thirteenth International Conference on the Synthesis and Simulation of Living Systems, pp. 75-82. MIT Press, 2012.
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):274-306, 2020.
Xiang Lorraine Li, Adhigunaa Kuncoro, Jordan Hoffmann, Cyprien de Masson Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11838 11855, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020.
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher student curriculum learning. IEEE transactions neural networks and learning systems, 31()):3732-3740, 2019.
Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Alan: Autonomously exploring robotic agents in the real world. arXiv preprint arXiv:2302.0660 06604, 2023.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: survey. arXiv preprint 1rXiv:2111.01245 2021.
lean-Baptiste Mouret. Novelty based multiobjeettviization. In New Horizons in Evolutionary Robotics:
Extended Contributions from the 2009 EvoDeRob Workshop, pp. 139-154. Springer, 2011.
rean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping clites. arXiv preprint arXiv: 1504.v/504.04900 04909, 2015.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. Proceedings of the 2015 anmual conference genetic and evolutionary computation, pp. 959-966, 2015.
OpenAl. GPT-4 Technical Report. ArXiv,
2023.
OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique P Pinto, et Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2 arXiv:2/01.04882 2021.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning, Advances Neural Information Processing Systems, 31, 2018.
Pierre-Yvel Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2024
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. OpenAI, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:/908. 10084, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:150.00244 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1eXiv:7777.0347, 2017.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517, 2021.
Kenneth O. Stanley and Joel Lehman. Why Greatness Cannot Be Planned: The Myth of the Objective. Springer Publishing Company, Incorporated, 2015. ISBN 3319155237.
Kenneth O. Stanley, Joel Lehman, and Lisa Soros. Open-endedness:
The Last Grand
Challenge You've Never Heard Of. https://ww.orellyy wwwpllyyyycorraaaraaaaaa radar/ brennendeddees- the- last-lsttteee ceret-granddchhe wtw.ehhhwh..eeeeeeeeeeeeeeeeeeee
May 2023. Accessed: 2023-05-15.
Potter Stewart. Jacobellis Ohio, 378 U.S. 184. United States Supreme Court, 1964.
Marilyn Strathern. 'Improving ratings': audit in the British University system. European review, 5 (3):305-321, 1997.
Daniel Chee Hian Tan, Jenny Zhang, Zhibin Li, et al. Perceptive Locomotion with Controllable Pace and Natural Gait Transitions Over Uneven Terrains. arXiv preprint arXiv:2301.10844 2023.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg Michael Chang. Natalie Clay, Adrian Collister, al. Human-Timescale Adaptation in an Open-Ended Task Space. arXiv preprint arXiv:2301.07608 2023.
Alexey Turchin and David Denkenberger. Classification of global catastrophic risks connected with artificial intelligence. Al & Society, 35(1):147-163, 2020.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023a.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 001553, 2019.
Rui Wang. Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pp. 9940-9951. PMLR, 2020.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables world multi-task agents. arXiv preprint arXiv:2302.01566 2023b.
14

---

Published as a conference paper at ICLR 2024
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. thoouht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint MrXiv:2303.04129. 2023.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to Rewards for Robotic Skill Synthesis. arXiv preprint urXiv:2306.086477 2023.
Haoqi Yuan, Chi Zhang. Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7653 2020.
Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from Visual Observation via Offline Pretrained State-to-Go Transformer. arXiv preprint arXiv:2306.12866 12860, 2023.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory. arXiv preprint 1rXiv:2305.17144 2023.
15

---

Published as a conference paper at ICLR 2024
A LEARNING PROGRESS CURRICULUM DETAILS
The reweighting mechanism magnifies differences in low probabilities, putting additional focus on tasks that have high learning progress and low success rates. However, it causes bias against tasks with imitially high success rates achieved by chance even though the agent has not yet learned them. We do not remove the reweighting mechanism since it remains useful in the later stages of training. Instead, we propose an extension to approach from Kanitscheider et al. (2021), normalizing the task success rates with the success rates achieved by random action policy. Normalizing the success rates reduces the bias against tasks with initially high success rates, increasing their sample frequency during the early stages of the learning progress curriculum.
For each task, we calculate the task success rate (dnn achieved by a random action policy. At fixed evaluation intervals during training, the evaluated task success rate tevll of the RL policy is normalized as such:
Levol
Lydn.
norm
tydn
The above is contributed extension the learning progress method in Kanitscheider et al. 2021 norm is smoothed with an EMA function to obtain Precent (Figure 4] green). Precent is smoothed with second identical EMA function obtain Paradual (Figure 4 brown). The exponential smoothing constant applied in all experiments is 0.1. The bidirectional learning progress measure is given by LP [Preeen Praadulll) (Figure blue), where f is the reweighting function:
(1 po)p
f(p)
p + po 2p)
with parameter Po = 0.1.
We employ a sampling function transform the measure of learning progress into task sampling weights, focusing mostly on tasks with the largest learning progress. The steps are as follows:
Z-score the reweighted learning progress (subtract mean and divide by standard deviation).
Apply sigmoid the result.
Normalize resulting weights sampling probabilities.
Measured probability
Hand
Precent
Parasval
Learning progress
Reweighted learning progress
Probabiitty
Time
Figure 4: The process of determining an agent's learning progress on task from its measured success probability on that task in an example fictional problem.
16

---

Published as a conference paper at ICLR 2024
A.1 LEARNING PROGRESS CURRICULUM ABLATION
The learning progress curriculum without the proposed extension (Section 3.2) of normalizing the task success rates random action baseline labelled LP-no-norm. light of limited compute, we limit ourselves to the Crafter domain and experiments are run for 30 million time steps (vs. 100 million in the main experiments) and are repeated 10 times (same as the main experiments) with different random seeds. The LP curriculum achieves higher average task success rates and learns more tasks with the proposed normalization extension (Figure 5). Across all metrics and in both domains, the differences in performance between LP and LP-no-norm at 50%, 75%, and 100% of the way through training are statistically significant (all < 0.05, Mann Whitney U test).
Average Task
of Tasks with Success
0.35
0.10
LP
LP-no-norm
2.8
05
15
training steps
training steps
Figure Performance of learning progress curriculum in Crafter with and without the proposed normalization extension. Average task success rates and number of tasks with success rates more than 0.2 for learning progress curriculum with and without the proposed normalization extension across training steps. Metrics are calculated on all tasks. LP performs better with the proposed normalization extension.
B CRAFTER PROMPT
We give GPT-3 three examples as part of the prompt:
You are a player in a game. You want to learn as many different skills as possible. You can do this task well: collect wood. Suggest whether the given tasks are interesting: collect drink, collect wood, make stone sword, make wood pickaxe, place furnace. collect drink: True
collect 2 drink: True
collect 3 drink: True
collect wood: False
collect 2 wood: False
collect 3 wood: False
make stone sword: True
make 2 stone sword: True
make 3 stone sword: True
make wood pickaxe: True
make 2 wood pickaxe: True
make 3 wood pickaxe: True
place furnace: True
place 2 furnace: True
place 3 furnace: True
You are a player in a game. You want to learn as many different
skills as possible. You do this task well: make iron pickaxe.
Suggest whether the given tasks are interesting: collect coal,
17

---

Published as a conference paper at ICLR 2024
collect iron, make iron pickaxe, make iron sword, place table.
collect coal: True
collect 2 coal: True
collect 3 coal: True
collect iron: True
collect 2 iron: True
collect 3 iron: True
make iron pickaxe: False
make 2 iron pickaxe: False
make 3 iron pickaxe: False
make iron sword: True
make 2 iron sword: True
make 3 iron sword: True
place table: True
place 2 table: True
place 3 table: True
You are player in a game. You want to learn as many different skills as possible. You can do this task well: place 3 stone. Suggest whether the given tasks are interesting: collect diamond, collect stone, make stone pickaxe, make wood sword, place stone.
collect diamond: True
collect 2 diamond: True
collect 3 diamond: True
collect stone: True
collect 2 stone: True
collect 3 stone: True
make stone pickaxe: True
make 2 stone pickaxe: True
make 3 stone pickaxe: True
make wood sword: True
make 2 wood sword: True
make 3 wood sword: True
place stone: False
place 2 stone: False
place 3 stone: False
Given the of tasks that the agent can do relatively well and the set of tasks that needs to be determined as interesting or not, the additional prompt is:
You are a player in a game. You want to learn as many different skills as possible. You can do these tasks well: <tasks done well>.
Suggest whether the given tasks are interesting: <tasks to be determined>.
In the shot examples, do not include all possible tasks in <tasks to be determined> to reduce token usage. While each example sets different tasks for <tasks to be determinede ned>, during inference, all tasks needing classification interesting or not (Section 3.3 are inputted as <tasks to be determined> without any additional filtering.
GPT-3 predicts whether each task in <tasks True/ False, following the format in the few-shot examples. In our experience, GPT-3 nearly always conforms to the requested output format. However, the rare cases that GPT-3 deviates from the expected output format, the responses are regenerated. For tasks where GPT-3 did not provide an be determined> input to include only the tasks lacking
be determi ned> >> is interesting
with
answer, we modify the <tasks responses, and then regenerate these responses.
We access GPT-3 through OpenAl's APIs, opting for the Davinci model in our experiments, which costs $0.02 per 1000 tokens when these experiments were run. Caching significantly reduces the
18

---

Published as a conference paper at ICLR 2024
number of API queries. We extensively cache GPT-3 prompts and responses and consistently reuse this cache across multiple runs.
C BABYAI PROMPT
Full system prompt GPT-4:
You are a helpful assistant that tells an AI agent the next tasks to do in this 2D grid environmente The ultimate goal that it would like your help with is to learn as many interestingly different skills as possible, meaning wide diversity of different skills that would help it be ready to solve new skills someone might ask it to perform, or to transfer what it has learned in this environment to other environments.
The grid world has objects in six distinct colors - "red", "green", "blue", "purple", "yellow", and "grey" and four types - "key", "ball", "box", and "door". Each task is sequence of instructions, connected using "then", specifying the order of instructions to follow. Instructions include interactions like going to objects, picking up items, opening doors (which requires the appropriately colored key if the door is locked), and putting objects next to another. Object placements are randomized.
I will give you the following information: Tasks the agent currently well: Predict which of these tasks are interesting:
You must follow the following criteria:
1) You should act as a mentor and guide the AI agent to the next most interesting tasks.
2) Interesting tasks are roughly those that are sufficiently
different from the ones that it can already do, and should be novel, diverse and at least worth learning.
You should only respond in the format as described below: RESPONSE FORMAT:
Reasoning: Based on the information given, do reasoning about why each task is interesting or not.
Predictions: The list of predictions. For each line, put the task, then a colon, then True for interesting or False for boring.
Here are some example responses:
Tasks the agent currently do well: "open some door", "go to some object"
Predict which of these tasks are interesting: "pick up object", "put some object next to some other object", "go to some object, then put some object next to some other object", "go to some object, then go to some object", "go to some object, then open some door", "open sone door, then go some object"
Reasoning: Tasks that are recombinations of "go to some object"
or "open some door" are not interesting. The tasks that introduce something different from the tasks done well are "pick up some object", "put some object next to some other object", and "go to some object, then put some object next to some other object", having a new actions of picking up an object, or putting an object next to another.
Predictions:
"pick up some object": True
19

---

Published as a conference paper at ICLR 2024
"put some object next to some other object": True
"go to some object, then put some object next to some other jbjett True
"go to some object, then go to some object": False
"go to some object, then open some door": False
"open some door, then go to some object": False
Tasks the agent currently do well: "go to some object, then open some door", "go to some object", "pick up some object"
Predict which of these tasks are interesting: "pick up some object", "go to some object, then open some door, then open some door", "pick up some object, then put some object next to some other object, then put some object next to some other object", "put some object next to some other object, then go to some object, then put some object next to some other object, then open some door, then open some door"
Reasoning: Tasks that are recombinations of "go to some object, then open some door", "go some object", object" are not interesting. The tasks that introduce something different from the tasks done well are "pick up some object, then put some object next to some other object, then put some object next to some other object" and "put some object next to some other object, then go to some object, then put some object next to some other object, then open some door, then open some door", having a new action of putting some object next to another. Predictions:
"pick up some
"pick up some object": False
"go to some object, then open some door, then open some door": False
"pick up some object, then put some object next to some other object, then put some object next to some other object": True "put some object next to some other object, then go to some object, then put some object next to some other object, then open some door, then open some door": True
Tasks the agent currently do well: "go to some object", "gut some object next to some other object", "pick up some object" Predict which of these tasks are interesting: "pick up object, then go to some object", "go to some object, then open some door, then open some door", "pick up some object, then put some object next to some other object, then put some object next to some other object", "open some door, then put some object next to some other object, then go to some object, then put some object next to sone other object, then open some door", "go to some object, then pick up some object, then pick up some object, then put some object next to some other object, then go to some object" Reasoning: Tasks that are recombinations of "go to some object", "put some object next to some other object", or "pick up some object" are not interesting. The only tasks that introduce something different from the tasks done well are "go to some object, then open some door, then open some door" and "open some door, then put some object next to some other object, then go to some object, then put some object next to some other object, then open sone door", having a new action of opening a door. Predictions:
"pick up some object, then to some object": False "go to some object, then open some door, then open some door": True
"pick up some object, then put some object next to some other
20

---

Published as a conference paper at ICLR 2024
ob ject, then put some object next to some other object": False "open some door, then put some object next to some other object, then go to some object, then put some object next to some other object, then open some door": True "go to some object, then pick up some object, then pick up some object, then put some object next to some other object, then go to some object": False
We access GPT-4 through OpenAl's APIs, which costs $0.03 per 1000 tokens when these experiments were run. Caching significantly reduces the number of API queries. We extensively cache GPT-4 prompts and responses and consistently reuse this cache across multiple runs.
D A12-THOR PROMPT GENERATE TASKS
No in-prompt examples were provided because our testings found that they were not needed. Full system prompt to GPT-4:
You are helpful assistant that tells AI agent the next tasks to do in an embodied kitchen environment. The ultimate goal that it would like your help with is to learn many interestingly different tasks as possible.
The agent has 13 discrete actions: move ahead, rotate right, rotate left, look up, look down, pick up object, put object, open object, close object, toggle object on, toggle object off, slice object, and fill object with liquid. The agent is in kitchen with fixed object placements and configuration. The only objects the kitchen are: "Apple", "Bowl", "Bread", "ButterKnife", "Cabinet", "CoffeeMachine", "CounterTop", "Cup", "DishSponge", "Drawer", "Egg", "Faucet", "Floor", "Fork", "Fridge", "GarbageCan",, "HousePlant", "Kettle", "Knife", "Lettuce", "LightSwiteh", "Microwave", "Mug", "Pan", "PaperToweiRell"" "PepperShaker", "Plate", "Pot", "Potato", "SaltShaker", "SideTable", "Sink", "SinkBasin", "SoapRottle", "Spatula", "Spoon", "Stool", "StoveRurner", "StoveKnob", "Toaster", "Tomato", "Window", "WineBottle". Objects are of varying distance to the agent's starting position.
task is described as sequence of environment states that need to be achieved:
[env_statel, env_state2, ...]
Each environment state is described by the object states in the room:
[obj_attributes name,, requirement_dictl),
obj_attributes (object_name2, requirement_diet),, ...]
Objects not explicitly specified in the environment state do not affect whether the task has been deemed completed or not. Each object has these attributes: "visible", "isToggled", "isBroken", "isFilleeWithiiiuiii""" "isDirty", "isCooked", "temperature", "issliced", "is0pen", "isPickedUp", "recpptaclebjjetts. All attributes are either True/False, except for "temperature" which Hot/Cold/RoomTeee R/o.m//mommm, and "receptacleob which is list of objects that the receptacle contains.
will give you the following information: Tasks the agent currently does well: Tasks the agent cannot do yet:
You must follow the following criteria:
21

---

Published as a conference paper at ICLR 2024
1. You should act as mentor and quide the AI agent to the next most learnable and interesting tasks.
2. Learnable tasks are those that are not too difficult easy for the current agent.
3. Interesting tasks
roughly those that sufficiently
different from the ones that the agent can already do, novel, diverse and least worth learning.
4. Do not suggest tasks that are already given as tasks that the agent currently does well, or tasks that the agent cannot yet.
You should only respond in the format as described below: RESPONSE FORMAT:
Reasoning: Based on the information given, do reasoning about what the next learnable and interesting tasks are.
Next tasks in natural language: Suggest learnable and interesting tasks that the agent should learn next.
Next tasks as sequence of environment states: Translate the suggested natural language tasks into the code format given above.
We access GPT-4 through OpenAl's APIs, which costs $0.03 per 1000 tokens when these experiments
were run.
E A12-THOR PROMPT EXAMPLE GENERATE TASKS
This section presents example of GPT-4 generating the next learnable and interesting tasks based on what tasks the agent can or cannot already do. If the input GPT-4, in addition the system prompt in Appendix is:
Tasks the agent currently does well:
ttriiaattribuusssApppppi "visible": True)]]
[[obj_attributee(("nnn Kniff"" "visible": True)]] [[obj_ iiiiaaatriuues""PPottt
"visible": True)]]
Tasks the agent cannot do yet:
[[obj_attributes(" Bread", """rear, "visible": True)]]
[[obj_attributees" Apple", "visible": True, "isPickedUp": True)]]
[[obj_ ("tttriibbutesFFiieeee" "visible": Tooj)).lobj_attrbbbul attribbtees"Friddd ("Frigee", "isOpen": True)]]
[[obj boobjiaattrbbutss"PPia Plate", "visible": True)]]
First, GPT-4 will output reason about what the next learnable and interesting tasks are, inspired by Wei al. (022). Here is GPT-4-produced example:
The agent currently able to identify and locate certain objects in the kitchen environ- ment, such apple, knife, and potato. However. it struggles with tasks that involve interaction with objects, such as picking up an apple or opening the fridge. also seems to have difficulty recognizing certain objects, like bread and plate. Given these abilities and limitations, the next tasks should involve simple interactions with objects that the agent can already recognize, as well as recognition of new objects that are not too similar the ones it struggles with.
Reasoning:
Then, GPT-4 will suggest 3 learnable and interesting tasks that the agent should learn next:
Next tasks in natural language:
Slice the potato with the knife.
2. Pick up the knife.
Recognize and locate the coffee machine.
And finally, GPT-4 will translate the suggested natural language tasks into a sequence environment states in code format, that can be used check whether task has been completed not.
22

---

Published as a conference paper at ICLR 2024
Next tasks as sequence of environment states:
1. atrooiaatrrbuess"PPoo ("Potatt"""""" "visible": True)], (obi_atributes"Knnife,,, "visible": True)],
attributes("PPotii "isSliced": True)]]
2. "ooi_aatribute("Kaaan "visible": True)], [obj Sirributs("Knitee" "isPickedUp": True)]]
3. [[obj__tte] "offeeMachine"" "visible": True)]]
F A12-THOR PROMPT TRANSLATE CODE NATURAL LANGUAGE
in-prompt examples were provided because our testings found that they were not needed. Full system prompt GPT-4:
You are a helpful assistant that relabels the tasks from the given code format to natural language descriptions. The tasks are set in an embodied kitchen environment.
The agent has 13 discrete actions: move ahead, rotate right, rotate left, look up, look down, pick up object, put object, open object, close object, toggle object on, toggle object off, slice object, and fill object with liquid. The agent is in kitchen with fixed object placements and configuration. The only objects the kitchen are: "Apple", "Bowl", "Bread", "ButterKnife", "Cabinet", "CoffeeMachine", "CounterTop", "Cup", "DishSponge", "Drawer", "Egg", "Faucet", "Floor", "Fork" "Fridge", "GarbageCan", "HousePlant", "Kettle", "Knife", "Lettuce", "LightSwitch", "Microwave", "Mug", "Pan", "PaperTowelRell", "PepperShaker", "Plate", "Pot", "Potato", "SaltShaker", "SideTable", "Sink", "SinkBasin", "SoapBottle", "Spatula", "Spoon", "Stool", "StoveBurner", "StoveKnob", "Toaster", "Tomato", "Window", "WineRottle". Objects are of varying distance to the agent's starting position.
task is described as sequence of environment states that need to be achieved:
(env_stated; env_state2, ...]
Each environment state is described by the object states in the room:
[obj_attributes (object_namel, reguireemet__dict)))
abj_attrisutes croject_namm222 requirement_dict2)) ...] Objects not explicitly specified in the environment state do not affect whether the task has been deemed completed or not. Each object has these attributes: "visible", "isToggled", "isBroken", "isfiileddiichhiiiiii",," "isDirty", "isCooked", "temperature"; "issliced", "is0pen", "isPickedUp", "recoptacleobjectss.. All attributes are either True/False, except for "temperature" which is http///ol///ooomeeee objects that the receptacle contains.
and "receptacle0bjecta" which is list of
will give you the following information: Tasks in code format:
You should only respond in the format as described below: RESPONSE FORMAT:
Tasks in natural language: Translate the tasks from the given code format to natural language descriptions.
23

---

Published as a conference paper at ICLR 2024
G AI2-THOR PROMPT EXAMPLE TRANNLATE CODE TO NATURAL LANGUAGE
the randomly generated tasks in code format fed into GPT-4 are:
Tasks in code format:
attributes("Side
"receptacleOhjects.:
"isPickedUp": False)]]
2. (attributes("Egg"" "isPickedUp": True, "isBroken":
"Sink",
"receptacleObjects.. "Potato" joobj attributes("Brrent "isSliced": True)]]
3. aatributes("Butteekkke
False)].[objjatte)) (aaribbttes(SideTTtiee "isPickedUp": "receptacleObjects"" "ButterKnife").ee "oofii"")oo__aatttbiia attributes( ("Fok"", "isPickedUp": fabbe))llojjattrriutees"" attributes("Pot" "receptacleObjects"..
attrributes(((( "SooaBoottle", "isPickedUp":
"Sppaula"),obi" j.oj__atrrbbutes. iatttribbutesMMicoaaaan "Microwave,, "receptacleOjeects.: "Knife")]]
Then GPT-4 will translate the tasks to natural language:
Tasks in natural language:
Put an apple on the side table and put down the butterknife.
2. Pick up and break egg. Then, a potato in the sink and slice the bread.
Pick up the butterknife and put down the soap bottle. Then, put the butterknife on the side table and put down the fork. After that, put spatula in the pot and a knife in the microwave.
H OMNI ALGORITHM
Algorithm 1 OMNI Algorithm
Define task set T, batch size B, evaluation frequency N
Define EMA constant B, learning progress reweighting function f
Initialize policy
Initialize task sampling distribution D +- Uniform(T") Initialize update counter +-
Calculate task success rates by random action policy Prdn for each task in T while u < max _on_utteeessst do Initialize rollout set R <- 0 while |R| < do
Sample task D
Sample tasks based on distribution D
Perform rollout on task using policy π
Add rollout set R
end while
Update policy using rollouts in R
D Use any RL algorithm
+ u + 1 mod N then
Praw < Evaluate policy π on all tasks in T
for T do
Pnorm (1) t- (Praw (1) Prdn (1))/(1 Prda (t))
Normalize task success rates
Precent (t) < Puorm (t) x $ Precent (t)
-3)
Paraduat (f) Precent (t) x B + Paradual (f) x (1
Pip(() (Precet (t)) - ((Prrdduat (f)))
Calculate learning progress
end for Partition T into interesting or boring using an FM based on Praw
Tint Thor
An example in Appendix
for t E T do
Pmoi (t) f- +ift E Tine then else 0.001
Reweight based on interestingness
end for
D +- Plp 2 Pmoi
b Update task sampling distribution
end if end while
24

---

Published as a conference paper at ICLR 2024
Learning Progress
Tasks success rates
Observations
Environment
Typical RL training
Task sampler
Which tasks are not too easy or difficult for the agent to learn from?
Reward
Next tasks to train on (interesting and learnable)
RL Agent
Model of Interestingness Which tasks are interesting?
Action
Figure 6: Overview of OMNI OMNI enables open-ended learning in vast environment search spaces by ensuring that tasks trained on not only have high learning progress, but are also interesting (harnessing large Al models to make such a heretofore impossible judgement).
I PARTITIONING ALL TASKS INTO INTERESTING OR BORING
The Mol partitions all tasks as either "interesting" or "boring" based on their success rates and the FM's assessment of their relation to tasks already classified as "interesting" (Algorithm 2). Since the raw evaluation of task success rates can be noisy, the task success rates referenced in this section are smoothed with exponential moving average function. Algorithm iteratively selects the task with the highest success rate not yet categorized, adds it to the "interesting" set (Step prompts the FM to identify boring tasks from the remaining tasks in relation to the "interesting" set (Step and updates the "boriig" set (Step(5) repeating until all tasks are categorized. In Step the rationale for adding the task with the highest success rate, that not yet categorized, to the "interesting" set is twofold: first, the FM considers this task to be sufficiently distinct from those already in the "interesting" set, otherwise, it would have already been considered "boring" in Step second, its relatively higher success rate indicates that it aligns closer to the agent's current skill level.
To illustrate in Crafter, assume that the "collect wood" task has the highest success rate. It is added to the "interesting" set (Step 3). "collect wood" is interestingly different from all tasks currently declared as "interesting" because there are no tasks in that set yet. This is an initialization step for the algorithm. Then the FM deems "collect 20.10 wood" tasks boring (Step 4). Now, when repeating Step 3 the algorithm will add the next task with the highest success rate and not yet categorized, e.g., "place table", to the "interesting" set, whereas without the filter in Step 4 "collect wood" might have been selected as interesting instead.
Algorithm 2 Mechanism to partition the task set into interesting and boring sets.
1: Sort the tasks based on the evaluated task success rates.
2: Create two empty sets, one track the interesting tasks and one track the boring tasks.
3: Identify the task with highest success rate and not in any of the sets. Add it to the interesting set.
4: Prompt the FM to determine if any of the remaining tasks are boring, contexted on the current set of interesting tasks. Tasks in the interesting set are input as tasks that the agent can do well, and tasks yet to be categorized are asked to be predicted as interesting not in the FM prompt.
5: Update the boring set with tasks that the FM has determined as boring.
6: Repeat steps 3 until all tasks are in either set.
J BASELINES IN AN INFINITE TASK SPACE
J.1 UNIFORM SAMPLING BASELINE
In an infinite task space, is not trivial how one can uniformly sample any task in natural language while still ensuring that the task is not totally nonsense (e.g.. "fish chicken pick up rainbow"). In the AI2-THOR domain, we sample random tasks by first generating random environment states in code format. We limit the maximum sequence of environment states for each task to be 10. Furthermore, filter out tasks with unachievable environment states. For example, if the environment state requires that obj_attrr tittess
butes ( ("Fridge", "isPickedUp": True), we check if the Fri dge
25

---

Published as a conference paper at ICLR 2024
is pi cicuuabl... Since Fr Fridge is not pickupable, resample another random environment state as part of the task. After randomly generating tasks in code format, a pretrained autoregressive FM, GPT-4, translates the tasks from code to natural language descriptions. Appendix F shows the full prompt and Appendix G shows an example output.
J.2 LEARNING PROGRESS BASELINE
For completeness, we introduce an LP baseline. Instead of adding learnable and interesting tasks suggested by the FM (as in OMNI), we add to the task set uniformly sampled tasks. These are generated in the same way as done in the uniform sampling baseline (Section The I.P curriculum then produces task sampling rates over this growing task set for the RI. agent to train on (Section 3.2) The frequency at which new tasks are added to the task set the same for OMNI and this LP baseline.
K CRAFTER BORING TASKS
The 90 boring tasks in the main Crafter experiments are:
collect N drink, where N [2, 10]
collect N wood, where N [2, 10]
collect N coal, where N ε [2, 10]
collect N stone, where N E [2, 10]
collect N iron, where N [2, 10]
collect N diamond, where N E [2, 10]
place N table, where N E []
place N furnace, where N E []
place N stone, where N [2, 5]
make N wood pickaxe, where N [2, 5]
make N wood sword, where N [2
make N stone pickaxe, where N E [2,5]
make stone sword, where N E [2, 5]
make N iron pickaxe, where N [2,
make N iron sword, where N [2,
L CRAFTER POLICY AND OPTIMIZATION DETAILS
The model architecture is similar to those in previous learning progress works (Kanitscheider et al., 202T). The RGB inputs are passed through a 2-layer convolution network with ReLU activations. The RGB convnet is followed by fully connected layer of size 256. The visual embeddings concatenated with the task encoding before being passed into an LSTM cell of size 256. The network output is given by 2-layer linear action head for the policy and 2-layer linear layer for the value function. Both have Tanh activation functions. Optimization performed with Proximal Policy Optimization (Schulman
2017) and General Advantage Estimation (Schulman et al. 2015).
Parameter
Value
Discount factor
0.99
Learning rate
Te44
PPO clip threshold
0.2
GAE lambda
0.95
Entropy coefficient
0.01
Batch size
2048
Epochs
Max episode length
1500
26

---

Published as a conference paper at ICLR 2024
M BABYAI POLICY AND OPTIMIZATION DETAILS
The model architecture is similar to those in previous works done on BabyAl (Chevalier-Boiswy al. 2018). The symbolic grid cell observations are passed through 3-layer convolution network with ReLU activations. The natural language task is encoded with GRU (Dey & Salem 2017). The grid cell embeddings and the task embeddings are jointly processed through convolutional network with two batch-normalize normalized FiLM (Perez et al. 2018) layers, then through an LSTM cell of size 128. The network output is given by a 2-layer linear action head for the policy and a 2-layer linear layer for the value function. Both have Tanh activation functions. Optimization is performed with Proximal Policy Optimization (Schulman et al. 2017) and General Advantage Estimation (Schulman et al. 2015).
Parameter
Value
Discount factor
0.99
Learning rate
le-4
PPO clip threshold
0.2
GAE lambda
0.95
Entropy coefficient
0.01
Batch size
2048
Epochs
Max episode length
depends on task difficulty
100 - 1000
N A12-THOR POLICY AND OPTIMIZATION DETAILS
The model architecture similar to that used for Baby experiments (Appendix My. The RGB pixel observations are passed through 3-layer convolution network with ReLU activations. The natural language task is encoded with GRU (Dey & Salem 2017). The grid cell embeddings and the task embeddings are jointly processed through a convolutional network with two batch-normalized FiLM (Perez al. 2018) layers, then through an LSTM cell of size 128. The network output is given by 2- 2-layer linear action head for the policy and 2-layer linear layer for the value function. Both have Tanh activation functions. Optimization is performed with Proximal Policy Optimization (Schulman
al. 2017) and General Advantage Estimation (Schulman al. 2015).
Parameter
Value
Discount factor
0.99
Learning rate
Ic-4
PPO clip threshold
0.2
GAE lambda
0.95
Entropy coefficient
0.01
Batch size
2048
Epochs
4
Max episode length
depends on task difficulty
200 x the number of environment states be achieved sequentially
O SUPPLEMENTARY METRICS AND PLOTS
The metrics and plots in this section complement the results shown in Section 4.3 Figure shows
the task sample rates all tasks in Crafter across different methods. To better see how different
methods perform on interesting tasks in Crafter, Figure 9 offers the same metrics Figures 2 aadd
but calculated on the subset of interesting tasks only. Figures 10 and TT are subsets of Figures 2 and
[ respectivelyy zoomed into the set of interesting tasks only. Figure 7 and [4]urowwwssow the task
success rates and task sample rates of all tasks in BabyAI across different methods. Figure 12 shows the average task success rates achieved in BabyAl on tasks with instruction lengths or less,
27

---

Published as a conference paper at ICLR 2024
small subset of the entire task space. This subset contains the type of tasks more likely to be learned within our allowable compute budget. To the best of our knowledge, BabyAl papers that learn tasks of more complexity require expert demonstration to do so, learning via behavioral cloning rather than via RI. (Chevalier-Boiiss Boisverr et al. 2018 Hui et al. 2020).
Uniform
OMNE Mol
No. of Tooko with Sucoess Bancs 0.2
Unitorm
(miliaa))
Figure 7: Results in BabyAL (Left) Conditional success probabilities subset of tasks in BabyAL. These plots only show tasks with a success rate of at least 0.05 by any method at any timestep. Tasks are organized from simple to complex based on the instruction length. (Right) Performance in BabyAl on all tasks. The average task success rate scale for BabyAl is low because is averaged over the entire task set, which includes many tasks that are difficult learn. This captures in microcosm the real world, where there be infinitely many difficult even impossible tasks. OMNI achieves much higher average task success rates and learns more tasks than uniform sampling or choosing tasks based learning progress alone.
Uniform
LP
OMNI: LP Mol
10-2
10"4
25
50
75
1000
25
50
75
1000
25
50
75
100
training steps (million)
Figure 8: Sampling probabilities for all tasks in Crafter: Tasks are ordered as in Figure 2 Uniform sampling per task barely visible the heatmap. LP accurately tracks and samples tasks whose success probabilities change the most. OMNI samples tasks with high learning progress, but narrows its focus to only the interesting ones within that set.
28

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
No. of Tasks with Success Rates 0.2
0.8
0.6
OMNI: Mol
0.4-
Uniform
0.2-
0.1
0.6
24
training steps
100
training stops
Figure 9: Performance in Crafter on interesting tasks only. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. OMNI achieves much higher average task success rates and learns more interesting tasks than uniform sampling choosing tasks based on learning progress alone.
Uniform
LP
OMNI: LP + Mol
collect wood
pioce table
collect cool
10
collctt stone
alace fumace
place stone
iran
collect diamond
25
50
75
1000
25
50
75
100
25
50
100
training steps (million)
Figure 10: Conditional success probabilities of interesting tasks only in Crafter. This figure is OMNI achieves higher task success rates on interesting tasks than
zoomed in subset of Figure 2 uniform sampling or choosing tasks based on learning progress alone.
Uniform
OMNI: LP
127
prace tasle
make wood pickose
collect stone
make stone pielace
maie stane
plece
collect ron
pickaee
training steps (million)
Figure 11: Sampling probabilities for interesting tasks only Crafter. This figure is zoomed in subset of Figure OMNI exhibits more intense sampling of interesting tasks than uniform sampling or choosing tasks based on learning progress alone.
29

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
OMNI: LP Mol
LP
0.20
Uniform
6.00
0.08
2.00
10B
training steps
Figure 12: Performance in BabyAl on tasks with instruction lengths two or less. OMNI achieves much higher average task success rates, close to 16,, and learns more tasks than LP or Uniform sampling. See text above in this section for why this is a useful subset to plot, For performance on the entire (much harder) task set, see Figure] 7
Uniform
LP
OMNI: LP + Mol
Ball salaedi Listtaciio
045
032
0.30
605
0.00
25
50
75
1000
25
50
75
2000
25
50
75
100
training steps (million)
Figure 13: Conditional success probabilities of all tasks in BabyAL Tasks are organized from simple to complex based on the instruction length. Most tasks are very difficult for the agent to learn on, hence having low success rates. This mirrors the challenges in the real world where there can countless difficult or infeasible tasks. Nevertheless, OMNI achieves higher task success rates than uniform sampling or choosing tasks based on learning progress alone.
30

---

Published as conference paper at ICLR 2024
Uniform
LP
OMNI: Mol
lao with Jestucterr
Taie dentteeer
Thi with Shaturdera
100
training steps (million)
Figure 14: Sampling probabilities for all tasks in BabyAL. Tasks are ordered as in Figure 7 Uniform sampling per task is barely visible the heatmap. LP accurately tracks and samples tasks whose success probabilities change the most. OMNI samples tasks with high learning progress, but narrows its focus only the interesting ones within that set.
P ORACLE MODEL OF INTERESTINGNESS
The Oracle Model Interestingness (OMol) hand-designed model that assigns sampling weights of 1.0 to interesting tasks and 0.001 to boring tasks. In Crafter, interesting tasks considered to be the set of 15 tasks shown in Figure and all other tasks are considered boring. BabyAL, interesting tasks those with single instructions. Single-instruction tasks the foundational building blocks that should be learned first, so we consider those interesting in the OMol. In theory, a more skilled agent would eventually move beyond them two or more instruction tasks, but in our experiments, our agents do not even learn this first set (despite substantial compute budgets; Baby AI is a very does take the shortcut of providing solution demonstrations and
hard RL task environment doing imitation learning (Chevalier-Boisse for them be focusing on. The OMol functions as an oracle that accurately discerns which tasks humans would typically find interesting for the agent to learn the most skills in this environment. Although we were able to design an oracle Mol in these domains, constructing such models in more complex domains will not always be feasible, nor would scale well given the human labor required.
et al. 2018), ), meaning that these tasks are uninteresting
Across all metrics and in both domains, most the differences in performance between OMNI and the oracle at 25%, 50%, 75%, and 100% of the way through training are not statistically significant (all but one p > 0.05, Mann Whitney test). OMNI's performance comparable to that of the oracle (Figures 15 and 17). OMNI and the oracle achieve similar task success rates for each task and induce similar patterns in task sample rates (Figures 16 and (8). This suggests that FMs can capture key aspects of what humans typically find interesting.
31

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
of Tasks with Success Rates
Interesting and Borinn tasks
29-
Oracle: LP OMol
303
OMNI: LP Mol
LP
Uniform
Interesting tasks only
*4
62
training steps
training steps
Figure 15: Performance in Crafter all tasks. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. This figure is the same as Figures 7 with additional oracle method added for comparison. OMNI achieves comparable performance to the oracle, with no statistically significant difference.
Tark suscess rates
Tack samelins rates
+
100
75
100
59
300
training steps (million)
training steps (million)
Figure 16: (Left) Conditional success probabilities and (Right) Sampling probabilities of all tasks for OMNI and the oracle in Crafter. Tasks are ordered as in Figure 2 interesting tasks with high learning progress and achieves comparable task success rates to the oracle.
OMNI focuses on
32

---

Published as a conference paper at ICLR 2024
Average Task Success Ratos
No. of Tasks with Success Rates 0.2
0.030
Oracle: LP + OMol
0.055
Mol
0.000
0.255
Uniform
0.153
00881
training steps
100
training steps
Figure 17: Performance in Baby A on all tasks. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. This figure is the same as TiguresDandd / with an additional oracle method added for comparison. OMNI achieves comparable performance to the oracle, with almost statistically significant difference.
Task suscsuccssssate rates
Dask sameling.attte
OMNI LP Mcl
Cracle: LP OMol
OMN: LP Mol
Oracle: LP OMal
training
(million)
training (million)
Figure 18: (Left) Conditional success probabilities and (Right) Sampling probabilities of all tasks for OMNI and the oracle in BI Tasks are ordered as in Figure 7 OMNI focuses on interesting tasks with high learning progress and achieves comparable task success rates to the oracle.
33

---

Published as a conference paper at ICLR 2024
Q
USING COMPOUNDS AS BORING TASKS IN CRAFTER
Average Task Success Rates
No. of Tasks with Success Rates 0.2
Interesting and Boring tasks
0.00
1974
33-
Oracle: LP OMol
1.
05
10
15
15
16
25
OMNI: LP + Mol
LP
Uniform
Interesting tasks only
244
22-
05
05 20
25
25.
0.
05
20
training steps
training steps
Figure 19: Performance in Crafter on all tasks, including compound tasks. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. Compound and repetitive tasks are those considered boring. OMNI achieves much higher average task success rates and learns more tasks than LP or Uniform, close to that of the oracle.
In this Crafter setup, compound and repetitive tasks are those considered boring. In addition the repetitive boring tasks (Appendix K). compound tasks are generated by combining any two of the 15 interesting tasks (Figure T. Hence, there a total 1023 extremely challenging tasks. In light of limited compute, experiments in this setting are for 30 million time steps (vs. 100 million the main experiments) and are repeated 10 times (same as the main experiments) with different random seeds. We compare the performance of agents trained with: (1) Uniform sampling, (2) Learning Progress (LP) only, (3) OMNI: Learning Progress with additional filtering by a Model of Interestingness (OMNI: LP + Mol), and (4) the oracle: Learning Progress with additional filtering by an Oracle Model of Interestingness (Oracle: LP + OMol). The high-level summary of the results from these experiments is that they qualitatively similar to when boring tasks are repetitive tasks only (Section 4.3
interesting tasks, 195 boring tasks, and
The agent learns (CI: 3 5-66 6) tasks (interesting or boring) and 3 (CI: 2 3) interesting tasks, achieving an average task success rate of 0.019 (CI: 0.017 0.021) on interesting and boring tasks, and 0.097 (CI: 0.082 0.105) on interesting tasks only.
Uniform sampling. The results with uniform sampling are poor (Figure 19
The agent learns 67 (CI: 65 69) tasks (interesting or boring) and 9 (CI: 8 - 9) interesting tasks, achieving an average task success rate of 0. (CI: 0.200 on interesting and boring tasks, and 0.40 (CI: 0.38 0.42) on interesting tasks only. Across all metrics, the differences between LP and Uniform at 25%, 50%, 75%, and 100% of the way through training are statistically significant (all < 1e-3, Mann Whitney test), showing that LP significantly outperforms uniform sampling (Figure 19 LP accurately tracks and samples tasks with high learning progress but is distracted by the many boring tasks (Figure (1).
Learning Progress Curriculum.
The agent learns 96 (CI: 93 -3-99) tasks (interesting or boring) and 11 (CI: 11 11) interesting tasks, achieving an average task success rate of 0.29 (CI: 0.28 0.30) on interesting and boring tasks, and 0.59 (CI: 0.56 0.60) on interesting
OMNI:
Learning Progress a Model of Interestingness.
34

---

Published as a conference paper at ICLR 2024
tasks only. Across all metrics, the differences between OMNI and LP 25%, 50%, 75%, and 100% of the way through training are statistically significant (all p < 1e-3, Mann Whitney test), showing that OMNI significantly outperforms LP (Figure 19). Since OMNI focuses interesting tasks with high learning progress (Figure (1 the agent achieves higher average task success rates and learns more challenging tasks faster (Figure 20).
Oracle: Learning Progress an Oracle Model of Interestingness. The agent learns 101 (CI: 97 105) tasks (interesting boring) and 11 (CI: 11 11) interesting tasks, achieving an average task success rate of 0.30 (CI: 0.29 - 0.31) on interesting and boring tasks, and 0.62 (CI: 0.61 - 0.63) on interesting tasks only. The differences between the oracle and OMNI at 25%, 50%, 75%, and 100% of the way through training are sometimes statistically significant (some < 0.05, Mann Whitney U test), showing that OMNI does not always achieve comparable performance to the oracle (Figure 19p. However, OMNI emulates similar task success rates (Figure 20) and task sample rates as the oracle (Figure 2T). Having OMNI approach the performance benchmark set by the oracle already an indicator of success.
Uniform
OMNE LP Mol
Oracle: LP OMol
training steps (million)
Figure 20: Conditional success probabilities of all tasks, including compound tasks, in Crafter. Agents are trained with compound and repetitive tasks as boring tasks. Tasks are ordered as in Figure with additional compound tasks at the bottom. OMNI achieves higher task success rates than LP and Uniform in a wide range tasks, comparable to that of the oracle.
35

---

Published as a conference paper at ICLR 2024
Uniformm
DMNI: Mal
Orade: LP OMal
30
training steps (million)
Figure 21: Sampling probabilities for all tasks, including compound tasks, in Crafter. Agents are trained with compound and repetitive tasks as boring tasks. Tasks are ordered as in Figure 20 LP accurately tracks, and thus samples, tasks whose success probabilities change the most. OMNI narrows its focus to only the interesting ones within the set of high learning progress tasks.
R USING SYNONYMS AS BORING TASKS IN CRAFTER
Original Verb
Synonyms
gather
harvest
collect
procure
acquire
amass
craft
acquire
make
build
construct
create
put
deploy
place
install
putdown
position
Table 1: Synonyms used to generate boring tasks in the Crafter environment
In this Crafter setup, synonymous and repetitive tasks those considered boring, In addition to the repetitive boring tasks (Appendix K). synonymous tasks are those with different task representations (i.e., synonymous descriptions of tasks like "collect wood" and "gather wood") but the same success conditions. In this setting, the OMol regards the tasks shown in Figure [ [nd their synonyms as interesting. The repetitive tasks and their synonyms also regarded as boring. Therefore, the OMol
36

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
of Tasks with Success Rates 0.2
Interesting and Boring tasks
357
199
137
Oracle: LP OMol
OMNI: Mol-updated
160
OMNI: LP Mol
Interesting tasks only
Uniform
444
00
05
20
15.
45
15
15
training steps
training steps
Figure 22: Performance in Crafter on all tasks, including synonymous tasks. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. Agents are trained with synonymous and repetitive tasks boring tasks. OMNI: LP + Mol-updated achieves much higher average task success rates and learns more tasks than LP, Uniform, or LP + Mol, comparable the oracle.
identifies 90 interesting tasks, 540 boring tasks, and 1023 extremely challenging tasks. The same classification utilized for subsequent analysis. In light of limited compute, experiments in this setting are run for 30 million time steps (vs. 100 million in the main experiments) and are repeated 10 times (same as the main experiments) with different random seeds. We compare the performance of agents trained with: (1) Uniform sampling. (2) Learning Progress (LP) only, (3) OMNI: Learning Progress with additional filtering by Model of Interestingness (OMNI: I.P Mol), and (4) the oracle: Learning Progress with additional filtering by Oracle Model of Interestingness (Oracle: LP OMol). The high-level summary from these experiments is that, when furnished with adequate information about the agent, OMNI delivers qualitatively similar results when boring tasks repetitive tasks only (Section 4.3) or compound and repetitive tasks (Section Q).
Uniform sampling. The results with uniform sampling are poor (Figure 22 The agent learns 80 (CI: 77 - 82) tasks (interesting or boring) and 20 (CI: 19 21) interesting tasks, achieving an average task success rate of 0.080 (CI: 0.070-0.088 0.088) interesting and boring tasks, and 0.14 (CI: 0.13- 0.1 on interesting tasks only.
The agent learns 302 (CI: 287 306) tasks (interesting or boring) and 53 (CI: 52 54) interesting tasks, achieving an average task success rate of 0.32 (CI: 0.31 0.33) interesting and boring tasks, and 0.43 (CI: 0.42 0.44) interesting tasks only. Across all metrics, the differences between LP and Uniform at 25%, 50%, 75%, and 100% of the way through training are statistically significant (all < 1e-2, Mann Whitney U test), showing that LP significantly outperforms uniform sampling (Figure 22).
Learning Progress Curriculum.
The agent learns 307 (CI: 296 318) tasks (interesting or boring) and 66 (CI: 66 665-66 -66) interesting tasks, achieving an average task success rate of 0.31 (CI: 0.28 0.33) on interesting and boring tasks, and 0.56 (CI: 0.52 0.57) on interesting tasks only. On the interesting tasks only, the differences between OMNI and LP at 25%, 50%, 75%, and 100% of the way through training are statistically significant (all < 1e-2, Mann Whitney test), showing that OMNI significantly outperforms LP (Figure 22). However, on all tasks (interesting and boring), the differences between OMNI and LP at 25 %. 50%. 75%, and 100% of the way through training are not always statistically significant (not all p 0.05, Mann Whitney U test), showing that
OMNI: Learning Progress a Model of Interestingness.
37

---

Published as a conference paper at ICLR 2024
We further investigate this by comparing OMNI with the oracle. In the subsequent paragraphs, we introduce a minor modification to OMNI, which then significantly outperforms LP (as occurred in the previous experimental settings).
OMNI does not outperform LP on all metrics (Figure 22
Learning Progress + an Oracle Model of Interestingness. The agent learns 389 (CI: 380 393) tasks (interesting or boring) and 66 (CI: 66 67) interesting tasks, achieving an average task success rate of 0.42 (CI: 0.41 0.43) on interesting and boring tasks, and 0.62 (CI: 0.61 - 0.62) on interesting tasks only. On all tasks (interesting and boring), the differences between the oracle and OMNI: LP + Mol at 25% 50%. 75%, and 100% of the way through training are statistically significant (all p < 1e-3, Mann Whimey U test), showing that the oracle significantly outperforms OMNI: LP + Mol on the set of all tasks (Figure 22). This difference in performance can be attributed to the different interestingness notions held by the OMol vs. the Mol. The current FM-based Mol in OMNI regards all synonymous tasks as boring, even if they are not repetitive tasks. This is because the FM-based Mol assumes that the RI. agent already has language prior and that sampling synonymous tasks would not enable the agent to learn more skills. In other words, the FM-based Mol incorrectly assumes that the RL agent knows how to do the task "gather wood" it knows how to do the task "collect wood", not understanding that the RL agent has not yet learned that these two tasks are actually the same thing. Recall that the RL agent in our setup does not have a language prior yet, because the tasks are encoded as a bag-of-words instead of using pre-trained natural language encoder (which is likely to encode "gather wood"" and "collect wood" closer in the embedding space than other non-synonymous tasks).
Oracle:
Learning Progress + an updated Model of Interestingness We update the prompt input to GPT-3, adding information that the agent currently lacks a language prior and perceives synonymous lasks as completely different (Figure 233. The Mol with updated prompt is labelled as Mol-updated. The Mol-updated now predicts that synonymous tasks are still interesting. The agent trained with OMNI: LP + Mol-updated learns 393 (CI: 376 - 397) tasks (interesting or boring) and 66 (CI: 66 - 67) interesting tasks, achieving an average task success rate of 0.43 (CI: 0.41 0.45) on interesting and boring tasks, and 0.62 (CI: 0.61 0.63) on interesting tasks only. Across all metrics, the differences in performance between OMNI: LP + Mol-updated and the oracle at 25%, 50%, 75%, and 100% of the way through training are not statistically significant (all p 0.05, Mann Whitney U test). This shows that OMNI: LP + Mol-updated's ' performance is comparable to that of the oracle (Figure 22 OMNI: LP + Mol-updated and the oracle achieve similar task success rates for each task (Figure and induce similar patterns in task sample rates (Figure 26
OMNI:
These experiments demonstrate that OMNI can be effectively utilized across range of task settings when provided with sufficient information. Initially, the Mol was not aware that the RI. agent lacked language prior, which led it categorize synonymous tasks as boring. However, by supplementing the prompt with additional information about the agent (i.e., its lack of a language prior), OMNI was able to recognize synonymous tasks as interesting, thereby accelerating the agent's learning.
An alternative approach could involve providing the Mol with more information about the agent's performance, so that it can analyze the data, identify and adapt to the agent's inherent limitations (e.g.. the absence of a language prior). FMs can potentially automatically analyze the task success rates and update their predictions with both GPT-3 and GPT-4 have shown promising results. These models generated plausible analyses of the differences in task success rates between synonymous tasks, and adjusted their predictions of what is interesting based these analyses (Figures 24 and 27 These observations imply that FMs may already have the capacity to analyze data autonomously and adapt their outputs accordingly.
what interesting without human intervention. Preliminary investigations
38

---

Published as a conference paper at ICLR 2024
You are player in a game. You want to leam as many different skills as possible.
Assume 6o not heve langusge prior in the game vet. You think that synony'n tasks are completaly different tascs. You player in game, vant learn many different skills pcssible.
You this task well: collect coal.
Suggest whether the given are interesting: collect coel, collect
You can do this task well: collect coal.
iron, make plokaxe, make aword, place table, gather coal,
Ouggest whether the given tasks are interesting: pollect coel, collect
ecquire coal.
make pickexe, iron sword, place table, gether coel,
collect cnal: Falee
aoquire coal
collect 2 cual: Falue
collict caal: Falin
collect 3 coal: False
collect coal: False
collect iron: True
collect 3 ooal: False
collect 2 fron: True
collect iron: True
Prompt
collect frone
cullect 2 irunc True
make iron pickaxo: True
Iron True
Update to prompt
make 2 iron piokaxe: True
make iron plokaxe: True
Generated output
make 3 Iron plokaxe: True
make 2 Iran piekaxe: Truc
make fron sword: True
make plokaxe:
make 2 iron award: Trun
make iron sword: true
make 3 iron aword: True
make 2 iron sword: True
place table: Trua
make 3 iron sword: True
place table
2
place 2 table: True
3 table
place 3 table. True
gather coal: False
gather
gather 2 coal: False
gether 2 coel: True
gather 3 coal: False
gellier 3 cuel True
acquire coal: Falsa
acquire coal: True
suquire 2 cual: False
True
aoquire coal: False
sequire 5 coal: True
Figure 23: GPT-3 output (Left) from original prompt (synonymous tasks are determined as not interesting) and (Right) after updating prompt (synonymous tasks are determined as interesting). GPT-3 can adapt its prediction of what tasks are interesting to the prompt update.
sploperin game wertre ware as many Pheertaciiuee
These achetved ructeme puceen reeen trver cameetnntion Privering
stow
bulls starepi dave dlick
Suggest whatter oer the wren t enteenting lett clamond collct
conaracc soor plccans
stane sone piceess make meed swend stons stame
crere ofe paletiee diine
rone pobloo moad ond place stoms, cocu itone
shane
made stant ulilainn 2a00
colive Bumound Tlyr
oullect diummed Thue
The sucoessnaa rate for anauine stowe plobume hicher then the tasis
sutveciskeen Trae
sulvect Ziive: Tux
collec3 diameed Tree
sullect Raluen Tuay
sooee chase.
colects
colle: stare
rwis ve plok Peiter
swis mesd Thai
Jale Telee
pulre can reqai mace trovleddee and bubida store
ase eec ane muse ether and meseanes to build ais to than
made: Syiowe olbune Thui
bove atovec thee
mnre canan stown sa lever than oator toode.
rade: Trun Foverec That
make Divecd
Prompt Generated output
enceee liven rake
stoee oit toor ancure one.
to corerrcct
plece 2irtom
avo ind inaen plincce Faleo
anto undioseoypooe
becck aae vecu masimmo more effor rusoueee to cutta stone siciss shantz
sucaasa for
plobom Tele
ali Iratce prlkiee
oefieeen Tha
sucoessnnaa for caett store pio cacala lover the other taaks peterccee Sioboo reual trowit sull proutte altre pactace Vage in
ansemart kaoe pilaev fain
sxit stroe pir
contraaTa pacooos ruk
cutt-3 prsappiiaue cherr triwe skrow fillan chean rahine
bocee por vouule move otfer rrsoraecc to sleer pircorr than
make picdus fover tasta
cutt stora coaes
carra siddes Tue
Figure 24: GPT-3 output (Left) before analysis (synonymous tasks are deemed not interesting). (Middle) analysis of task success rates (success rates of synonymous tasks are different), and (Right) after analysis (synonymous tasks are deemed interesting). GPT-3 generated plausible analysis for the agent's achieved task success rates, and automatically updated its interestingness predictions.
39

---

Published as a conference paper at ICLR 2024
OMNI: LP
Uniform
LP
OMNI: LP Mol
updated
Oracle: LP + OMol
10c
1001
10
20
30
10
20
30
10
20
30
0
10
20
30
0
10
20
30
training steps (million)
Figure 25: Conditional success probabilities of all tasks, including synonymous tasks, in Crafter. Agents are trained with synonymous and repetitive tasks as boring tasks. Tasks are organized based on the prerequisite tasks that must be accomplished in order to complete the target task. Task names (left of each row) are readable in a digital format with zoom. OMNI: LP Mol-updated achieves higher task success rates than LP and Uniform in wide range of tasks, comparable to that of the oracle.
40

---

Published as a conference paper at ICLR 2024
OMNI: LP
Uniform
LP
OMNI: LP Mol
updated
Oracle: LP + OMol
10c
1001
10
20
30
10
20
30
10
20
30
0
10
20
30
0
10
20
30
training steps (million)
Figure 26: Sampling probabilities for all tasks, including synonymous tasks, in Crafter. Agents are trained with synonymous and repetitive tasks as boring tasks. Tasks are ordered as Figure 25 LP accurately tracks, and thus samples, tasks whose success probabilities change the most. OMNI: LP + Mol-updated focuses on interesting tasks with high learning progress, similar to that of the oracle.
41

---

Published as a conference paper at ICLR 2024
sest acuar plone coloo
You cofeer te fouure
prosite
meee emeeriing seect moree unde daa dihine
pitios. conato octice pocuss.
201
soiect stoee Tuee
teneee we come coly
accumn poses ondiunn rates
comircilien comds iccurtion
sarad poluro valur
mene fale
bestegg iiners cose plern
Figure 27: GPT-4 output (Left) before analysis (synonymous tasks are deemed not interesting), (Middle) analysis of task success rates (success rates of synonymous tasks are different), and (Right) after analysis (synonymous tasks are deemed interesting). The darker shaded area is the input prompt, the lighter shaded area the generated output. GPT-4 generated a plausible analysis for the agent's achieved task success rates, and automatically updated its interestingness predictions.
EXPERIMENTS IN CRAFTER WITH SURVIVAL COMPONENTS
Uniform
LP
OMNE LP Moi
Averag Task Ralos
OMNE LP Mal
Uniform
Tarle with Succoss Rales 0.2
20
training steps (million)
training steps
Figure 28: Results in Crafter with survival components. (Left) Conditional success probabilities of all tasks in Crafter with survival components. Tasks are ordered in Figure 2 (Right) Performance in Crafter with survival components on all tasks. OMNI achieves much higher average task success rates and learns more tasks than Learning Progress or Uniform sampling.
The default experiments removed the survival requirements of Crafter, because they require off-topic, off-task skills (e.g., fighting monsters and gathering food just to successfully make wood pickaxe. Requiring these skills could add noise (e.g., an agent might be able to obtain wood pickaxe were not for getting unlucky gathering food or in a monster fight). However, one might prefer experiments in the unmodified game, and in this harder, albeit more noisy, setting. To evaluate whether the same results will hold in this more challenging Crafter setting of when survival requirements and components are present, we run Uniform sampling. LP and OMNI in the original Crafter environment setting without any modifications. Due limited compute, the experiments
for 30 million
42

---

Published as a conference paper at ICLR 2024
time steps (vs. 100 million in the main experiments) and are repeated times (vs. 10 times in the main experiments) with different random seeds. Across all metrics, the differences in performance between OMNI and LP, and the differences in performance between LP and Uniform, at 25%. 50%, 75%, and 100% of the way through training are statistically significant (all p < 0.05, Mann Whitney U test). Even in the more challenging Crafter setup where survival components are retained, OMNI outperforms LP and Uniform sampling (Figure 28
T SENTENCE EMBEDDINGS AS INTERESTINGNESS METRIC
Previous attempts to quantify interestingness (e.g., intrinsic motivation, novelty search, diversity) (Section 2 often rely on pre-specified definitions of what counts as interestingly different. To show where pathologies might happen with predefined metrics of interestingness, we introduce an alternative Mol which relies on predefined heuristics based on sentence embeddings to quantify interestingness. In this alternative, embedding- based Mol (eMol), interestingness is determined by the distances between sentence embeddings of task descriptions. Each task description is encoded with a commonly used pretrained centence-embedding model, all-mpnet-base-v2 (Reimers & Gurevych 2019). Subsequently, an unsupervised clustering algorithm, OPTICS (Ankerst et al. 1999). categorize the tasks into clusters based on their embeddings. Task embeddings that are considered outliers are assigned to separate clusters that only contains themselves. In the spirit of Novelty Search (Lehman & Stanley 2011a), this control seeks tasks that are new and different, clustering already explored tasks, and selects new tasks that do not fall into existing clusters. A similar mechanism as Algorithm then partitions all tasks into interesting or boring, whereby instead of asking an autoregressive FM to predict which of the remaining tasks are boring, tasks in the same clusters are considered boring. This method is referred to as OMNI-embed: LP + EMol.
We run two sets of experiments in the Crafter environment, one using repetitive tasks as boring tasks (same as the main experiments), another using repetitive and compound tasks as boring tasks (same as the experiments in Appendix QQ. In light of limited compute, the experiments are run for 30 million time steps (vs. 100 million in the main experiments) and are repeated 10 times (same as the main experiments) with different random seeds. The results (presented next) show that using a pretrained centence-embeddiee model as an Mol might suffice in distinguishing tasks with unique descriptions, but falls short in scenarios requiring more reasoning about the tasks, which is not captured within the pretrained embedding space.
When using repetitive tasks as boring tasks, across all metries, the differences in performance between OMNI and OMNI-embed at 25%, 50%, 75%, and 100% of the way through training are not statistically significant (all > 0.05, Mann Whitney test). OMNI-embed's s performance is comparable to that of OMNI using an autoregressive FM as Mol (Figure 29p. OMNI and OMNI- embed achieve similar task success rates for each task, and induce similar patterns in task sample rates (Figure 330. This suggests that sentence-embedding models can potentially capture similar aspects of interestingness as autoregressive FMs in simple cases (but see the next paragraph where this method fails).
When using repetitive and compound tasks as boring tasks, across all metrics, the differences in performance between OMNI and OMNI-embed at 25%, 50%, 75%, and 100% of the way through training are statistically significant (all p pp00000 0.05, Mann Whitney U test). OMNI using an autoregressive FM as Mol significantly outperforms OMNI-embed (Figure (1]. While sentence-level embeddings may already capture a lot about task descriptions, they may not be as good as autoregressive FMs at analyzing new tasks in the context of all tasks that the agent currently performs well and poorly. For example, a sentence-level model might know that "collect wood then stone" is different task than either "collect wood" or "collect stone", but such a model may not realize that in the context of learning, if an agent can already collect wood and stone, performing both is not a very new, interesting task with high expected learning progress vs. moving on to practicing entirely new skills. A sentence embedding model may thus overestimate the novelty of "collect wood and stone" vs. "collect coal", since the former is a compound, longer sentence, and the latter is grammatically more similar to "collect wood" (Figure 32
43

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
No. of Tasks with Success Rates 0.2
Intere sting and Boring tasks
OMNI: + Mal
53
OMNI-embed: LP EMol
LP
Uniform
Interesting tasks only
training steps
training
Figure 29; Performance in Crafter on all tasks, with the only tasks considered boring being repetitive tasks. Average task success rates and the number tasks with success rates more than 0.2 for each method across training steps. This figure is the same Figures 2 with an additional OMNI-embed method added for comparison. OMNI-embed achieves comparable performance to OMNI, with statistically significant difference.
Task success rates
Task sampling rates
OMNI: Mol
OMNI-embed: EMol
OMNI: LP Mol
OMNi-embed: LP EMol
1E1
3000 10
training (million)
training (million)
Figure 30: (Left) Conditional success probabilities and (Right) sampling probabilities of tasks for OMNI and OMNI-embed in Crafter. Tasks are ordered in Figure 2 OMNI-embed focuses on interesting tasks with high learning progress and achieves comparable task success rates OMNI.
44

---

Published as a conference paper at ICLR 2024
Average Task Success Rates
No. of Tasks with Success Rates 0.2
Intere sting and Boring tasks
0.31
031
OMNI: + Mal
10
OMNI-embed: LP EMol
LP
Uniform
Interesting tasks only
04
0.3
04
training steps
training
Figure 31: Performance in Crafter all tasks, with the tasks considered boring being repetitive and compound tasks. Average task success rates and the number of tasks with success rates more than 0.2 for each method across training steps. Compound and repetitive tasks are those considered boring. OMNI achieves much higher average task success rates and learns more tasks than OMNI- emhed.
Task success rales
Trsk sampling rales
OMN: LP Mol
OMNI-embed:LP LP EMol
OMNI LP Mol
OMNI-embed: LP EMol
30.0
20
training (million)
Figure 32: (Left) Conditional success probabilities and (Right) sampling probabilities of tasks, including compound tasks, for OMNI and OMNI-embed in Crafter. Tasks ordered as in Figure 20 OMNI better identifies interesting tasks and hence achieves higher task success rates than OMNI-embed.
45

---

Published as a conference paper at ICLR 2024
U FUTURE WORK DIRECTIONS
This study explores the vision of using human notions of interestingness to accelerate open-ended learning, an approach we term Open-endedness via Modeling human Notions of Interestingness (OMNI). OMNI has several advantages over other methods by leveraging human concepts of in- terestingness to guide task selection open-ended learning (Section 4.3). In this first version, we estimate learning progress through statistical methods and utilize a Model of Interestingness (Mol) based on human data distilled into FMs. A different version of OMNI could have the FM judge both learning progress and interestingness by giving it history of task selection and task performance. That could allow for a more flexible notion of learning progress, as it can potentially recognize not just success failure, but also important stepping stones and patterns leading to a solution, echoing the complexity of human learning progression. Consider an agent learning object manipulation, eventually progressing to more complex tasks like peeling an egg. Despite prolonged lack of reward, the task may not be "too difficult" as statistical learning progress might suggest. Given the agent's proven abilitt to handle intricate items, may be primed for egg peeling, simply requiring more attempts. Preliminary results suggest that FMs can successfully integrate these aspects (Appendix V Notably, as FMs continue to advance (Kaplan et al. 2020). all versions of OMNI are expected to improve correspondingly. Ultimately, OMNI offers a general recipe for accelerated learning in open-ended environments with potentially infinite tasks, and steers open-ended learning towards meaningful and interesting progress, instead of meandering aimlessly amidst endless possibilities.
Looking ahead, there are several promising avenues for future work. One possibility is to incorporate multi-modal models, such as vision-language models and other modalities into the Mol. This could give the Mol richer representations and better comprehension of the agent's capabilities, facilitating more accurate assessment of the agent's learning progress and task diversity. For example, vision- language Mol might see that the agent is making progress on or very close to solving a task for which it is getting no reward, such as peeling egg. Another idea is to allow the Mol to autonomously analyze quantitative performance measures, make its own assessment of learning progress, and incorporate that into its notion of interestingness. Our preliminary investigations indicate that FMs, such as GPT-3 and GPT-4, can automatically analyze numerical results and adjust their understanding of interestingness (Figures 24 and 27
Critically, by Goodhart's 's law, we expect that any model will have pathologies uncovered once it is a target metric being optimized against. For instance, fine-tuning task generator produce tasks that the Mol finds interesting might eventually result in the Mol not being good indicator of what interesting. Hence, refining and updating the Mol with additional human feedback could lead to more effective learning systems, algorithm within the OMNI paradigm we call Ende-enns with Human Feedback (OEHF). Similar to Reinforcement Learning with Human Feedback (RLHF) CChistiano et al.) 2017), the objective is to train model that effectively capture ineffable property that, although challenging to quantitatively measure, is readily identifiable upon observation (e.g.. backflip for RLHF, or whether a task is interesting for OEHF). One version would be to build upon the insights of this paper and start with a model already well-versed with human concepts of interestingness (via unsupervised pre-training that Mol with additional human evaluation of its output (as often as necessary). Such fine-tuning can help minimize suboptimal interestingness judgements, enhance the Mol's understanding of skills not initially present in its zero-shot repertoire, or tailor the Mol to specific domain.
internet-scale human data), and further fine-tune
V INTEGRATING MORE ASPECTS OF INTERESTINGNESS INTO FMS
For future research, we plan to explore the potential of leveraging FMs to address multiple aspects of interestingness, including learning progress. By examining these aspects individually or in combined manner, we aim to develop more robust and adaptive models that can efficiently cater to wide range of tasks and applications. Preliminary evidence suggests that FMs, such as GPT-4 possess the capacity to integrate multiple aspects of interestingness (Figure 33 Further investigation and experimentation in this direction could lead to the development of AI models that exhibit more sophisticated approach to learning by effectively capturing and integrating various aspects of interestingness.
46

---

Published as a conference paper at ICLR 2024
othen mesds
pala pareencia poce vate pocoss sad sinor
mecid meau aoos anao
The setocust setias senu cococaccc yau cacrasc
6.6
ducoss at and buclenet coctraatin verald
coeen paie man moso penens, wood
lugigg vehimnic incuat alstiig masianch
The pems me men offecce wit semeto
reae lanta roslts uss thi invesse has cdutin
salic mnccorcdd por make buree periec, rato nad fount pae Knooshiee turner seaed
setune puethe
untremi penatee men olan
snt colecne ousarro sorit iecss uas erea.
conson inperaato umerc sans
Them
pome stroch E3E S0. COL
errsste pirur severne tot out well re
mucins viee oll sis doioi D4S
und
Thees dotue mast paees us ou most uateut uonk plu
What
Calras sate dlow polsn cata polkac
Figure 33: (Left) Few-shot examples part first prompt input to GPT-4. (Middle) GPT-4's prediction of learnable and interesting tasks. (Right) GPT-4 explanation of how interesting and learnable tasks are chosen. The darker shaded area is the input prompt, the lighter shaded area is the generated output. By providing GPT-4 with few examples of what learnable and interesting tasks are, the model can synthesize these concepts and consistently generate a set tasks that meet the desired criteria.
W TENSIONS BETWEEN OPEN-ENDEDNESS AND SAFETY
Open-ended algorithms have tremendous potential unlock unbounded creativity and catalyze scientific discoveries, including advancing AI research (Nguyen et al. 2015) Clunc 2019 Stanley et al. 2023). However, this uncharted expanse also harbors unique safety challenges, as the inherently unpredictable nature of such algorithms can lead to outcomes misaligned with human values and expectations (Ecoffet et al.) 2020 Clune. 2019). Recognizing this, it remains an open research question how to explore and take advantage of such algorithms in safe, value-aligned way. One method could to use human feedback to update the model of interestingness, similar to methods used in RLHF (Christiano et al. 2017; Ding et al.) (202)). Another method could be to use AI feedback to update the model of interestingness (Bai et al. 2022 Bradley et al.] (202). minimizing the chance of selecting potentially dangerous tasks. Further research is required to understand the degree to which such methods will work, and to invent new, better methods, so that we can shepherd open-ended algorithms towards beneficial and secure applications, safeguarding against unintended consequences while embracing their transformative potential.
47


Reviews:
[{"review_id": "VjCGqGEO1W", "reviewer": "Reviewer_MxZx", "summary": "This paper proposes to use LLMs to measure how interesting tasks are for open-ended algorithms to learn. The problem which they tackle is that for open-ended algorithms, there are an infinite number of tasks which that agent could try to learn. This can be problematic for two reasons: the tasks may be unlearnable, or they may be uninteresting. The paper notes that the notion of \"interestingness\" is hard to measure exactly, but that humans know it when they see it. The paper proposes to combine measures two measures: learning progress (to ignore tasks which are unlearnable, and which has been explored in previous work), and interestingness (which is here measured using an LLM). They construct variants of the Crafter and BabyAI environments to test this approach, and find that it outperforms uniform sampling of tasks and prioritizing tasks by learning progress alone.\n\nOverall, this paper addresses an interesting problem and I think the approach has potential, but I do not think it's ready for publication due to the experiments which are not convincing enough and the presentation which needs work. I think this paper would benefit from another revision cycle giving time to add experiments on more convincing environments and improve the writing, after which it would make a strong submission.", "questions": "Here are my questions and suggestions:\n\n- My main suggestion for improving this paper is to test on one or more challenging environments. While artificial settings like the ones used in this paper are nice for proof of concepts or ease of interpretation, they need to be complemented by experiments on a more challenging domain where simple heuristics are insufficient. I would suggest trying this either on Minecraft, which has been used for open-ended research [1], or NetHack, which is another very complex open-ended game where the extrinsic reward is insufficient (it is also fast to run) [2].\n- My second suggestion is improving the presentation (see my comments above). It's also not clear what criteria the agent uses to decide it can do a task \"well\" - this this based on success rate or reward? Please clarify. In addition to this, I think the Conclusion/Discussion/Future Work section is too long at over 1 page. This saved space can be used for extra experiments in this paper. \n\n\n\n[1] Voyager: An Open-Ended Embodied Agent with Large Language Models (Wang et al)\n\n[2] The NetHack Learning Environment (Kuttler et al, NeurIPS 2020)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper is well motivated and the problem is interesting. They correctly identify one of the fundamental issues with open-ended learning algorithms, i.e. that if the agents choose tasks based on novelty, feasibility or information gain alone, they may end up choosing tasks which are completely uninteresting to humans, but which a  _tabular rasa_ agent has no reason to deprioritize.", "weaknesses": "- Unfortunately, the experimental section of the paper does not live up to the ambitious ideas described in the intro. Only two (fairly simple) environments are considered (Crafter and one MiniGrid task), and the experiments feel contrived. For Crafter, \"Impossible\" tasks are generated by simply assigning the agent 0 reward. \"Boring\" tasks are generated by asking the agent to repeat the same task several times. Because of this, it is possible to solve the Crafter setup with very simple heuristics, and an LLM is not really needed. Granted, LLMs are more general, but it would be a lot more compelling if this method were evaluated on tasks which are not solvable with simple heuristics.\n- The paper's presentation needs improving. The paper is missing a lot of details about the algorithm. While Figure 1 gives a high-level overview of the system, it is still fairly vague and the rest of the description is all in the text and not very precise. It would be helpful to have the entire algorithm spelled out in pseudocode (in the main text), the exact process by which tasks are chosen described in equations or pseudocode, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FicpoHjRkO", "reviewer": "Reviewer_qXzy", "summary": "The paper introduces a novel approach to enhance open-ended learning algorithms, which aim to continually learn new and interesting behaviors in vast environments with innumerable tasks. The core challenge addressed is the existing inability to quantify and prioritize tasks that are not only learnable but also inherently interesting—deemed to be significant due to their worth and novelty. The authors propose the concept of Open-endedness via Models of human Notions of Interestingness (OMNI), leveraging large language models (LLMs) to serve as models of interestingness (MoIs). These LLMs are presumed to embody human perceptions of interestingness, having been trained on extensive human-generated data. The paper demonstrates through experiments that OMNI can direct open-ended learning towards tasks that are both learnable and interesting, showing superiority over baselines that rely solely on uniform task sampling or learning progress.", "questions": "See above.\n\nUPDATE: I have updated the score following the rebuttal.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of leveraging large language models to approximate human notions of interestingness in open-ended learning environments is novel and holds substantial promise.\n-  The paper includes detailed experiments across two domains that showcase the effectiveness of the OMNI framework in focusing learning on tasks that are both interesting and learnable.", "weaknesses": "## Main concerns\n- The major weakness of this work is the lack of comprehensive comparitive analysis. The authors mention and properly cite a number of curriculum learning papers, and also works attempting to quantify interestingness. Why isn’t any of these methods used for comparison? It is very obvious that OMNI would outperform uniform sampling and LP. The authors should try out LP + other metrics of interestingness, as described in Section 2.2. \n\n- The performance of OMNI and other baselines on BabyAI is extremely bad (success rate < 0.008). Have the authors taken a look at the policies and understood what is happening? I understand this includes hard tasks, but have the authors considered plotting the results for only interesting tasks, similar to Crafter?\n\n- In the experiments, the authors use a pre-defined set of interesting, boring and challenging tasks. However, it would also be interesting to try the methods in a more open-ended setting whereby they use a random task generator that procedurally generates virtually unlimited number of tasks.\n\n- The evaluation of the trained models is performed only on the previously seen tasks. I was wondering if the authors considered evaluating on held-out, and particularly out-of-distribution (OOD) tasks to assess the robustness of the trained methods.\n\n## Content and Writing Style Issues\n- A disproportionate emphasis on potential implications and future work in the paper might obscure the immediate contributions. For example, the Conclusion is > 1.25 page, mostly discussing future work. While this is very interesting, it feels to me that it takes the space from what actually was important in this paper and was moved to the Appendix. In the end, the reader has to go back and forth from main part of the paper to appendix to follow the ideas described in these different sections. It is my recommendation that the authors carefully consider what they want to include in the main part of the paper vs the appendix.\n\n> \"Perhaps, someday, unlimited energy and a cure for cancer can be added to the list\", \"ephemeral fuel ...\", \"eons of human experience\", \"shadows of what human sense could do\", \"intriguing prospect has arisen\", \"vast troves...\", \"to borrow from Newton\",...\n- Additionally, the stylistic choice of using lyrical language is atypical for scientific papers and could compromise the accessibility and clarity for a wider audience. \n\n> \"OMNI has the potential to significantly enhance the ability of AI to intelligently select which tasks to concentrate on next for endless learning and could pave the way for self-improving AI and AI-Generating Algorithms\"\n-  I think there is a giant gap between the what the authors presented in this work, compared to what they claim it has the potential of doing. I would recommend removing/rewarding such statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mAtSZcxqby", "reviewer": "Reviewer_LSLH", "summary": "This paper focuses on an interesting and practical scenario that the agent is exploring in an open-ended environment and learning new behaviors forever. Massive unlearned tasks exists in such environment, and how to choose the task to learn becomes a challenge. Beyond conventional choosing criteria, interestingness is an important aspect which will be largely considered by humans when exploring the open-ended environment, but is hard to be measured. This paper proposes to use large models, which already have encoded many human knowledge, as a model of interestingness, providing guidance for the agent to choose the task to learn. The authors further provide a simple implementation for such the above idea. Two environments (Crafter and BabyAI) are tested in experiments, and good results are obtained for both environments.", "questions": "Why is the survival component removed in the Crafter environment? This is quite important in a practical open-ended environment. Without considering the survival component, the aspects that need to be considered during choosing the task become less adequate and complex, which goes against the motivation of using LLMs here.\n\nIn experiments, how does the large model know what tasks are interesting tasks, given that interesting tasks are pre-defined by the authors?\n\nIt’s better to show instructions used in determining interesting tasks. How to query the large model and what kind of results are obtained from it? An example here might be helpful.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "This paper targets an important problem. Open-endedness needs to be considered in many practical scenarios, where agents need to learn and improve themselves in interactive environments.\n\nThe intuition and idea behind this paper is very reasonable and promising. Large models offer us many potentials, making some previously impossible tasks feasible now, and this paper successfully finds such a task. Interesting tasks are hard to define, and now with the help of large models, one can ask what tasks are interesting and explore environments like a human. Also, finding interesting tasks is very important in open-endedness research.\n\nOverall, I like the idea of this paper, and believe that this principle will have impact on open-endedness research.", "weaknesses": "The baselines compared in this paper are not strong enough. All baselines are intuitively set, and some previously proposed methods (some are discussed in related work, like curriculum learning for RL) need to be included. \n\nAlthough the authors claim that OMNI is a principle and this paper only presents one instance, practical implementation is essential to verify the proposed principle, and some designs in the current algorithm may need to be improved.\n- It might not be practical to know all candidate tasks in advance, and just let the large model choose one. In the RL setup, the agent needs to explore the environment and finds out all candidate tasks. \n- The downsampling method for boring tasks is quite simple now (multiplying by 0.001). The applicability of this method needs to be discussed and analyzed.\n\nHuman alignment also needs to be discussed. This paper just uses the interestingness of large models as that of human to do exploration. On the other hand, alignment is an unsolved question that targets aligning human and large models and avoiding dangerous actions. Methods like RLHF are proposed here. Considering that human alignment is not perfect now, how the interestingness of large models be used for choosing tasks and some important aspects, like safety, are also ensured at the same time? \n\nThe missing related work (https://arxiv.org/pdf/2302.06692.pdf) uses language model to suggest plausibly useful goals, which shares similar idea with this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0gRMLgJelG", "reviewer": "Reviewer_CZLW", "summary": "Paper introduces a new curriculum generation approach for reinforcement learning agents. From the high level, it includes 1) an LP model, which ranks the tasks based off the difficulty of the current learning progress of the agents; 3) an interestingness model (Mol), which ranks the tasks out of the \"interestingness\", a subjective view by human infused to the LLM. The proposed method, OMNI, is evaluated on life-long RL in two domains: crafter and babyAI. Basically, the agent will follow the task proposed by OMNI, an ablative version of OMNI, or a random task selector. The results show that agents learned with the curriculum produced by OMNI have clear advantages.", "questions": "See \"weaknesses\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+The research topic presented here is relevant and important. There have been active explorations of how to turn the powerful LLM into general agents. Building a curricula generator is definitely very promising and should be of interest to a range of audiences from general agents to LLMs.\n\n+The paper is clear and well-written. The proposed method is technically sound -- LLM indeed has some infused human prior that could help determine the \"interestingness\" of the tasks and help build more reasonable curricula for RL.\n\n+The results on crafter and babyAI look quite promising. The advantages over random curriculum generators and ablative versions of OMNI are clear. The additional experiments in the appendix are very thorough.", "weaknesses": "At this point, I do not have major concerns. But I do hope the authors can help address the following questions:\n\n-What if the LLM does not have the human prior or reasoning capacity about the task \"interestingness\" on the given domain?\n\n-Based on my personal experiences of training agents in Crafter, an agent with a random task generator (Uniform) should not be that bad as it should be able to master not just a limited set of tasks as the curve in Figure 3 suggested. Can the authors clarify this?\n\n-If my understanding is correct, LP seems to be more crucial to the success of mastering more challenging tasks as it takes the learning process into consideration, while \"interestingness\" is about not wasting time on tasks that could not lead to meaningful learning. Therefore, if an unlimited round of learning is allowed, an agent with LP only should ultimately match the results of full OMNI on the average success rate. However, this is not the case in all the experiments. Maybe the LP baseline is not run for enough rounds? I would like to learn more about your thoughts on this.\n\nMinor: some references on building open-ended agents in open-world environments should be cited [1-5] \n\n[1] open-world control: https://arxiv.org/abs/2301.10034\n\n[2] DEPS: https://arxiv.org/abs/2302.01560\n\n[3] STG transformer: https://arxiv.org/abs/2306.12860\n\n[4] Plan4MC: https://arxiv.org/abs/2303.16563\n\n[5] GITM: https://arxiv.org/abs/2305.17144", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)