OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code

## Abstract

Abstract Open-ended and AI-generating algorithms aim to continuously generate and solve increasingly complex tasks indefinitely, offering a promising path toward more general intelligence. To accomplish this grand vision, learning must occur within a vast array of potential tasks. Existing approaches to automatically generating environments are constrained within manually predefined, often narrow distributions of environment, limiting their ability to create any learning environment. To address this limitation, we introduce a novel framework, \ours , that augments previous work in Open-endedness via Models of human Notions of Interestingness (OMNI) with Environments Programmed in Code (EPIC). \ours leverages foundation models to autonomously generate code specifying the next learnable (i.e., not too easy or difficult for the agent’s current skill set) and interesting (e.g., worthwhile and novel) tasks. \ours generates both environments (e.g., an obstacle course) and reward functions (e.g., progress through the obstacle course quickly without touching red objects), enabling it, in principle, to create any simulatable learning task. We showcase the explosive creativity of \ours , which continuously innovates to suggest new, interesting learning challenges. We also highlight how \ours can adapt to reinforcement learning agents’ learning progress, generating tasks that are of suitable difficulty. Overall, \ours can endlessly create learnable and interesting environments, further propelling the development of self-improving AI systems and AI-Generating Algorithms. Project website with videos: https://dub.sh/omniepic .

## 1 Introduction

In recent years, the field of artificial intelligence (AI) has witnessed groundbreaking achievements, particularly with advancements in reinforcement learning (RL) (OpenAI et al., ; Silver et al., ) and foundation models (FMs) (Brown et al., ; Radford et al., ).
Yet, the most ambitious goal of AI — building generalist agents capable of autonomously perceiving, reasoning, deciding, and acting within complex environments — remains a formidable challenge. The creation of general-purpose agents promises immense societal benefits, provided we can address the critical safety and existential risks involved (Bengio et al., ), including those unique to open-ended and AI-generating algorithms (AI-GAs) (Clune, ; Ecoffet et al., ).
While researchers have made substantial progress in developing powerful learning architectures (Hafner et al., ; Jaegle et al., ; Vaswani et al., ), these algorithms have been applied to a relatively narrow range of tasks due to the limited availability of diverse datasets. Compounding this issue, current models require extensive amounts of data and fine-tuning to be trained. Consequently, over the past decade, the bottleneck in advancing AI has shifted from improving learning algorithms to acquiring the necessary data to train them (Jiang et al., ). In other words, the main challenge has become: how to effectively acquire large amounts of data for diverse tasks.

A fundamentally different approach involves developing open-ended algorithms that continuously generate and solve new challenges endlessly (Stanley et al., ). The objective of open-ended algorithms is to ignite an explosion of creativity and complexity in a computer, akin to the processes observed in biological evolution and human culture, including science and technology. Replicating open-ended evolution *in silico* extends beyond understanding biological life or developing engaging simulations. It delves into fundamental questions about the nature of creativity, the potential for machines to exhibit life-like characteristics, and the emergence of general intelligence. Given that biological evolution is the only known process that has successfully created general intelligence, its principles hold invaluable insights for advancing the field of AI, and creating AI-GAs (Clune, ).

For open-ended algorithms and AI-GAs to succeed, they must operate within a *vast* task space capable of generating an infinite array of potential challenges. Previous open-ended algorithms (Sudhakaran et al., ; Wang et al., ; ; ; Zhang et al., ) have been applied to limited domains with specific types of worlds (e.g., obstacle courses) and confined to predefined parameterizations (e.g., obstacle size and type), restricting their potential to exhibit true open-endedness. Achieving *Darwin Completeness* — the potential to generate *any possible learning environment* — is essential for realizing the full potential of AI-GAs (Clune, ).
However, Darwin Completeness presents a significant challenge: the ability to generate infinitely many environments makes the search potentially intractable, or even impossible. Exploring this immense environment search space may involve endlessly generating trivial, redundant, or overly complex tasks that do not effectively contribute to the agent’s learning progress. The main challenges are to ensure that generated environments are novel, interesting, solvable, and appropriately matched to the current capabilities of the learning agents. Biological evolution required an unfathomable amount of computation over billions of years to produce intelligence. Therefore, one key scientific challenge is to determine how we can optimize the process of generating and solving interesting tasks within a Darwin Complete environment search space, so as to create an AI-GA given the computational capabilities we expect to have in the future.

Previous work in Open-endedness via Models of human Notions of Interestingness (OMNI) (Zhang et al., ) leverages FMs to improve open-ended learning by focusing on tasks that are both learnable and interesting. However, OMNI, like all prior open-ended works (Sudhakaran et al., ; Wang et al., ; ; Zhang et al., ; Wang et al., ), was confined to generating tasks within a narrow environment search space, inhibiting the generation of *any* possible learning environment. This paper introduces a novel framework, \ours, that augments OMNI with Environments Programmed in Code (EPIC). \ours utilizes FMs to choose the next interesting and learnable task and subsequently generate environment code to enable the agent to learn how to solve that task. Our approach generates not only the simulated world but also the reward and termination functions, allowing it, in principle, to create any simulatable task. We take advantage of pre-existing simulators and \ours writes code to create tasks within it. For example, if the task is to kick a ball to hit a moving target, \ours would generate the environment code to simulate the physics, the agent, the ball, and the moving target, rewarding the agent when the ball hits the target. Conversely, for a task involving maneuvering the ball around a moving target, the simulated world remains the same, but the reward function could differ, penalizing the agent for any contact with the target. A model of interestingness (MoI) is employed both when generating the next task and checking if any newly proposed task is interestingly new compared to similar ones in the archive. Finally, we introduce a success detector that can automatically determine whether the agent has successfully completed any proposed task.

Our vision is for this algorithm to generate any code, including downloading, installing, using, and modifying any existing simulator, or even writing the code for a new simulator. Given that code is Turing Complete, \ours could potentially create any computable environment (including any type of challenge, from logic and math problems to quests in virtual worlds, such as building a computer in Minecraft). As a first step toward this ambitious goal, in this work, we constrain our method to write code for one simulator, namely PyBullet (Coumans & Bai, ). By continuously generating learnable and interesting environments, \ours advances the development of self-improving AI systems, bringing us closer to achieving Darwin Completeness and realizing AI-GAs.

## 2 Related Work

Unsupervised Environment Design.
Unsupervised environment design has garnered increasing interest in RL. Several works have explored the development of auto-curricula to perpetually generate new training environments for agents (Dennis et al., ; Jiang et al., ; Parker-Holder et al., ; Samvelyan et al., ; Wang et al., ; ). However, a significant limitation of these methods is their reliance on predefined or manually curated distributions of tasks or environment parameters (Dennis et al., ; Heess et al., ; Jiang et al., ; Parker-Holder et al., ; Samvelyan et al., ; Wang et al., ; ), inhibiting the generation of any possible learning environment.
Regret-based approaches (Jiang et al., ; Parker-Holder et al., ; Samvelyan et al., ) prioritize tasks with high regret, measured by the difference between the highest known return and the mean return across simulations. Alternative methods calculate learning progress by measuring the difference in the agent’s task success rates across training steps (Kanitscheider et al., ). By focusing on tasks with high learning progress, these approaches aim to guide the agent’s learning towards the most promising areas of the task space.
However, a critical challenge remains in distinguishing which environments are genuinely interesting (Jiang et al., ; Zhang et al., ). In the vast space of any task describable in natural language, there may be countless learnable but not meaningful environments (e.g., kicking a ball into a goal at slightly different positions). Inspired by Zhang et al. (), \ours uses human notions of interestingness distilled into FMs to generate environments that are not only learnable, but also interesting.

Foundation Models for Environment Design.
Recent advancements in FMs have showcased their remarkable ability to capture extensive knowledge across diverse subjects by training on vast text corpora (Bommasani et al., ). Consequently, this has sparked interest in applying FMs to environment design. Ma et al. () utilize FMs to generate code for reward functions while Wang et al. () employ FMs to generate simulation environments and expert demonstrations. However, these methods do not build upon the agent’s previous performance on different tasks, and hence lack an auto-curriculum that can provide an endless stream of environments and tasks. In game level generation, Sudhakaran et al. () and Todd et al. () fine-tune FMs to create domain-specific levels, while Bruce et al. () propose training a world model to generate game environments. These approaches focus on level creation but do not train agents or adapt the difficulty based on the agent’s performance or learning progress. Zala et al. () generate environments as a curriculum to learn a fixed set of tasks but is limited in its ability to generate truly open-ended environments and adapt to the agent’s evolving capabilities. Wang et al. () rely on a predefined set of objects and is limited in reflecting real-world dynamics in its simulation.
Despite significant progress in applying FMs to environment design, opportunities remain for further exploration, such as integrating the generative capabilities of FMs with adaptive auto-curricula. \ours addresses this challenge, aiming to unlock the potential for truly open-ended and effective learning environments.

Foundation Models in Open-Endedness.
The field of open-endedness seeks to create algorithmic systems that produce never-ending innovation (Stanley et al., ). There has been increasing interest in leveraging FMs to generate intelligent variations for code or text in evolutionary algorithms. However, these approaches are often confined to a fixed archive with hand-crafted characteristics (Bradley et al., ; Ding et al., ; Lehman et al., ; Lim et al., ). The core objective of open-endedness algorithms is to generate and solve an endless stream of tasks. One way to do so is to keep an ever-expanding archive of tasks or solutions. Zhang et al. () and Wang et al. () have adopted FMs as a mechanism for auto-curricula, proposing both learnable and interesting tasks for agent training. By restricting the agent’s interactions to a predefined range of environmental conditions, these methods may hinder the development of truly adaptable and versatile agents capable of handling the complexities of real-world scenarios. \ours addresses this limitation by leveraging FMs to generate not only tasks but also the simulated worlds and reward functions, potentially exposing agents to a wider range of challenges and learning opportunities.

## 3 Method

Figure: Figure 1: \ours overview. \ours continuously generate and solve new, interesting tasks in simulation. Our approach maintains a task archive of learned and failed tasks.
Refer to caption: x1.png

leverages FMs, including large language models (LLMs) and vision-language models (VLMs), to autonomously create an endless stream of learnable and interesting tasks for open-ended learning (Figure 1). \ours maintains a growing task archive (Section 3.1) that catalogs successfully learned and completed tasks, as well as unsuccessfully attempted ones. The task generator (Section 3.2) uses information from the archive about what has been learned and what has not, proposing the next *interestingly new* task, described in natural language, for the agent to attempt. Because the model has distilled a sense of interesting from reading the internet, it has a model of interestingness (MoI) that emulates the human capacity for nuanced judgments of interestingness in open-ended learning (Zhang et al., ), which it brings to bear when generating tasks. These tasks are then translated into environment code by an environment generator (Section 3.3), specifying the simulated world and functions required for RL. The newly generated task and its environment code are assessed by a second, post-generation MoI (Section 3.4), to ensure the task is indeed interesting given what has come before (it compares the new task to the most similar tasks already in the archive with retrieval-augmented generation (Lewis et al., ). Tasks deemed interesting are then used to train an RL agent (Section 3.5). If deemed uninteresting, the task is discarded, and a new task is generated. After training, a success detector (Section 3.6) assesses whether the agent has successfully completed the task. Successfully completed tasks are added to the archive. Failed tasks are iterated upon a maximum number of times and added to the archive as failed tasks if the RL agents are not able to solve them. Then, the cycle of generating the next task restarts. Each component of \ours is explained in more detail below, and hyperparameters are shown in Table 1. \ours’s iterative process ensures continuous generation and learning of new interesting tasks, forming a never-ending, growing collection of environments and learned agents.

### 3.1 Task Archive

maintains a continuously expanding archive of tasks, including successfully learned ones and those attempted but failed. Successful tasks serve as stepping stones for creating more complex yet learnable ones, while failed tasks provide insights into generating new tasks within the agent’s current capabilities. \ours uses past experiences to generate novel and diverse challenges, continuously pushing the boundaries of what’s already learned. The task archive is initialized with a few task description seeds in natural language (Appendix J). This archive is unbounded and can grow indefinitely as new tasks are generated and learned. Given the generality, interpretability, and universality of natural language and programs (Hopcroft et al., ), each task is represented by its natural language description and the corresponding environment, represented as executable code.

### 3.2 Task Generator

Open-ended algorithms require focusing on tasks that are both learnable (i.e., not too difficult or too easy for the agent to learn) and interesting (i.e., worthwhile and sufficiently novel). Previous attempts have resulted in pathologies when optimizing against definitions and quantifications of interestingness (Zhang et al., ). Inspired by Zhang et al. (), we harness FMs to model ineffable human notions of interestingness, gleaned from large text corpora of human-generated data. Here, the task generator is an LLM, which proposes novel task descriptions in natural language that are distinct from those already discovered while remaining learnable (full prompt in Section H.1).

To ensure the task generator remains open-ended and continuously suggests new and diverse tasks, it uses the content of the task archive as context. Given the limited context length of current LLMs, we retrieve a predefined number of the most similar tasks (both completed successfully and attempted but failed) to a randomly selected task in the archive (Appendix F). These similar tasks serve as examples and are input into the LLM, which then generates the next learnable and interesting task. We opt for the most similar tasks rather than the most different ones to use previous tasks as stepping stones. As LLMs and FMs improve, we expect that a larger portion of the task archive, or potentially the entire archive, could be used as context. However, partial knowledge of stepping stones might be advantageous for creativity and diversity, much as human scientists and artists benefit from not being aware of everything that has come before. The task generator outputs a natural language description of the next task, crafted to be both achievable and interesting for the agent. This description serves as the basis for the subsequent environment generation step, where the natural language task descriptions are translated into executable code to create learning environments.

### 3.3 Environment Generator

The environment generator, powered by an LLM, translates a given natural language task description into executable (here, Python) code defining the learning environment. Section H.2 shows the full prompt. This code includes specifications for creating the simulated world and functions needed for RL (Sutton & Barto, ) based on the standard API Gymnasium (Towers et al., ): reset, step, reward, and terminated. The reset function resets the environment to an initial state, including setting up the initial positions and orientations of the agent and objects. The step function updates the environment according to the simulated physics, the agent’s action, and any other dynamic behaviors (e.g., moving platforms or activated doors). The reward function returns a scalar number, whose cumulative maximization defines the task. The terminated function indicates whether the agent has reached a terminal state.

For example, if the task is to “cross a bridge with moving segments”, a bridge should be created in the simulated world. The reset function should initialize the agent at the start of the bridge and each segment’s position. The step function should update the environment based on the agent’s actions and each segment’s movement. The reward function should reward the agent’s progress across the bridge, and the termination function should indicate when the agent falls off the bridge.

If compilation errors occur when generating the environment code, the errors (with the traceback) are fed back into the environment generator, which then modifies and improves the environment code. Section H.3 shows the full prompt. This loop is limited to a maximum of five iterations per task. If the code still fails to compile after these attempts, the responsibility reverts to the task generator to propose a new task, potentially one less complex or differently structured.

### 3.4 Post-Generation Model of Interestingness

While ideally all tasks proposed by the task generator should be interesting (owing to its MoI), the limited context length of current FMs prevents the task generator from considering the entire task archive at once. Furthermore, it is often easier for an FM to evaluate whether a solution is good rather than generate a good solution (Bradley et al., ).
\ours draws inspiration from the dynamics of human culture. For example, researchers often study a specific subset of previous works to inspire new ideas. Once a new idea is produced, it is crucial to check the literature to determine whether the contribution is truly novel. If the idea is deemed interestingly new, it is published, adding to the ever-growing archive of human knowledge. This process creates a growing set of stepping stones to leap off of, which is a key ingredient of open-endedness (Stanley & Lehman, ). \ours captures these important dynamics of open-ended algorithms, considering a small batch of related stepping stones when creating a new task (Section 3.2) and then verifying its novelty against the task archive.
Given a newly generated task and its corresponding environment code, we compare it against a predefined number of the most similar tasks from the archive (Appendix F). The post-generation MoI then evaluates if the new task is interesting (e.g., novel, surprising, diverse, worthwhile) (full prompt in Section H.4). If the task is deemed interesting, we proceed to train an RL agent on it. If not, the task is discarded, and a new one is generated.

### 3.5 Training Agents with Reinforcement Learning

A key objective of open-ended learning is to enable agents to master an ever-expanding set of tasks. To achieve this, \ours generates a diverse array of tasks along with their corresponding environments programmed in code. Agents are then trained in these environments using RL to solve the generated tasks. In this work, we utilize the PyBullet physics simulator (Coumans & Bai, ). While any RL algorithm could be used, we employ DreamerV3 (Hafner et al., ). Table 2 details the hyperparameters and compute resources used.
Agents receive proprioceptive (joints positions and velocities) and visual information (images of size $64\times 64\times 3$). While \ours can be applied to any robot, due to computational limitations, we demonstrate results using an R2D2 robot with a discrete action space of six actions: do nothing, go forward, go backward, rotate clockwise, rotate counterclockwise, and jump. This simple action space allows the agent to learn tasks more efficiently, requiring fewer time steps than agents with complex action spaces.

Following Wang et al. (), we train one specialist RL agent for each task. For tasks used to initialize the archive, the RL agents are trained from scratch. For newly generated tasks, the agent continue training from an existing policy previously trained on tasks in the archive. The trained policy is selected from a successfully completed task with the closest embedding to the new task (Appendix F). This approach allows the agents to build upon their existing knowledge and adapt more efficiently to challenges presented by the new tasks. Additionally, color variation in the environments is a form of domain randomization, as an RL agent trained on an environment with specific colors may not perform well in an identical environment with different colors (Tobin et al., ).

### 3.6 Success Detector

In this infinite task space of any task describable in natural language, an essential ingredient to training agents to learn any generated task is a universal reward function, which can evaluate if *any* proposed task has been completed or not. We employ a success detector, instantiated as an LLM or VLM, that assesses whether the agent has successfully completed the given task.

When using an LLM as the success detector, we ask the environment generator to generate an additional success-checking function get_success, alongside the environment code, that checks if the agent has successfully completed the task. The success-checking function serves a different purpose from the reward function. The reward function is designed to enable the RL agent to learn efficiently, which often results in a more complex function than the success-checking function. The complexity of the reward function arises from the need to shape rewards to facilitate optimal learning (Krakovna et al., ). Meanwhile, the success-checking function aims to evaluate whether the agent has accomplished the task. This function is less susceptible to reward hacking, as it does not affect how the agent learn.
For example, consider the task of “run forward”. The success-checking function would evaluate whether the agent’s x-axis position has exceeded a certain threshold. In contrast, the reward function should encourage efficient learning by rewarding the agent’s x-axis velocity and promoting cyclic joint movements that resemble a natural running motion.

The success-checking function can be easily implemented in simulated environments where information can be readily accessed through code. However, it may face challenges in real-world scenarios, off-the-shelf closed-source video games, or even within simulated environments for tasks that involve visual assessment (e.g., building a castle, arranging boxes to resemble an elephant), where the required information is not directly available or difficult to evaluate using only code. A natural solution is to use VLMs as success detectors (Radford et al., ). VLMs can potentially detect success on a wider range of tasks (e.g., tidying a room, doing a backflip (Christiano et al., )) than code generation. We input snapshots of the agent’s behavior every second, the natural language task description, and environment code (Section H.5). Although our preliminary testing found that current VLMs are not yet accurate enough to be used as success detectors, we expect their capabilities to rapidly grow over time and achieve higher accuracy.

If a task is deemed to have been successfully learned by the trained agent, it is added to the archive. If not, the task is returned to the environment generator for modifications to aid in the agent’s learning (e.g., changing the reward function or making the physical world less complex). This feedback loop is repeated for a max number of attempts. If the task remains unlearned after these attempts, it is added to the archive as a failed task and a new task is generated.

## 4 Long Run with Simulated Learning

Figure: Figure 2: Long Run with Simulated Learning. \ours generates a diverse array of tasks, ranging from wildly different objectives to interesting variations of similar overarching tasks. The node color reflects the generation number of the task. A check mark in the node means that the task was successfully learned. A ZZZ symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate a similar yet new and interesting task. Grey nodes show task description seeds that initialized the run.
Refer to caption: x2.png

To illustrate the creative explosion of generated tasks, we run \ours without training RL agents, assuming all generated tasks can be successfully completed. This allows us to showcase a larger number of generated tasks, as training RL agents on each task is more time-consuming. Excluding tasks that did not generate executable code, Figure 2 shows 200 iterations of \ours with the R2D2 robot. Each node represents a generated task, which includes the natural language task description and environment code. The color of the nodes corresponds to the generation number. For better visualization, each task (natural language description and environment code) is encoded using a pre-trained encoder language model (OpenAI’s text-embedding-3-small (OpenAI, )) and then projected into a 2-dimensional space using t-SNE (Van der Maaten & Hinton, ). We manually selected a few tasks that are well-distributed across the embedding space to show a sample of the diversity and creativity of \ours. Connections between nodes indicate which tasks were used as stepping stones to generate the next task, meaning they were provided in context to the task generator. For better readability, Figure 2 only displays the parent task that is closest to the child task in the embedding space and the header text of task descriptions. Appendix B shows the full natural language descriptions of the magnified tasks and provides a more comprehensive visualization of all parent-child task connections. Appendix J shows the three task descriptions that seed the run.

creates tasks that evolve and diverge across the embedding space, forming clusters of related and increasingly complex challenges (Figure 2). For example, the bottom-right region features tasks involving kicking a ball into a goal. Moving towards the bottom-left, the tasks gradually include dynamic elements such as moving obstacles or targets, while still focusing on kicking a ball. As we traverse to the top-left region, the focus shifts to tasks that require pushing or delivering objects into designated receptacles or target places. This marks a significant departure from the ball-kicking tasks and showcases the diversity of the generated environments. The top-right portion of the space is characterized by generations that emphasize navigation challenges, such as traversing (often moving) platforms or navigating across varied terrains (Bauer et al., ).

Furthermore, there are niches where some generations appear to be variations of each other (Figure 2). For example, in the top-right corner, the generations surrounding task 90 involve retrieving an object and returning it to a designated location. While the overarching objective surrounding that niche remains consistent, the tasks and their learning environments exhibit notable differences. These variations manifest in several aspects, such as the simulated world in which the task takes place (e.g., task 90 is outdoors, task 24 is in connected rooms, task 27 is in two levels connected by a ramp). Other variations include the number of objects that need to be retrieved (e.g., task 102 requires multiple objects to be retrieved, while task 90 only requires one), and the allotted time for completing the task (e.g., task 27 has a time limit of 3 minutes, while task 24 has a time limit of 5 minutes). These task variations can potentially allow for the development of robust and adaptable agents capable of handling a wide range of scenarios (Bauer et al., ).

Overall, \ours generates tasks that significantly diverge from the seed tasks used to initialize the archive (the grey nodes in Figure 2). For example, despite the absence of dynamic objects in the seed tasks, \ours generates a substantial number of tasks that incorporate dynamic objects and interactions, such as platforms that move horizontally and vertically, buttons and levers to activate, and moving obstacles. \ours not only explores different task niches (e.g., navigating across different terrains vs. retrieving objects) but also generates interesting variations within each niche (e.g., retrieving objects in different simulated world settings).

## 5 Short Run with Learning

Figure: Figure 3: Short Run with Learning. \ours adapts to the current capabilities of trained RL agents, generating tasks that are both interesting and learnable. Tasks deemed interesting that are successfully learned are marked by a check and failures by a cross. Uninteresting tasks are not trained on and hence not included here. Arrows between tasks indicate instances where \ours modified a task that the RL agent failed to learn, adjusting the task difficulty to facilitate learning.
Refer to caption: x3.png

Previous work (Bauer et al., ) has demonstrated that training a large model on a sufficiently diverse set of tasks results in a generalist agent capable of generalizing to new tasks within that distribution. However, generating such a distribution is both time-consuming and challenging. Furthermore, to achieve efficiency, it is crucial to automatically create a curriculum within this distribution. Since those are the main challenges, our results section focuses on demonstrating that \ours addresses both issues effectively. While our academic lab lacks the resources to train generalist agents on a large set of \ours-generated tasks, our findings suggest that this would be straightforward to do, since (as we show) \ours continuously generates increasingly interesting, complex, and diverse tasks, ensuring a progressive, never-ending curriculum for training.

To demonstrate \ours’s ability to generate tasks of suitable difficulty for training RL agents, we conducted 5 short runs with RL agent training. Due to limited computational resources in our academic lab, the runs are shorter, but still show the creative potential of \ours and its ability to tailor tasks to the agents’ abilities. The success detector (Section 3.6) evaluates if the agent has successfully completed each task. All short runs are initialized with 3 task description seeds (Appendix J). We find that \ours can effectively generate tasks and environment code that are not only creative and interesting but also learnable and appropriately challenging for RL agents (Figure 3, Appendix D). In the run shown in Figure 3, the RL agents successfully completed 16 tasks, failed at 6 tasks, and 1 task was deemed uninteresting by the post-generation MoI. Figure 3 shows images of the trained agents and their corresponding tasks. For better readability, Figure 3 only displays the header text of the task descriptions. Environment code with full task descriptions, images, and a task graph can be found in Appendix C.

leverages previously learned tasks as stepping stones to generate and master more challenging tasks. This iterative process allows RL agents to build upon existing skills to tackle increasingly complex environments. For example, since RL agents in the archive successfully learned to cross a pride-colored bridge with gaps (seed 1) and cross a bridge with moving segments (task 1), \ours branched off these tasks to generate a more challenging task (task 4) combining these elements. This new task required the agent to cross a rainbow bridge with gaps and moving segments. By continuing training from the policy learned on task 1, the RL agent successfully completed task 4.
Furthermore, \ours considers tasks that the RL agents agents failed to learn. For example, since the RL agent failed to learn how to push a box to a target location on a dynamic platform (task 9), future tasks involving crossing platforms did not include pushing objects across them (e.g., tasks 11, 14, 15). This ensures that the generated tasks remain learnable and do not repeatedly incorporate overly difficult challenges.
Similarly, when the RL agent failed to navigate through a terrain with obstacles (task 12), \ours generates an easier obstacle course (task 13) that maintains the same objective but reduces the number of obstacles.
By using the agent’s past experiences (both successes and failures) as building blocks, \ours generates a curriculum of tasks that progressively and interestingly increases in difficulty while remaining learnable. This adaptive task generation process showcases \ours’s capacity to create a tailored curriculum that maintains an appropriate level of challenge, ensuring that the generated tasks are neither too simple nor too complex.

## 6 Quantitative Results

Figure: Figure 4: OMNI-EPIC generates significantly more diverse tasks and continues to innovate throughout the run. (Left) Cell coverage of archive diversity plots in long runs with simulated learning by OMNI-EPIC and the controls. (Right) ANNECS-OMNI measure of progress for OMNI-EPIC and the controls. Dotted lines are median values, shaded regions are 95% confidence intervals.
Refer to caption: x4.png

To evaluate the impact of having a task archive and the contribution of the notion of interestingness in OMNI-EPIC, we compare against two controls: (1) OMNI-EPIC without the task archive (OMNI-EPIC w/o archive), and (2) OMNI-EPIC without the models of interestingness (by removing the request for interesting tasks from the task generator prompts and skipping the post-generation MoI step) (OMNI-EPIC Learning Progress Only). The quantitative measures for comparisons are described next, and OMNI-EPIC significantly outperforms the controls on both metrics.

Task Diversity.
To measure the diversity of generated task archives, we plot the cell coverage of a task archive in a 2D discretized plot. First, we use a pretrained text embedding model (OpenAI’s text-embedding-3-small (OpenAI, )) to encode the generated tasks, then reduce the dimensionality to two via PCA (Maćkiewicz & Ratajczak, ) for easier visualization. Each task in the archive is placed in the appropriate bin, and we count the number of unique bins the algorithm fills, a standard measure of diversity (Mouret & Clune, ; Pugh et al., ). Each method is run with simulated learning, repeated 3 times. OMNI-EPIC achieves significantly higher cell coverage than the controls (p < 0.05, Mann-Whitney U test). This shows that both the task archive and the model of interestingness significantly and quantifiably contribute to OMNI-EPIC’s ability to generate more diverse tasks (Figure 4). Archive diversity plots for each method can be found in Appendix E.

Measure of Progress.
While open-ended systems aim to endlessly create and learn new tasks, the question of how to measure progress in such systems remains. Wang et al. () proposed tracking the Accumulated Number of Novel Environments Created and Solved (ANNECS) throughout the run of an open-ended system. Specifically, ANNECS requires that an environment created at a particular iteration (1) is not too hard or easy for the agent to learn and (2) must eventually be solved by the system. However, ANNECS lacks a measure of how interesting or novel the newly generated task is. To address this limitation, we introduce a new metric to measure progress in open-ended systems, ANNECS-OMNI. Inspired by ideas in Zhang et al. (), ANNECS-OMNI adds a third criterion to ANNECS: the new task must be considered interesting compared to previous tasks (approximated here by asking an FM if the task is interesting given the archive of already-solved tasks).

Each method is run with RL training, repeated 5 times. OMNI-EPIC achieves significantly higher ANNECS-OMNI scores than the controls (p < 0.05, Mann-Whitney U test). As the run proceeds, the ANNECS-OMNI metric consistently increases for OMNI-EPIC, indicating that, for as long as we could afford to run it, the algorithm continuously creates meaningfully new and interesting tasks without stagnation (Figure 4). That is a new high watermark in our field’s longstanding quest to create open-ended algorithms. The ANNECS metric for the different methods can be found in Appendix E.

## 7 Discussion, Future Work, Conclusion

provides a general recipe for generating an endless stream of learnable and interesting environments. By leveraging FMs to generate tasks and their corresponding environment (and reward) code, \ours eliminates the need for handcrafted parameters or predefined task distributions, unshackling algorithms to have the potential to be truly open-ended. \ours can produce a wide variety of tasks, from simple to complex, and allows for the creation of environments that continuously adapt in response to an agent’s developing capabilities. The results demonstrate its effectiveness in generating diverse and creative tasks, highlighting the potential of this approach for training more capable and intelligent agents. By leveraging the generative capabilities of FMs to create a vast array of unique and challenging environments, \ours brings us one step closer to achieving Darwin Completeness, the ability to create *any* possible learning environment.

However, it is important to acknowledge that the current implementation of \ours is not yet Darwin Complete, primarily due to the limitations imposed by the choice of simulator. While \ours can generate a wide variety of tasks and environments, it is constrained by the capabilities and assumptions of the underlying simulation platform. To achieve true Darwin Completeness, \ours could simply be allowed to generate any code, including the ability to download, install, use, or modify any existing simulator, or even write the code for an entirely new simulator from scratch. Since code is Turing Complete, generating code can, in principle, create any computable environment. Enabling the generation of arbitrary code would unlock the full potential of \ours and bring us closer to realizing the vision of Darwin Completeness. Of course, ever smarter FMs are required to take advantage of this opportunity.

The current implementation of \ours trains a population of specialist agents on distinct tasks. However, this approach may not fully capitalize on the potential for agents to generalize across a broader range of tasks. Future research could explore alternative training modalities. One promising direction is to train a single policy across all environments (Bauer et al., ), which could encourage the development of more versatile and adaptable agents. Another approach is to prioritize environments based on learning progress (Kanitscheider et al., ), focusing on tasks that provide the most opportunities for improvement. Each of these strategies introduces unique dynamics into the open-ended environment generation process, and understanding their effects on agent performance and generalization represents an exciting avenue for future research.

In conclusion, \ours represents a leap towards open-ended learning by generating an endless stream of learnable and interesting tasks. Intriguingly, it also provides a new way of creating human entertainment and educational resources by offering a limitless supply of engaging challenges (Appendix A). \ours could potentially be applied in myriad ways, covering anything from math problems and poetry challenges to games and virtual worlds. By leveraging FMs to create tasks and environment code, \ours opens up a vast space of possibilities for AI and human agents to explore and master. By combining that expressive power with human notions of interestingness, \ours presents a promising path towards the development of truly open-ended and creative AI.

#### Acknowledgments

This research was supported by the Vector Institute, the Canada CIFAR AI Chairs program, a grant from Schmidt Futures, an NSERC Discovery Grant, the Center for AI Safety Compute Cluster, and a generous donation from Rafael Cosman. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. We also thank Aaron Dharna, Arthur Braida, Ben Norman, Cong Lu, Gabriel Béna, Luca Grillotti, Rach Pradhan, and Shengran Hu, for insightful discussions and feedback.

## Appendix A Game Interface

Figure: Figure 5: \ours in a game interface.
Refer to caption: x6.png

Apart from training agents, \ours can also be a good form of human entertainment! We created a game interface whereby players can control the R2D2 robot with keyboard inputs, and play the generated tasks (Figure 5). \ours dynamically generates the next interesting level for the player, adjusting based on the player’s skill level by suggesting tasks that are not too easy or difficult. \ours opens a new era for games, where procedural content is automatically generated and tailored to the player’s abilities, ensuring a consistently engaging experience.

## Appendix B Supplementary Materials for Long Run with Simulated Learning

Figure: Figure 6: Long run with simulated learning task graph with all parent-child task connections. This figure presents the same task graph as Figure 2, but with all parent-child task connections displayed. The node color reflects the generation number of the task. A check mark in the node means that the task was successfully learned. A ZZZ symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate a similar yet new and interesting task. Due to the high density of tasks and connections, visualizing all relationships clearly in a static image is challenging. To better understand and navigate the intricate web of task relationships, an interactive version of the task graph is available at [https://dub.sh/omni-epic](https://dub.sh/omni-epic).
Refer to caption: x7.png

The natural language descriptions and environment code for all tasks shown in Figure 2 are available at [https://dub.sh/omni-epic](https://dub.sh/omni-epic). For conciseness, we present the full natural language descriptions of all magnified tasks, and the environment code of one magnified task (Task 76):

ıin 23, 60, 74, 84, 88, 90, 103, 122, 155, 176, 178, 196, 197
Task ı

## Appendix C Supplementary Materials for Short Run with Learning

Figure: Figure 7: Short run with Learning task graph. The node color reflects the generation number of the task. A node with a check mark indicates successful task learning, while a cross mark denotes a task that was attempted but failed to be learned. A ZZZ symbol means that the task was deemed uninteresting and discarded. The node connections illustrate which tasks were conditioned on when asking an FM to generate a similar yet new and interesting task.
Refer to caption: x8.png

ıin 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 23, 25
Task ı

Figure: Figure 8: Task ı of the \ours run presented in Section 5.
Refer to caption: figures/short_run/short_run_%C4%B1.png

## Appendix D Additional Short Runs with Learning

Figure: Figure 9: Additional Short Run with Learning (Run 2). This figure shows another instance of the short run experiment under the same settings as in Figure 3. It demonstrates the progression of tasks generated by \ours and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
Refer to caption: x9.png

Figure: Figure 10: Additional Short Run with Learning (Run 3). This figure shows another instance of the short run experiment under the same settings as in Figure 3. It demonstrates the progression of tasks generated by \ours and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
Refer to caption: x10.png

Figure: Figure 11: Additional Short Run with Learning (Run 4). This figure shows another instance of the short run experiment under the same settings as in Figure 3. It demonstrates the progression of tasks generated by \ours and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
Refer to caption: x11.png

Figure: Figure 12: Additional Short Run with Learning (Run 5). This figure shows another instance of the short run experiment under the same settings as in Figure 3. It demonstrates the progression of tasks generated by \ours and the learning outcomes of the RL agent. Checkmarks indicate successful learning, crosses indicate failures, and arrows show iterations on failed tasks.
Refer to caption: x12.png

## Appendix E Supplementary Materials for Quantitative Results

Figure: Figure 13: Archive diversity results. Archive diversity plots of long runs with simulated learning of OMNI-EPIC and the controls. Fewer tasks fall into the same discretized cells for OMNI-EPIC than OMNI-EPIC Learning Progress only or OMNI-EPIC w/o archive. The substantial difference between the left and center plots is more easily observed in Figure 4.
Refer to caption: x13.png

Figure: Figure 14: (Left) ANNECS and (Right) ANNECS-OMNI results. Short runs with RL training by OMNI-EPIC and the controls are repeated five times. Darker lines are median values, shaded regions are 95% confidence intervals. OMNI-EPIC significantly outperforms the controls on both metrics. There is no difference between ANNECS and ANNECS-OMNI for OMNI-EPIC, indicating that all tasks learned by OMNI-EPIC are considered interesting.
Refer to caption: x14.png

## Appendix F Selecting the most similar tasks

To generate new tasks that are both relevant and challenging, it is essential to select tasks from the archive as a reference. This selection process ensures that the generated tasks build upon the agent’s prior knowledge and skills. \ours’s process of selecting the most similar tasks, given a query task, takes inspiration from Lewis et al. (). The first step is to embed all tasks within the archive into a high-dimensional vector space using a pretrained language encoder. This embedding allows us to represent each task as a dense vector that captures its semantic meaning and characteristics. Once all tasks are embedded, we can efficiently compare the similarity between tasks using cosine similarity. For a given query task, we retrieve a predefined number of tasks from the archive that exhibit the closest cosine similarity to the query’s embedding, ensuring that the selected tasks are the most relevant and similar to the current learning context.

## Appendix G Hyperparameters

**Table 1: \ours hyperparameters**
| Component | Parameter | Value |
| --- | --- | --- |
| General | Max number of iterations | $5$ |
| Embedding method | OpenAI<br>(text-embedding-3-small) |  |
| Task Generator | Number of learned examples | $5$ |
| Number of failed examples | $5$ |  |
| Client | Anthropic |  |
| Model | Claude 3 Opus<br>(claude-3-opus-20240229) |  |
| Temperature | 0 |  |
| Environment Generator | Number of few-shot examples | $5$ |
| Client | Anthropic |  |
| Model | Claude 3 Opus<br>(claude-3-opus-20240229) |  |
| Temperature | 0 |  |
| Post-Generation MoI | Number of similar tasks | $10$ |
| Client | OpenAI |  |
| Model | GPT-4o<br>(gpt-4o-2024-05-13) |  |
| Temperature | 0 0 |  |
| Success Detector | Client | OpenAI |
| Model | GPT-4o<br>(gpt-4o-2024-05-13) |  |
| Temperature | 0 0 |  |
| Task Reflection | Max number of iterations | $1$ |
| Number of few-shot examples | $5$ |  |
| Client | OpenAI |  |
| Model | GPT-4o<br>(gpt-4o-2024-05-13) |  |
| Temperature | 0 0 |  |

**Table 2: DreamerV3 hyperparameters**
| Parameter | Value |
| --- | --- |
| Total time steps | $2\times 10^{6}$ |
| Replay buffer size | $10^{6}$ |
| Batch size | $16$ |
| Batch length | $64$ |
| Discount factor | $0.997$ |
| Learning rate | $3\times 10^{-4}$ |

Compute Resources
For each task, the R2D2 agent is trained for approximately 1 hour using 2 NVIDIA RTX 6000 Ada Generation GPUs and 32 CPU cores.

## Appendix H Prompts

### H.1 Task Generator

System Prompt:

User Prompt:

### H.2 Environment Generator

System Prompt:

User Prompt:

### H.3 Environment Generator Reflection

System Prompt:

User Prompt:

### H.4 Post-Generation Model of Interestingness

System Prompt:

User Prompt:

### H.5 Success Detector

System Prompt:

User Prompt:

Image Examples:

[Uncaptioned image]: x15.png

## Appendix I Few-shot Examples

## Appendix J Task Description Seeds

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Is there a way you could have OMNI-EPIC design simpler problems for which you could thoroughly evaluate RL agents? Even with the amount of computational resources you have available in your research lab?
2. How can you verify whether OMNI-EPIC is solving a retrieval problem or an environment-generation problem?
3. Why is Darwin-completeness defined as the potential to generate any learning environment? How is Darwin's theory of evolution related to the ability to generate learning environments?
soundness: 1
strengths: The paper attempts to tackle the ambitious problem of continually generating a diverse set of problems, from which an RL agent could learn. The idea of leveraging foundation models for this task is a promising idea. These models contain what one might call "general knowledge" that can be helpful in tackling this problem.
summary: The paper presents OMNI-EPIC, a system to design a large library of environments for reinforcement learning agents. OMNI-EPIC relies on Foundation Models, both large language models and vision language models. The system uses a pipeline where problems are specified in natural language, and then it uses an LLM as a synthesizer to generate programs encoding the environments. The environments are also evaluated with another LLM with respect to their interestingness; environments that aren't "interesting" are discarded. Finally, the system also has a module that checks whether a task was solved or not.

A few results of the system show that it is able to generate a diverse set of environments.
weaknesses: There are two key weaknesses with the paper that justifies my scores and recommendation. 

**Scholarship**

First, the paper lacks scholarship, in the sense that the writing is sloppy (you read the sentences and say 'yes, I see what you mean, but what you wrote isn't technically correct.') and it lacks comparison with a very large body of works devoted to generating content in simulated environments: procedural content generation. I will give concrete examples and suggestions next.

The introduction of the paper talks about the grandiose plan of open-endedness, where systems would be able to come up with any computable environment. However, the paper is about generating environments in a restricted environment, with one specific simulator and small and discrete action spaces. It is not clear how the constrained experiments performed in this paper contribute to the grand goals of open-ended learning.

The environments that OMNI-EPIC generates could just be environments that humans implemented, made available online in GitHub, and an LLM was trained on. In this case, OMNI-EPIC is working as a retrieval system that can create different environments within a single codebase. While I see value in such a retrieval system, the paper is not about it. The paper doesn't even consider the possibility that the LLMs are simply retrieving knowledge and replicating it. From what I understand, this doesn't align with the open-ended goal of being... open-ended. A retrieval system will be bounded by whatever we created and made available online. The paper would have to go through a deep re-write to adjust these claims or to perform the proper evaluations to address these concerns. 

Here are three examples of sentences that are sloppy.

1. *Because the model has distilled a sense of interesting from reading the internet.* The model does not read; we should use the correct technical terms. The model is trained on Internet data.
2. *Since code is Turing complete.* This is not true. You could have a language that is not Turing complete. The authors might be referring to commonly used general-purpose languages that are Turing complete. 
3. *While our academic lab lacks the resources to train generalist agents on a large set of OMNI-EPIC-generated tasks, our findings suggest that this would be straightforward.* While I appreciate the use of the word "suggest," I don't see how we can conclude that learning a generalist agent is straightforward. Training RL agents is tricky (catastrophic forgetting and saturated neural networks are examples of issues one might come across). Evidence suggests that doing this in dozens of environments cannot be straightforward.

Overall, the paper currently reads like an informal report, which makes it difficult to pinpoint the problem that is being solved and the claims that are being made.

Finally, OMNI-EPIC is closely related to procedural content generation, an active research area in the computer games community. See the book *Procedural Content Generation in Games* by Noor Shaker, Julian Togelius, and Mark J. Nelson for an outdated overview.

**Evaluation**

The experiments are unfinished, as the environments generated with OMNI-EPIC are not used to actually evaluate RL algorithms and how such environments can potentialize them. The evaluation focuses on metrics of diversity of the environments used, instead of focusing on the actual goal of improving learning. If the goal is to learn a generalist agent (is that true?), then why not evaluate it that way? The current evaluation serves as a proxy for the actual evaluation, and it is not clear how good of a proxy this is.

The paper justifies why the evaluation is restricted by saying they do not have enough computational resources. While I sympathize with the issue, I cannot recommend acceptance of a paper that doesn't evaluate on the problem it is trying to solve.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Is the code open source?
- To what extent will this be limited by the dataset of the coding LLM? Will there ever be truly novel environments generated?
- It is a bit unclear to me, but when starting the process, do you first sample a random task from the archive, and then use this as the reference task to choose the N most similar ones.
- Consider the following case:
	- You have 10 tasks in the archive, say 5 of type A and type B. Type A tasks are all somewhat similar to each other, but different to type B. As an example, type A tasks involve kicking a ball and type B involves crossing a bridge. Suppose the number of tasks given to the LLM if 5, say.
	- Would the following cyclic behaviour be possible? Please comment/explain how or how not, and what possible solutions there are.
		- The task generator receives 5 tasks of type A, and is asked to generate a novel and new task. Then it does this by generating a type B task. However, the post generation MOI rejects this because the archive already has many of these, so the task is not added. Whenever the randomly-chosen task is type A, this happens, and the opposite for type B.
		- If this happens, it seems like the process will just remain stuck and never generate anything new.
- Would there be axes of variation that are not explored at all by an LLM, potentially due to the bias in its data, or by virtue of how it predicts the next token? Concretely, suppose you apply this to a more general environment representation (where the LLM can change everything), and it happens to generate a physics-based R2D2 environment. Then, when generating new tasks, will it ever do something very different, like generating a game like Pong (different controls, doesn't use a robot anymore, etc.) or will it continuously generate variations of R2D2 physics tasks? Please elaborate on this
soundness: 2
strengths: - I agree, using code as an environment representation is indeed much more expressive than changing hand-designed features within a very constrained environment, as prior work does.
- It is great that this paper does provide a full pipeline that theoretically can be run for a long time to continuously generate novel tasks.

I am not giving it a lower score because I think the idea of using code as an environment representation is a good idea that future work can build on. Furthermore, demonstrating a full end-to-end pipeline is useful.
I am not giving it a higher score because I think the results are not that compelling. That is not to say that this approach does not hold promise---and I hope future work/the authors build on this---but the outcomes in this work are not that open ended.
summary: This paper uses an LLM to generate environment code for an RL agent. This is combined with another foundation model that serves as a proxy for human models of interestingness. Together, the first LLM generates code whereas the second guides this process towards novel and interesting tasks. Theoretically, treating the environment as code is a much less constrained way to generate environments compared to much of the prior work. The paper also provides two experiments, one with simulated learning to focus on the environment generation, and another with actual RL agent training to identify how these two components interact.
weaknesses: While code as a representation is expressive in theory, all of the different environments seem quite similar. This is partly due to the agent in all of these tasks being the same.
	- However, if you remove the restriction of using this particular robot in a particular simulator, then the task may become too difficult for current LLMs, or you could end up generating only a certain type of task. 
- It feels like the claims made are that we can express any possible task, and that, when run long enough, we will eventually obtain a very large number of semantically diverse environments; however, the results do not seem to substantiate this claim. 
- Figure 2 is very hard to understand and parse, is there a better way to show the same information?
- This sentence is a bit hard to parse/can be rewritten
> Because the model has distilled a sense of interesting from reading the internet, it has a model ofi nterestingness (MoI) that emulates the human capacity for nuanced judgments of interestingness in open-ended learning (Zhang et al., 2023), which it brings to bear when generating tasks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: **Q1:** Do you have any result or hypothesis on how much the initial set of environments in the task archive conditions the generation of new environments?

**Q2:** How does the task similarity method scale with the number of tasks? If the idea is to endlessly generate tasks, this similarity method should be very computationally efficient.

**Q3:** Does the simplified action space limit the diversity and complexity of the generated environments? 

**Q4:** L283 states "We employ a success detector, instantiated as an LLM or VLM", later L304-305 mentions "Although our preliminary testing found that current VLMs are not yet accurate enough to be used as success detectors". Given the second sentence, I find very surprising the usage of VLMs as success detectors in  OMNI-EPIC. Why does OMNI-EPIC employ VLMs for this purpose even though the authors mention that VLMs are still not ready for the task?
soundness: 2
strengths: - The paper is generally well written and the ideas are mostly clearly transmitted. The figures are clean and support the text, helping to transmit the paper's main ideas.

-  I think that the contribution is substantial. The fact that OMNI-EPIC can successfully generate diverse, novel, and learnable environments through code (Python+PyBullet) constitutes a valuable opportunity for the open-endeness community to analyze these systems and advance the field. 

- Moreover, OMNI-EPIC successfully employs foundation models to leverage notions of human interestingness. Generating novel, but interesting environments (that are relevant to humans) is a key challenge in unsupervised environment design and the open-endedness field. Although previous works have pointed to the possibility of using foundation models for this purpose [1], this work is an excellent example of how foundation models can be successfully used for this purpose, which is also a valuable contribution by itself. 

[1] Hughes, Edward, et al. "Position: Open-Endedness is Essential for Artificial Superhuman Intelligence", ICML24.
summary: The paper introduces OMNI-EPIC, an open-ended system that generates novel and learnable environments and tasks for RL agents. The method leverages current language and vision foundation models to generate the environments (through code) and filter the novel and learnable ones. The authors show how the system generates diverse, interesting, and learnable tasks for RL agents, starting from a very small set of initial environments. Experiments show the potential of the system to produce endless automatic curricula for increasingly complex agents.
weaknesses: **W1:** One of my main concerns with this work is the gap between the claims made and the actual demonstrations in the paper. 
Authors claim multiple times that OMNI-EPIC can generate an endless stream of environments (e.g., L082, L151, L424). See line 424: *"[...] ensuring a progressive, never-ending curriculum for training"*, note that this is a **very** strong claim. However, the experimental results are far from demonstrating this capability. Experiments show the generation of 200 environments, but considering that all are solvable (which I consider a very strong assumption). When considering the performance of RL agents in the generated environments, the authors only show a sequence of 22 tasks.  These are some suggestions to address the issue:

- Use simpler alternatives to Dreamer V3 (e.g., PPO with simpler NN architectures) to reduce the computational cost of training the agent and demonstrate that the novelty and learnability of the tasks generated by OMNNI-EPIC hold for longer environment generation steps.

- Reduce the claims made in the paper. Under the current experimental results, there's no real evidence that OMNI-EPIC could continue to generate novel and learnable environments for very large runs.

**W2:** Although the performance of current LLMs is excellent for many coding and text generation tasks, these also present some limitations, which some relate to difficulties in generating novelty. The fact that OMNI-EPIC completely relies on these models, makes me believe that the presented method would especially suffer from the mentioned novelty issue. This weakness is related to the previous, as no evidence is shown on the capability of OMNI-EPIC to  "endlessly" generate novel and solvable environments.

**W3:** As many deep NN models, LLMs also suffer from brittleness. As OMNI-EPIC relies on multiple LLMs working together, this makes me believe that the presented system can greatly suffer from this brittleness issue, as there is no evidence for the contrary. Further experiments on this topic would greatly help to analyze the actual capabilities and limitations of the proposed method. What is the ratio between successful and failed attempts at generating working/solvable environments? This is only an idea, but many similar experiments could be included that would significantly improve the soundness of the paper.

**Minor issues:** 
- Authors mention multiple times that "code is Turing complete" (e.g., L93, L520). This is very vague, please be rigorous, as many types of "code" (i.e., programming languages) exist but not all are Turing complete. 
- Python being Turing complete does not guarantee that a foundation model (and by consequence, OMNI-EPIC) can generate any computable environment. There would be many computable environments that would have zero probability of being generated by a specific foundation model. Present rigorous arguments or remove/modify this claim.
- I think that the "cell coverage of archive diversity" metric is not explained in the main text. I think that the overall clarity of Section 6 could be improved.
- Check the coherence of the references. For example, some references include the proceedings with different styles: some use "In Proceedings [...]", others "In [...]", and others just directly mention the specific proceeding.

I want to emphasize that I think that the contribution of the paper is valuable to the field and that I'm willing to update my score if the mentioned issues are addressed.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: This section includes questions and comments. 

The "never-ending curriculum" claim:
* The approach does not train a single generalist agent to solve all tasks, why not? Was it tried and did it fail? I assume there would be issues with continual learning / catastrophic forgetting, did the authors find this was a problem?
* Is there transfer learning across tasks? Here, there is a different agent for each new task, but an agent for a new task is initialized to the weights of the agent for the nearest solved task. This supposedly enables some form of transfer learning, but I don't think this is empirically tested anywhere? 
  * Do agents initialized this way empirically learn faster than agents trained from scratch?
  * Is training an agent in a given task generally faster when this task is invented later, than earlier? When invented later, there should be a closer nearest neighbor and if there is transfer learning (previous point), then agents should learn faster and faster as the run progresses.  This would indicate that the overall agent (collection of agents) is actually learning something on the way (getting better at achieving tasks). 
* To show evidence for a curriculum effect driven by task generation, one would also need to show that the difficulty of tasks is increasing with time, i.e. by showing that the success rate of agents trained from scratch is decreasing as the experiment advances. In POET for instance, the latest tasks are shown to be out of reach for agents trained from scratch, is this the case here? Showing this would be strong evidence to support the "curriculum" claim (not sure about the "never-ending" bit).
* The claim that the task generator optimizes for learning progress is also not empirically verified, is it really shifting the distribution towards harder tasks as the experiment advances (point above)? Here one would need to show a baseline that doesn't leverage the success/failure feedback or simply is not prompted to generate tasks of intermediate difficulty and compare the relative abilities of the two algorithms to solve new tasks in the end (by taking the nearest neighbors in the respective archive and training from them). Is the "learning progress" bit actually helping training more competent agents?
* Since the agents don't seem to be trained again on past tasks, the archive does not contain any information about the current capabilities of the meta agent (population of agents). As a result, the task generator cannot generate "tasks of intermediate difficulty" for a progressing agent (it doesn't get information about the current agent). It can only generate tasks of intermediate difficulty for the "average" of all agents since the start of the experiment. 
* Without evidence for transfer learning across tasks, agent (population) improvement over time and task difficulty increase over time, the claim of a "never-ending curriculum learning" is not supported and should thus be removed.

Measuring success / performance
* Success is measured by a VLM, but this VLM is not validated. Does it agree with people? I see human judgements in the video on the website, is there an evaluation of how good the VLM is using these ground truth feedbacks?
* If this signal is not reliable (as suggested early in the paper), then the final metric measuring the number of different tasks solved is also not reliable. Having an unreliable fitness metric is fine during training, but becomes a problem when it's used for evaluation and algorithms comparison.
* The "long run without learning" experiment is interesting to get a sense of the diversity of the tasks, but it would be good to know how many of these tasks are actually learnable? I feel like there is a high chance that while the code is executable, the task might be purely impossible to solve. Maybe this could be obtained by subsampling generated tasks and estimating the proportion of tasks that can be solved? The second experiment does test for solvability (through learning and the solved/failed label), but it would be interesting to know if the in-context learning of the task generator (prompted to generate tasks with intermediate difficulty) is actually working, ie if the ratio of solvable tasks is higher in the second versus the first condition?


Measuring novelty, interestingness, diversity:
* The diversity metric in Figure 4 is computed on the simulated learning variants, which means that it's not filtered for solvable tasks and it's not clear which of these are actually solvable. I feel like we care more about the diversity of solved tasks (so the diversity of learned behaviors).
* Is the PCA computed over all tasks from all algorithms? The discretization size is very important here and I'm not sure these results would hold for different step sizes?
* Wouldn't sampling in context examples uniformly in the archive generate even more diversity by increasing the possibilities for recombinations?
* Is the MoI really doing its job? In Figure 3, tasks are considered novel enough if they just change the surface form of the goal, eg "jumping across platform" is considered new if it's above water, or above lava, although these don't change anything to the task, only the color of the background. More generally, this model is not validated against human judgements but is here also used for algorithm evaluations. 

Open-endedness claim:
* As far as I understand, this claim is only supported by the graph in Figure 4. I appreciate the extra criterion on whether a new problem is "interesting" but this decision mechanism is probably quite unstable, e.g. the order in which tasks are presented probably influences the judgment of whether a new task is deemed novel. 
* Stating that we don't see it plateauing considering there are only 13 environments generated is true but also not a strong marker for open-endedness and shouldn't be used to make this claim.
* Support for actual curriculum learning would help support the claim for open-endedness, although the "never-ending" aspect would probably require much longer runs of the algorithm.

Related Work: The related work mentions key research in environment design but does not acknowledge the older and wider field of intrinsic motivations, which I feel is foundational for the presented work. 

* Learning progress has an older history than Kanitscheider et al. LP in prediction networks dates from Schmidhuber's 91 work on curiosity (see history in https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5508364), and was formalized in 2007 by Oudeyer and Kaplan (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4141061, https://pmc.ncbi.nlm.nih.gov/articles/PMC2533589/)
Around 2013 Oudeyer's lab has been using the "competence progress" form, measuring progress in goal-achievement / task completion to drive exploration and skill learning (https://www.sciencedirect.com/science/article/pii/S0921889012000644 and others in 2013-2014). Since 2018, it's been using with intrinsically motivated deep RL approaches too (http://proceedings.mlr.press/v97/colas19a/colas19a.pdf, https://proceedings.neurips.cc/paper_files/paper/2019/file/b6f97e6f0fd175613910d613d574d0cb-Paper.pdf)
* "Focusing on interesting tasks" is basically what intrinsic motivations are about. In Oudeyer and Kaplan's initial framework on IM, there is already the "competence-based IM" aiming at selecting the most useful goals for exploration and skill learning (https://pmc.ncbi.nlm.nih.gov/articles/PMC2533589/). Since 2018, it's been combined with deep RL and there is now a whole field of methods focusing on that (see review in https://arxiv.org/pdf/2012.09830). 
* VLM success detectors: https://arxiv.org/pdf/2303.07280, I don't think Christiano 2017 was a VLM? There is no language there and the model is for a single task.



Clarification questions and other suggestions:
* Do the authors plan on releasing the code? This approach has many components and many parameters, probably making it hard to reproduce. Releasing the code would ensure reproducibility.
* "Previous attempts have resulted in pathologies when optimizing against definitions and quantifications of interestingness" → which ones? How is the current approach overcoming these issues?
* Are the task embeddings computed from the linguistic descriptions? the code? the agent trajectories? combinations of these?
* It's not so clear what happens to tasks who keep on failing to compile, are they added as failed to archive? If not added to the archive, how is this information routed back to the task generator? If added as failed, shouldn't we differentiate "failed because of compilation" and "failed because the agent couldn't solve it"?
* It would be good to show somewhere the success rates at various stages, e.g. the % of tasks that are successfully turned into executable environment code, the % of these that are accepted by the post-hoc MOI checker, the % of the latter that are successfully solved. In the long run without training, how many of the 200 iterations led to executable environment code? And how is executability checked here? By running a random policy maybe?
* I think it would generally be good to conduct experiments for variants/ablations of the task generator:
  * does the LP prompt actually generate tasks more tailored to the agent?
  * does combining it with success/failure actually help generate tasks that are more solvable and not trivial?
  * does conditioning on nearby tasks help the LP modeling and the effects above compared to sampling tasks uniformly from the archive?


Typos
* L17: "narrow distributions of environment" → +s
* L175: "OMNI-EPIC continuously generate and solve new, interesting tasks in simulation." → generates, solves
soundness: 2
strengths: * The idea is original and forward-looking: this is making first steps towards the future of autotelic learning / open-ended learning by extending the space of tasks an agent can generate to drive its own learning.
* The paper is well written and well motivated, the methods are well described with many details in the appendix.
* Generated tasks are fun and I appreciate the accompanying website showing task descriptions and codes.
summary: OMNI-EPIC is a novel framework that aims to create truly open-ended learning environments by combining foundation models with reinforcement learning. The system generates new tasks (descriptions + code) with potential applications for open-ended learning.

Overall I really like the ambition of this paper, and I think it would be good to show new directions to the community. It's tackling a very important problem and proposes a sensible approach to tackle it.  This said, I think the claims made in the paper are strongly exaggerated and mostly unsupported. Below I will list ideas of follow-up experiments that could help support the claims. 

I feel like the computational constraints of the project associated to its ambition caused a necessary tradeoff between running the main experiments of this paper and running the sufficient set of side experiments (variations, ablations, baselines). 

I will assign a score of 6 as I think this paper is already valuable in its current state (if the claims were to be toned down). I'm willing to raise the score if new experiments are shown to support the claims.

I updated my score in light of the discussion (see below)
weaknesses: * The paper doesn't make an explicit list of contributions. Is it about showing open-ended learning? Is it about extending task generation from goals and levels to full environments? Is it about this and tailoring task distributions to the agent's learning capabilities? I would appreciate a clear statement about the contributions of the paper.
* The paper makes a lot of strong claims about interestingness, diversity, curriculum learning and open-endedness, but I would argue that the experiments don't support most of them (see more below).

Meta-review: This is an ambitious paper that provides a compelling demonstration that an LLM-based code synthesis pipeline, combined with inner-loop RL, can achieve continual discovery of novel environments. Albeit their experiments focus on a limited domain, the ideas and experimental decisions here are useful for the community to build on. Most reviewers agree that the work provides interesting ideas and insights, and that the experiments, as presented, are compelling. Echoing reviewer kcfh's concerns, I encourage the authors to clarify their discussion to emphasize that their work does not directly investigate whether the agents trained via OMNI-EPIC in fact do benefit in general capabilities from the evolutionary process they introduce. Showing the OMNI-EPIC agents show significant (and continued) improvements on held-out environment designs would be useful to support this latter hypothesis. In its current form, this work, however, does show OMNI-EPIC's potential as an engine for discovering novel environments with LLM-based code synthesis, which is the main contribution. It is this experimental result, alongside the proposed approach, that serves as the key, valuable contribution.

Decision: Accept (Poster)